Dataflow gen2 refresh Error

smpa01 Hey there! I am PM for Incremental refresh for DFgen2. If you encounter issues, please feel free to open a support ticket to get to the bottom of this.

Looking at your scenario, we think that switching from non-incremental to incremental for the same table is causing the issue. If you check out the docs here: Incremental refresh in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

you see that we mention the following: 

• When you write to a lakehouse, the dataflow maintains it owns bookkeeping of the files that are written to the lakehouse. This is in line with the standard lakehouse pattern. However, this means that if other writers like Spark or other processes write to the same table, it may cause issues with the incremental refresh. We recommend that you don't use other writers to write to the same table while using incremental refresh. If you do, you need to ensure that the other writers don't interfere with the incremental refresh process. Processes like table maintenance and vacuuming are not supported either while using incremental refresh.

What probably happens in your situation is that you try to write to an existing table in which the files are not partitioned to the dates we use for incremental refresh. We advise if you need to have historic data loaded sepretly, to make that a seperate table and use a view that combines both tables into one. 

Hope this helps! and to better understand we need to have a support ticket such as we can dive deeper into your scenario 🙂