Forum Discussion
Incremental refresh appending not replacing
I've created an incremental refresh of a table in Fabric, using a lakehouse as the default destination. When I run the Gen2 dataflow, it appends the data to the table, rather than replacing it. I belive this is a known bug, is there any suggested fix for this?
Hello, no - none of the guidance supplied has provided a resolution. Instead, we switched to a pipeline and used a PySpark notebook. The dataflow simply didn't work as expected.
6 Replies
Hi georgesmith,
Thanks for confirming and sharing the screenshot.Since the destination is already set to Replace, the behaviour is likely related to how Dataflow Gen2 incremental refresh processes the defined time ranges rather than the destination setting itself. Incremental refresh only processes data within the configured RangeStart and RangeEnd window, and the Replace operation applies only to that specific time slice, not the entire table. Because of this, existing historical data from previous loads remains unchanged, which can sometimes appear as though new rows are being appended.
You may want to verify that the query includes the correct filter using the incremental parameters (for example, DateColumn >= RangeStart and DateColumn < RangeEnd) and that the same date column is used in the incremental refresh policy. If the table already contains duplicated rows from earlier runs, performing one full refresh to rebuild the table and then running incremental refresh again can help ensure the partitions are replaced as expected.
Thank you again for using the Microsoft Fabric Community Forum.
Hi georgesmith,
Just checking in to see if the issue has been resolved on your end. If the earlier suggestions helped, that’s great to hear! And if you’re still facing challenges, feel free to share more details happy to assist further.Thank you.
Hi georgesmith,
Thank you for reaching out to the Microsoft Fabric Community Forum.With Dataflow Gen2 incremental refresh, the dataflow processes only the incremental portion of the data, but how the data is written to the destination depends on the destination update method. If the destination is configured with Append, each refresh will add the new rows to the existing Lakehouse table, which can appear as though incremental refresh is appending data instead of replacing it.
You may want to check the Data destination settings in your Dataflow Gen2. Open the dataflow in edit mode, select the query that loads the table, go to Data destination → Settings (gear icon), and verify the Update method. If it is set to Append, consider changing it to Replace, then republish and run the refresh again. The documentation below explains how destination update methods work in Dataflow Gen2: https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-data-destinations-and-managed-settings
Hope this clarifies. Let us know if you have any doubts regarding this. Please feel free to reach out to us. We will be happy to help.
Thank you for using the Microsoft Fabric Community Forum.
