Forum Discussion

georgesmith's avatar
georgesmith
New Member
6 months ago
Solved

Incremental refresh appending not replacing

I've created an incremental refresh of a table in Fabric, using a lakehouse as the default destination. When I run the Gen2 dataflow, it appends the data to the table, rather than replacing it. I bel...
CICD
  • georgesmith's avatar
    georgesmith
    5 months ago

    Hello, no - none of the guidance supplied has provided a resolution. Instead, we switched to a pipeline and used a PySpark notebook. The dataflow simply didn't work as expected.

v-kpoloju-msft's avatar
v-kpoloju-msft
Icon for Community Support rankCommunity Support
6 months ago

Hi georgesmith, 
Thank you for reaching out to the Microsoft Fabric Community Forum.

With Dataflow Gen2 incremental refresh, the dataflow processes only the incremental portion of the data, but how the data is written to the destination depends on the destination update method. If the destination is configured with Append, each refresh will add the new rows to the existing Lakehouse table, which can appear as though incremental refresh is appending data instead of replacing it.

You may want to check the Data destination settings in your Dataflow Gen2. Open the dataflow in edit mode, select the query that loads the table, go to Data destination → Settings (gear icon), and verify the Update method. If it is set to Append, consider changing it to Replace, then republish and run the refresh again. The documentation below explains how destination update methods work in Dataflow Gen2: https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-data-destinations-and-managed-settings 

Hope this clarifies. Let us know if you have any doubts regarding this. Please feel free to reach out to us. We will be happy to help.

Thank you for using the Microsoft Fabric Community Forum.