Forum Discussion
Incremental refresh appending not replacing
Hello, no - none of the guidance supplied has provided a resolution. Instead, we switched to a pipeline and used a PySpark notebook. The dataflow simply didn't work as expected.
Community SupportHi georgesmith,
Thanks for confirming and sharing the screenshot.
Since the destination is already set to Replace, the behaviour is likely related to how Dataflow Gen2 incremental refresh processes the defined time ranges rather than the destination setting itself. Incremental refresh only processes data within the configured RangeStart and RangeEnd window, and the Replace operation applies only to that specific time slice, not the entire table. Because of this, existing historical data from previous loads remains unchanged, which can sometimes appear as though new rows are being appended.
You may want to verify that the query includes the correct filter using the incremental parameters (for example, DateColumn >= RangeStart and DateColumn < RangeEnd) and that the same date column is used in the incremental refresh policy. If the table already contains duplicated rows from earlier runs, performing one full refresh to rebuild the table and then running incremental refresh again can help ensure the partitions are replaced as expected.
Thank you again for using the Microsoft Fabric Community Forum.