Forum Discussion
Best Practices for Handling Incremental Data Loads in Dataflow Gen2
Hi everyone, I am exploring different approaches for handling incremental data loads with Dataflow Gen2 in Microsoft Fabric. For larger datasets, refreshing the entire dataset every time can ...
tayloramy
Super User
Super UserHi codeautomation,
Personally I would (and do) use notebooks to handle incremental data loads.
Dataflow Gen 2 can do it, but not in the way you expect.
https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-incremental-refresh
Dataflow gen 2 will essentially partition your data, and then refresh everything in the newest partition, meaning that if a row from a previous partition was modified you could end up with duplicate records.