Best Practices for Handling Incremental Data Loads in Dataflow Gen2

Hi everyone,

I am exploring different approaches for handling incremental data loads with Dataflow Gen2 in Microsoft Fabric.

For larger datasets, refreshing the entire dataset every time can become inefficient, so I am interested in understanding how others are designing their dataflows to process only new or changed records.

A few questions:

How are you identifying and filtering changed records between dataflow runs?

Is it better to manage incremental logic directly inside Dataflow Gen2, or use a Fabric pipeline to control the process?

How do you handle failed runs or partially processed data without creating duplicate records?

Are there any recommended patterns for maintaining good performance as the volume of historical data grows?

I would appreciate hearing about approaches that have worked well in real Fabric environments.