Clarification on Dataflow Gen2 for Incremental Loads (Delta/SCD Type 2)

Hi JohnT_2307 ,

You mentioned about the Scalability and Limitations in your previous response. Please refer below things.

1. Performance Bottlenecks : Query folding is critical, Without query folding the performance will be degrades. Alternatively avoid complex transformations in Power Query that prevent folding. Use staging in Lakehouse/Warehouse to simplify logic.

2.Data Volume: There is No hard limits published, but performance may degrade with tens of millions of rows. Please consider partitioning and incremental refresh to manage large volumes.

3.Timeouts: Long-running queries may time out, especially if folding is not enabled.

Solution: Break data into smaller parts using filters. Use staging layers to isolate heavy transformations. Offload to Notebooks for more control.

Please refer below Microsoft official documents.

Incremental refresh in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

Slowly changing dimension type 2 - Microsoft Fabric | Microsoft Learn

Best practices for getting the best performance with Dataflow Gen2 in Fabric Data Factory - Microsoft Fabric | Microsoft Learn

Dinesh

Hi @JohnT_2307 ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.

Regards,

Dinesh