Forum Discussion
Dataflow Gen 1 Scalability
Hi. It depends. Can you share more details about the scenario? It's not the most scalable feature and they are removing the support soon, but it depends what you are building. The data integration evolution it's fabric data pipelines or fabric dataflows gen2.
I hope that helps,
Dataflow Gen1 scalability is somewhat limited compared to Gen2 where it depends heavily on your capacity - Pro vs Premium or Fabric capacity and how the dataflow is structured.
On Pro there are refresh timeouts and no support for incremental refresh or DirectQuery. So very large datasets or complex transformations can struggle to scale.
With Premium or Fabric capacity, Gen1 dataflows scale much better since they support incremental refresh, reusable ETL and higher compute allocation through power query engine running on dedicated capacity. Even then, they are less scalable than Gen2 dataflows for very large scale ETL workloads.
Please note that Dataflow Gen 1 is in Legacy state and will be retiring soon.
Dataflow Gen1 scalability is somewhat limited compared to Gen2 where it depends heavily on your capacity - Pro vs Premium or Fabric capacity and how the dataflow is structured.
On Pro there are refresh timeouts and no support for incremental refresh or DirectQuery. So very large datasets or complex transformations can struggle to scale.
With Premium or Fabric capacity, Gen1 dataflows scale much better since they support incremental refresh, reusable ETL and higher compute allocation through power query engine running on dedicated capacity. Even then, they are less scalable than Gen2 dataflows for very large scale ETL workloads.
Please note that Dataflow Gen 1 is in Legacy state and will be retiring soon.