How do you decide between Dataflow Gen2 and Notebooks for transformation workloads?

Hi ravee432​,

For me, it depends on the data source. If my data source is coming from a connection that I cant use in a notebook, for example a connecting using a data gateway, then I will use a dataflow or a pipeline to copy the data to fabric first. 

If I can access the data in Spark, I default to always using notebooks, they give me more control and are easier for me to maintain.  

Hi ravee432​ 

Adding a few practical signals I use to decide:

Cost/compute pattern — Dataflow Gen2 spins up its own compute per refresh, which is fine for occasional runs but gets expensive if you're refreshing frequently or on tight schedules. Notebooks let you reuse a live Spark session across multiple transformations, which is more efficient when you're running things often.

Debugging and observability — When something breaks in a Dataflow Gen2 pipeline, you're often stuck reading Power Query error messages with limited visibility into what happened row-by-row. Notebooks give you cell-by-cell execution, .show()/.printSchema() checks, and proper stack traces — much easier to troubleshoot as logic gets complex.

Team skill mix — If you have a mixed team (some analysts, some engineers), a common pattern is to use Dataflow Gen2 for the initial ingestion/light cleanup that analysts maintain, then hand off to notebooks for the heavier, performance-sensitive transforms that engineers own. You don't have to pick one for the whole pipeline.

As others said — start with Dataflow Gen2, and move to notebooks once you hit scale, need transformations that don't fold, or need the engineering rigor (testing, version control, CI/CD) that a GUI tool can't give you.

Thanks!

Hi ravee432​,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. 

Thank you.

Hi ravee432​,

May I check if this issue has been resolved? If not, Please feel free to contact us if you have any further questions.

Thank you

Hi ravee432​,

I agree with the points already made, but I would add one 2026-specific nuance: I would no longer use data volume alone as the trigger for moving from Dataflow Gen2 to Spark.

Microsoft's current Fabric transformation guidance positions Dataflow Gen2 as the low-code option for cleaning, filtering, joining, aggregating and shaping data, while notebooks are the code-first option for advanced transformations, custom algorithms and external libraries.

Dataflow Gen2 has also changed quite a bit recently. Current capabilities such as Fast Copy and the Modern Evaluator can materially improve larger workloads, and Partitioned Compute is now available in Preview for partitioned/high-scale transformation scenarios.

So for me the decision is more about transformation shape than a specific row-count threshold.

I would stay with Dataflow Gen2 when the logic is naturally expressed in Power Query and remains easy for the team to understand and operate.

I would move a stage to a notebook when I need custom Python/Scala libraries, Spark-native distributed processing, complex iterative logic, reusable code frameworks, or transformations that become difficult to express and test cleanly in M.

One other current-product point: since April 2026, Microsoft says all newly created Dataflow Gen2 items include CI/CD and Git integration support by default. So version control alone is no longer necessarily a reason to rewrite a Dataflow Gen2 workload as a notebook.

I also would not assume that Spark automatically consumes fewer CUs. Active Spark sessions consume capacity, while Dataflow Gen2 has its own compute engines and pricing characteristics. For an important workload I would run the same representative transformation both ways and compare execution time, CU consumption and operational maintainability in the Capacity Metrics app.

In practice I often prefer a hybrid:

Dataflow Gen2 for ingestion and straightforward shaping -> Lakehouse -> notebook only for the transformations that genuinely benefit from Spark/code.

That lets each engine do the work it is best suited for without forcing the whole pipeline into one tool.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.