Forum Discussion
Helper IVDifference between data flow gen and Pipeline Data Flows
Hi XhevahirMehalla,
Thanks for reaching out to the Microsoft Fabric Community forum.Given your 94 Synapse pipelines with complex Mapping Data Flows, I’d recommend evaluating Fabric’s built-in migration experience for Mapping Data Flow (MDF) transforms in Dataflow Gen2 before considering a complete rewrite using Spark notebooks.
Microsoft Fabric supports migrating eligible Synapse and Azure Data Factory pipelines, converting supported Mapping Data Flows into MDF transforms while migrating the pipeline and transformation logic together.
However, MDF transforms are currently in public preview, and not all Mapping Data Flow capabilities are supported. I’d suggest testing a few representative flows first and validating their compatibility, transformation results, and performance. For flows that rely on unsupported capabilities, alternatives include Spark notebooks, Power Query in Dataflow Gen2, or Warehouse SQL, depending on the requirements.
For more details, please refer to the below Official Microsoft Documentation:
• Mapping Data Flow Transforms in Dataflow Gen2 (Preview) - Microsoft Fabric | Microsoft Learn
• Upgrade Best Practices for Azure Data Factory to Fabric Data Factory - Microsoft Fabric | Microsoft Learn
• Fabric decision guide - copy activity, dataflow, Eventstream, or Spark - Microsoft Fabric | Microsoft LearnI hope this helps. Please feel free to reach out if you have any further questions.
Thank you.
Solution SageHi XhevahirMehalla,
I think the earlier replies already cover the basics, so I will add the part that usually decides which one to pick in practice.
The two are not really competing options. A Dataflow Gen2 is a Power Query transformation item, while a pipeline is the orchestrator and its Copy activity is the bulk mover. A pipeline can also call a Dataflow Gen2 through the Dataflow activity, so they are often used together.
The choice normally comes down to transformation shape, data volume and who will maintain it. Microsoft summarizes the trade-offs in its Fabric decision guide
A simple way to think about it is like this
move data as is, large volume
-> pipeline Copy activity
reshape, join, clean with a low code UI
-> Dataflow Gen2
heavy custom logic or very large transforms
-> notebook, called from a pipelineTwo practical points are worth checking. A Dataflow Gen2 consumes capacity based on the Power Query engine run time, so a heavy refresh can cost more than a Copy activity doing the same movement, and staging also adds storage and an extra hop. It also helps a lot if your source steps fold back to the source, because then the work happens upstream and not in the mashup engine.
So I would start with a pipeline Copy activity for ingestion, add a Dataflow Gen2 only for the shaping steps that really need Power Query, and move that part to a notebook if the volume grows.
AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.