Forum Discussion
Difference between data flow gen and Pipeline Data Flows
- 23 days ago
Hi XhevahirMehalla,
Thanks for reaching out to the Microsoft Fabric Community forum.Given your 94 Synapse pipelines with complex Mapping Data Flows, I’d recommend evaluating Fabric’s built-in migration experience for Mapping Data Flow (MDF) transforms in Dataflow Gen2 before considering a complete rewrite using Spark notebooks.
Microsoft Fabric supports migrating eligible Synapse and Azure Data Factory pipelines, converting supported Mapping Data Flows into MDF transforms while migrating the pipeline and transformation logic together.
However, MDF transforms are currently in public preview, and not all Mapping Data Flow capabilities are supported. I’d suggest testing a few representative flows first and validating their compatibility, transformation results, and performance. For flows that rely on unsupported capabilities, alternatives include Spark notebooks, Power Query in Dataflow Gen2, or Warehouse SQL, depending on the requirements.
For more details, please refer to the below Official Microsoft Documentation:
• Mapping Data Flow Transforms in Dataflow Gen2 (Preview) - Microsoft Fabric | Microsoft Learn
• Upgrade Best Practices for Azure Data Factory to Fabric Data Factory - Microsoft Fabric | Microsoft Learn
• Fabric decision guide - copy activity, dataflow, Eventstream, or Spark - Microsoft Fabric | Microsoft LearnI hope this helps. Please feel free to reach out if you have any further questions.
Thank you.
Hi XhevahirMehalla,
Thanks for reaching out to the Microsoft Fabric Community forum.
Given your 94 Synapse pipelines with complex Mapping Data Flows, I’d recommend evaluating Fabric’s built-in migration experience for Mapping Data Flow (MDF) transforms in Dataflow Gen2 before considering a complete rewrite using Spark notebooks.
Microsoft Fabric supports migrating eligible Synapse and Azure Data Factory pipelines, converting supported Mapping Data Flows into MDF transforms while migrating the pipeline and transformation logic together.
However, MDF transforms are currently in public preview, and not all Mapping Data Flow capabilities are supported. I’d suggest testing a few representative flows first and validating their compatibility, transformation results, and performance. For flows that rely on unsupported capabilities, alternatives include Spark notebooks, Power Query in Dataflow Gen2, or Warehouse SQL, depending on the requirements.
For more details, please refer to the below Official Microsoft Documentation:
• Mapping Data Flow Transforms in Dataflow Gen2 (Preview) - Microsoft Fabric | Microsoft Learn
• Upgrade Best Practices for Azure Data Factory to Fabric Data Factory - Microsoft Fabric | Microsoft Learn
• Fabric decision guide - copy activity, dataflow, Eventstream, or Spark - Microsoft Fabric | Microsoft Learn
I hope this helps. Please feel free to reach out if you have any further questions.
Thank you.
Hi XhevahirMehalla,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.
Thank you.