Forum Discussion
Databricks connection's in Dataflow Gen 2 vs Data Pipeline
Dataflow Gen2 supports Databricks connectivity well because it’s built on the mature Power Query engine with a rich connector ecosystem inherited from Power BI and Power Platform.
Fabric Data Pipelines are primarily orchestration-focused and currently have more limited native Databricks connector capabilities.
Key difference:
Dataflow Gen2 → Data ingestion & transformation layer
Fabric Pipelines → Workflow orchestration & automation layer
Fabric is also heavily optimized around:
Hello Mani_DNPM0430,
Your understanding is generally correct.Dataflow Gen2 and Fabric Data Pipelines serve different purposes:
Dataflow Gen2 is designed for data ingestion and transformation. It uses the Power Query engine and supports a broad range of connectors, including Databricks, making it well suited for ETL/ELT scenarios.
Fabric Data Pipelines are primarily intended for orchestration and automation. They coordinate activities such as Copy, Notebook execution, Stored Procedures, and Dataflows, but currently have more limited native connectivity compared to Dataflow Gen2.
A common architecture is:
Dataflow Gen2 → Connect to Databricks, ingest and transform the data.
Data Pipeline → Orchestrate the end-to-end workflow, schedule executions, manage dependencies, and trigger downstream processes.
This separation allows each service to focus on its strengths - Dataflow Gen2 for data movement and transformation, and Pipelines for orchestration.
Best regards,
Omkar Shinde
Microsoft Fabric Enthusiast | Power BI Consultant
💡If you found this response helpful, please consider giving it a Kudos.
✅If this resolves your question, please mark it as the Accepted Solution to help others in the community.
1 Reply
- Omkar_1712
Impactful Individual
Hello Mani_DNPM0430,
Your understanding is generally correct.Dataflow Gen2 and Fabric Data Pipelines serve different purposes:
Dataflow Gen2 is designed for data ingestion and transformation. It uses the Power Query engine and supports a broad range of connectors, including Databricks, making it well suited for ETL/ELT scenarios.
Fabric Data Pipelines are primarily intended for orchestration and automation. They coordinate activities such as Copy, Notebook execution, Stored Procedures, and Dataflows, but currently have more limited native connectivity compared to Dataflow Gen2.
A common architecture is:
Dataflow Gen2 → Connect to Databricks, ingest and transform the data.
Data Pipeline → Orchestrate the end-to-end workflow, schedule executions, manage dependencies, and trigger downstream processes.
This separation allows each service to focus on its strengths - Dataflow Gen2 for data movement and transformation, and Pipelines for orchestration.
Best regards,
Omkar Shinde
Microsoft Fabric Enthusiast | Power BI Consultant
💡If you found this response helpful, please consider giving it a Kudos.
✅If this resolves your question, please mark it as the Accepted Solution to help others in the community.