Forum Discussion
Gen2 dataflows and deployment pipelines
Hi mikesmall
As of now, Gen2 dataflows (Dataflows Gen2) are not yet fully supported in Power BI Deployment Pipelines, which is likely why you're not seeing them show up in your pipeline environments during testing. Deployment pipelines currently focus on semantic models, reports, and dashboards, and do not automatically include Gen2 dataflows—especially those not tied to a specific destination like a Lakehouse or Warehouse. Since Gen2 dataflows rely on Microsoft Fabric's OneLake infrastructure, their integration with deployment pipelines is still evolving and subject to future roadmap updates.
Furthermore, Gen2 dataflows often require a staging destination (usually a Lakehouse or Warehouse) to properly materialize and store their output, which is not just a convenience—it's essential for how they function and persist data. If you left the destination unset for simplicity, the dataflow wouldn't create or associate with any deployable artifact (like a Lakehouse table), meaning there's nothing concrete for the pipeline to pick up or compare across environments.
In practice, if you're testing Gen2 dataflows and want to prepare for future deployment pipeline integration, it's recommended to assign an explicit destination (e.g., a Lakehouse in your Fabric workspace). Also, monitor the Fabric release roadmap or official Microsoft documentation closely, as support for deploying Gen2 dataflows through pipelines is a highly requested feature and may be included in future updates.
In the meantime, deployment of Gen2 dataflows across environments typically needs to be done manually (e.g., exporting and importing definitions), or automated using APIs or Fabric CLI once such tools support the feature.
Hi mikesmall
As of now, Gen2 dataflows (Dataflows Gen2) are not yet fully supported in Power BI Deployment Pipelines, which is likely why you're not seeing them show up in your pipeline environments during testing. Deployment pipelines currently focus on semantic models, reports, and dashboards, and do not automatically include Gen2 dataflows—especially those not tied to a specific destination like a Lakehouse or Warehouse. Since Gen2 dataflows rely on Microsoft Fabric's OneLake infrastructure, their integration with deployment pipelines is still evolving and subject to future roadmap updates.
Furthermore, Gen2 dataflows often require a staging destination (usually a Lakehouse or Warehouse) to properly materialize and store their output, which is not just a convenience—it's essential for how they function and persist data. If you left the destination unset for simplicity, the dataflow wouldn't create or associate with any deployable artifact (like a Lakehouse table), meaning there's nothing concrete for the pipeline to pick up or compare across environments.
In practice, if you're testing Gen2 dataflows and want to prepare for future deployment pipeline integration, it's recommended to assign an explicit destination (e.g., a Lakehouse in your Fabric workspace). Also, monitor the Fabric release roadmap or official Microsoft documentation closely, as support for deploying Gen2 dataflows through pipelines is a highly requested feature and may be included in future updates.
In the meantime, deployment of Gen2 dataflows across environments typically needs to be done manually (e.g., exporting and importing definitions), or automated using APIs or Fabric CLI once such tools support the feature.
Thanks Poojara.
Since that post I have now started using Pipelines to extract data into a bronze Lakehouse and Notebooks to transform data, even API calls are being done in Notebooks wherever possible. I don't think I actually use any Dataflows anymore.
I do have some on prem API calls that need to use a Gateway so can't be done in a Notebook (I need to vote for Notebooks being able to use Gateways!) but can still run them as a copy job in a Pipeline.