Forum Discussion
Gen2 dataflows and deployment pipelines
Hi
We are looking to move from legacy dataflows to Gen2, but in my testing I can't see any Gen2 dataflows in the deployment pipeline.
I wonder if it is because lakehouses are not suported yet and they require staging in order to be published, so need the system generated staging lakehouse. I didn't set a destination in the dataflow to keep it simple for testing. I also tried recreating the dataflows in Dev and Test, hoping that if the same dataflow existed in both it might then deploy (as the staging lakehouses had been created) but will can't see any Gen2 artifacts in the pipeline.
Has anyone managed to use deployment pipelines with Gen2 dataflows?
Thanks.
Hi mikesmall
As of now, Gen2 dataflows (Dataflows Gen2) are not yet fully supported in Power BI Deployment Pipelines, which is likely why you're not seeing them show up in your pipeline environments during testing. Deployment pipelines currently focus on semantic models, reports, and dashboards, and do not automatically include Gen2 dataflows—especially those not tied to a specific destination like a Lakehouse or Warehouse. Since Gen2 dataflows rely on Microsoft Fabric's OneLake infrastructure, their integration with deployment pipelines is still evolving and subject to future roadmap updates.
Furthermore, Gen2 dataflows often require a staging destination (usually a Lakehouse or Warehouse) to properly materialize and store their output, which is not just a convenience—it's essential for how they function and persist data. If you left the destination unset for simplicity, the dataflow wouldn't create or associate with any deployable artifact (like a Lakehouse table), meaning there's nothing concrete for the pipeline to pick up or compare across environments.
In practice, if you're testing Gen2 dataflows and want to prepare for future deployment pipeline integration, it's recommended to assign an explicit destination (e.g., a Lakehouse in your Fabric workspace). Also, monitor the Fabric release roadmap or official Microsoft documentation closely, as support for deploying Gen2 dataflows through pipelines is a highly requested feature and may be included in future updates.
In the meantime, deployment of Gen2 dataflows across environments typically needs to be done manually (e.g., exporting and importing definitions), or automated using APIs or Fabric CLI once such tools support the feature.
16 Replies
Good news. Dataflows Gen2 with CI/CD (preview) are supported in a deployment pipeline.
"Dataflow Gen2 now supports Continuous Integration/Continuous Deployment (CI/CD) and Git integration [...] Additionally, you can use the deployment pipelines feature to automate the deployment of dataflows from your workspace to other workspaces. [...]"
See Dataflow Gen2 with CI/CD and Git integration - Microsoft Fabric | Microsoft Learn
Just a shame you can't refresh these CI/CD flows yet from a pipeline... pretty limited when you have two dozen flows with all kinds of dependencies (for which the pipeline is very neat).
Hi
Are there deployments pipeline stages ( test / prod) rules to modify source or target connections and to modify dfgen2 parameters ?
--
How about :
- copy job change
-Pbi paginated reports sourcing change
- shortcuts target change
on deployments pipelines ?
--
Data factory pipeline can use the new librairies variable (on preview)
Did you ever figure this out? I'm in the same boat. Trying to transition to Gen2 DF's and just realized I can't see them in my existing deployment pipeline. 😕
As of June 2024, Gen2 data flows are still not supported in deployment pipelines. I also cannot find a mention on any roadmap.
Can anyone confirm when this feature is expected? We cannot take a system into production that uses Gen2 flows without deployment pipeline support.
Thanks!Second this - could not recommend moving to production until there is full support for deployment (one way or another) for DFG2.
No, my guess is this is something they are working on for GA.
There is obviously an issue with the auto generated staging lakehouses for Gen2 dataflows (eg DataflowsStagingLakehouse) as they shouldn't be visible, so I suspect that could be the reason the artifacts can't be deployed with a pipeline yet.
This is still not available after GA.
Exactly in the same situation. Instead of the Dataflow Gen2
the depoyment pipeline shows the lakehouse used by the DF Gen2 for staging purposes......
What is more when we recreated the deployment pipeline following error message appeared
So, seems clear that DF Gen2 are not yet supported by now (no idea if the will available after end of preview phase)
I am wondering then what will be approach for separating Development to Production for all related objects. Some managed with deployment pipeline and some with git integration?
Is there any alternative to overcome this shortcoming?
Hi mikesmall
As of now, Gen2 dataflows (Dataflows Gen2) are not yet fully supported in Power BI Deployment Pipelines, which is likely why you're not seeing them show up in your pipeline environments during testing. Deployment pipelines currently focus on semantic models, reports, and dashboards, and do not automatically include Gen2 dataflows—especially those not tied to a specific destination like a Lakehouse or Warehouse. Since Gen2 dataflows rely on Microsoft Fabric's OneLake infrastructure, their integration with deployment pipelines is still evolving and subject to future roadmap updates.
Furthermore, Gen2 dataflows often require a staging destination (usually a Lakehouse or Warehouse) to properly materialize and store their output, which is not just a convenience—it's essential for how they function and persist data. If you left the destination unset for simplicity, the dataflow wouldn't create or associate with any deployable artifact (like a Lakehouse table), meaning there's nothing concrete for the pipeline to pick up or compare across environments.
In practice, if you're testing Gen2 dataflows and want to prepare for future deployment pipeline integration, it's recommended to assign an explicit destination (e.g., a Lakehouse in your Fabric workspace). Also, monitor the Fabric release roadmap or official Microsoft documentation closely, as support for deploying Gen2 dataflows through pipelines is a highly requested feature and may be included in future updates.
In the meantime, deployment of Gen2 dataflows across environments typically needs to be done manually (e.g., exporting and importing definitions), or automated using APIs or Fabric CLI once such tools support the feature.
Thanks Poojara.
Since that post I have now started using Pipelines to extract data into a bronze Lakehouse and Notebooks to transform data, even API calls are being done in Notebooks wherever possible. I don't think I actually use any Dataflows anymore.
I do have some on prem API calls that need to use a Gateway so can't be done in a Notebook (I need to vote for Notebooks being able to use Gateways!) but can still run them as a copy job in a Pipeline.
