Forum Discussion
Gen2 dataflows and deployment pipelines
- 1 year ago
Hi mikesmall
As of now, Gen2 dataflows (Dataflows Gen2) are not yet fully supported in Power BI Deployment Pipelines, which is likely why you're not seeing them show up in your pipeline environments during testing. Deployment pipelines currently focus on semantic models, reports, and dashboards, and do not automatically include Gen2 dataflows—especially those not tied to a specific destination like a Lakehouse or Warehouse. Since Gen2 dataflows rely on Microsoft Fabric's OneLake infrastructure, their integration with deployment pipelines is still evolving and subject to future roadmap updates.
Furthermore, Gen2 dataflows often require a staging destination (usually a Lakehouse or Warehouse) to properly materialize and store their output, which is not just a convenience—it's essential for how they function and persist data. If you left the destination unset for simplicity, the dataflow wouldn't create or associate with any deployable artifact (like a Lakehouse table), meaning there's nothing concrete for the pipeline to pick up or compare across environments.
In practice, if you're testing Gen2 dataflows and want to prepare for future deployment pipeline integration, it's recommended to assign an explicit destination (e.g., a Lakehouse in your Fabric workspace). Also, monitor the Fabric release roadmap or official Microsoft documentation closely, as support for deploying Gen2 dataflows through pipelines is a highly requested feature and may be included in future updates.
In the meantime, deployment of Gen2 dataflows across environments typically needs to be done manually (e.g., exporting and importing definitions), or automated using APIs or Fabric CLI once such tools support the feature.
Did you ever figure this out? I'm in the same boat. Trying to transition to Gen2 DF's and just realized I can't see them in my existing deployment pipeline. 😕
- raschaoot2 years agoFrequent Visitor
As of June 2024, Gen2 data flows are still not supported in deployment pipelines. I also cannot find a mention on any roadmap.
Can anyone confirm when this feature is expected? We cannot take a system into production that uses Gen2 flows without deployment pipeline support.
Thanks!- DanielAmbler2 years agoHelper II
Second this - could not recommend moving to production until there is full support for deployment (one way or another) for DFG2.
- cwilce2 years agoRegular Visitor
Just adding this from reddit where the same question was asked https://www.reddit.com/r/MicrosoftFabric/comments/1dogga0/dataflows_gen2_with_deployment_pipelines/
TLDR: Gen2 Dataflows with Deployment Pipelines isn't in the current roadmap which goes up until October. Check back then to get the next 6 month roadmap (which will hopefully have this feature included!).
- mikesmall2 years agoAdvocate I
No, my guess is this is something they are working on for GA.
There is obviously an issue with the auto generated staging lakehouses for Gen2 dataflows (eg DataflowsStagingLakehouse) as they shouldn't be visible, so I suspect that could be the reason the artifacts can't be deployed with a pipeline yet.
- abpgupta2 years agoContinued Contributor
This is still not available after GA.
- mikesmall2 years agoAdvocate I
I have had a look at the roadmap and in Q2 2024 they are going to provide Git integration for dataflows, I wonder if this is the route they will go down for deploying content. Would be useful to know if including Gen 2 DFs in deployment pipelines was even a consideration though.
- alfBI2 years agoResponsive Resident
Exactly in the same situation. Instead of the Dataflow Gen2
the depoyment pipeline shows the lakehouse used by the DF Gen2 for staging purposes......
What is more when we recreated the deployment pipeline following error message appeared
So, seems clear that DF Gen2 are not yet supported by now (no idea if the will available after end of preview phase)