Dataflow gen2 in deployment pipelines

[Gpop13]

The dataflow gen2 has been listed as preview feature in the deployment pipeline.

https://learn.microsoft.com/en-us/fabric/cicd/deployment-pipelines/intro-to-deployment-pipelines?tab...

however, I don't see its still compatible to be moved in a deployment pipeline.

Anyone knows when this will be available?

[v-prasare]

We are following up once again regarding your query. Could you please confirm whether you have raised ideas forum with Microsoft?

If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community. If we don’t hear back, we’ll go ahead and close this thread.

Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.

Thank you for your understanding and participation.

[Stinkys]

I am hitting the same issue. How are people actually deploying DFG2 resources to multi-stage environments via pipelines currently? Or is it just that nobody is doing it because it's not possible?

[Gpop13]

Hi @Stinkys - The DFG2 are still not supported in depl pipelines. But DFG2 CICD versions are.

I did try that approach but the biggest issue I faced was even if the DFG2 is deployed to next stage, all the destination settings point to the source stage. Example, if in a DFG2, Query1 writes to a lakehouse table in dev env, after its deployed to Test, this setting will still be the same and NOT pointing to corressponding Test lakehouse. 

This was a very manually work for me. 

In the first go, I recreated all the DFG2 in all stages, updated the target connections accordingly. Now whener there is a change to existing query, I manually update (advanced query) in each env (I know this is not the best solution but closest to an easier approach I could have)

[Stinkys]

Thanks for the response @Gpop13 . I am facing the exact same issue as you. I guess manually updating the default data destination in the next stage, as you are, is the only answer. Looks like the functionality we want will be introduced sometime in Q3 this year.

[Gpop13]

Yeah, I even checked with Microsoft by opening a ticket but they mentioned this is a limitation for now. Unless someone has any other better solution, I guess that is what we have to live with as of now. Where did you see Q3 for this feature, can you share please?

[Stinkys]

On the Microsoft Fabric Roadmap

[Gpop13]

This seems to have been addressed in a recent announcement. We can save the dataflow ge2 as dataflow (CI/CD) - that will allow us to integrate them with GIT and also invoke from the pipeline.

Fabric March 2025 Feature Summary | Microsoft Fabric Blog | Microsoft Fabric

[Gpop13]

Please vote this idea here to support 

Orchestrating a refresh of a Dataflow Gen2 with CI/CD and Git should be supported in Fabric data pipelines.

Microsoft Idea

[Madalina2801]

Hello,

Will this limitation be removed in a future release? This is a major problem because the main purpose of the dataflows is to transform data in pipeline orchestration.

• Orchestrating a refresh of a Dataflow Gen2 with CI/CD and Git support isn't possible in Fabric data pipelines.

[Gpop13]

Please vote this idea here

Microsoft Idea

[v-prasare]

@Gpop13, Thanks for reaching out to MS Fabric community support.

as per docs the release of CI/CD and Git integration support has been rescheduled to January to ensure the quality of the release and ensure that there is no impact to existing operations. We apologize for any inconvenience and appreciate your understanding. 

I’ll keep a close watch for updates regarding its integration and will notify you as soon as I learn more, and I recommend keeping an eye on the official Microsoft documentation, as it is regularly updated with the latest information and feature availability

please refer to below similar reasolved thread from community and let me know if this help you

Solved: What's up with Dataflow Gen2 and Dataflow Gen2 (CI... - Microsoft Fabric Community

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

[Gpop13]

Thank you for the response. We are already at the end of January, hope to get the release/update in a couple of days now.

Also, I do not see the option below too

Once the CICD is enabled, and once we are able to use the dataflow gen2 in a deployment pipeline, I hope that it will support existing dataflow gen2 as well, and not just the ones newly created with this checkbox?

[v-prasare]

@Gpop13, Thanks for your valuable insights and feedback.

I will closely monitor updates on its integration and inform you as soon as I receive any new information. meanwhile, I request you to submit your feedback here in documentation. so, that product team will take appropriate action here. this will help community members who are dependent on this usecase.

Thanks,

Prashanth Are

MS Fabric community support

please Mark post as a solution, this will help others! If our response(s) assisted you in any way, don't forget to drop me a "Kudos".

[Gpop13]

I can see that the feature is in preview now, however, the CICD and deployment pipeline support is only for any items created after today? What about the dataflow gen2 created before?

[v-prasare]

To enable CI/CD and Git integration for your existing Dataflow Gen2 items, you will need to recreate them with the integration enabled. This involves creating a new Dataflow Gen2, enabling Git integration during its creation, and then replicating the configurations and dataflows from your existing dataflow into the new one. Once this process is complete, the new dataflow will support CI/CD and Git integration.

Thanks,

Prashanth Are

MS Fabric community support

[Gpop13]

Thanks @v-prasare for the response. I have started recreating the dataflow gen2 with enabled Git integration. However, I have hit a major blocker since it seems the data pipelines won't be able to add these new dataflows for orchestration. So it is of no use to me even if I recreate, unless I can use them in a data pipeline activity, which the the main orchestration process I am using. Is there a workaround for this?

• Orchestrating a refresh of a Dataflow Gen2 with CI/CD and Git support isn't possible in Fabric data pipelines.

[PeterAnton]

We just ran into the same problem.  Major roadblock for our Fabric ETL implementation.  We may need to consider abandoning Dataflow Gen2 and use notebooks.  Anyone else having the same problem?

BYW --> the term Fabric data pipelines and deployment pipelines are very similar which is causing a lot of confusion in posts and discussions

[v-prasare · Accepted solution]

Your feedback is extremely valuable and highlights a key area for improvement. as Microsoft's official support for for pipeline orchestration support with CI/CD-enabled Dataflow Gen2 is limited we recommend to submit your ideas here to Fabric team: https://ideas.fabric.microsoft.com/ideas/search-ideas/

as feedback given here will closely monitored by PG team. We are committed to ensuring our solutions meet your business needs and workflows effectively. Thank you for your patience and understanding as we work to enhance this functionality.