Forum Discussion
Deploy dataflow gen2 using deployment pipeline
How to migrate the dataflow between multiple stages dev test and prod using deployment Pipeline. The dataflow is connected to lakehouse and datawarehouse. Could you some one please help on this?
Thankyou, MasonMA, for your response.
Hi DiKi-I,We appreciate your question posted on the Microsoft Fabric Community Forum.
From what I understand, right now Dataflow Gen2 does not automatically update its Lakehouse or Data Warehouse connections when you deploy it. This creates some difficulties while moving across different environments because you need to change the connections manually.
Please follow these steps that might help solve the problem:
- Create three separate Fabric workspaces named Dev, Test, and Prod.
- Go to Deployment Pipelines in Fabric. Create a new pipeline and assign the Dev workspace to Stage 1. Then link the Test and Prod workspaces to the next stages.
- After setting up the pipeline, the Dataflow Gen2 will show up as an artifact. Click on Deploy to Test, and then Deploy to Prod. After deployment, update the Lakehouse or Data Warehouse connections manually in each stage.
Also, please check these links for more information:
The Microsoft Fabric deployment pipelines process - Microsoft Fabric | Microsoft Learn
Dataflow Gen2 with CI/CD and Git integration - Microsoft Fabric | Microsoft Learn
Overview of Fabric deployment pipelines - Microsoft Fabric | Microsoft LearnIf you find this response helpful, please mark it as the accepted solution and give kudos. This will help other community members who may have the same question.
If you have any more questions, feel free to reach out to the Microsoft Fabric community.
Thank you.
7 Replies
You can refer to this topic below.
Thankyou, MasonMA, for your response.
Hi DiKi-I,We appreciate your question posted on the Microsoft Fabric Community Forum.
From what I understand, right now Dataflow Gen2 does not automatically update its Lakehouse or Data Warehouse connections when you deploy it. This creates some difficulties while moving across different environments because you need to change the connections manually.
Please follow these steps that might help solve the problem:
- Create three separate Fabric workspaces named Dev, Test, and Prod.
- Go to Deployment Pipelines in Fabric. Create a new pipeline and assign the Dev workspace to Stage 1. Then link the Test and Prod workspaces to the next stages.
- After setting up the pipeline, the Dataflow Gen2 will show up as an artifact. Click on Deploy to Test, and then Deploy to Prod. After deployment, update the Lakehouse or Data Warehouse connections manually in each stage.
Also, please check these links for more information:
The Microsoft Fabric deployment pipelines process - Microsoft Fabric | Microsoft Learn
Dataflow Gen2 with CI/CD and Git integration - Microsoft Fabric | Microsoft Learn
Overview of Fabric deployment pipelines - Microsoft Fabric | Microsoft LearnIf you find this response helpful, please mark it as the accepted solution and give kudos. This will help other community members who may have the same question.
If you have any more questions, feel free to reach out to the Microsoft Fabric community.
Thank you.
Just to add to this answer
Step 4 - every time you re-deploy the dataflow gen2 you will need to re-update the Lakehouse or Data Warehouse connection manually (since it is overwritten by the deployment pipeline).
This wasn't the case with dataflow gen1+datamarts, so hopefully Microsoft will look to streamline or parametrize the outputs in the future.
Hi DiKi-I,
We would like to follow up and see whether the details we shared have resolved your problem.
If you need any more assistance, please feel free to connect with the Microsoft Fabric community.
Thank youHi DiKi-I,
We are checking in to know if the information we provided helped in resolving your issue.
If you have any further questions, please do not hesitate to contact the Microsoft Fabric community.
Thank you.Hi DiKi-I,
We are following up to see if your query has been resolved. Should you have identified a solution, we kindly request you to share it with the community to assist others facing similar issues.
If our response was helpful, please mark it as the accepted solution, as this helps the broader community.
Should you have any further queries, kindly feel free to contact the Microsoft Fabric community.
Thank you.DiKi-I hey,
Kindly refer below link to fulfill your requirement. -Microsoft Fabric: Moving Dataflows Gen 2 to different workspaces - Simple Talk
ThanksHarish KM
Please accept this as a solution if this solves your problem and give kudos as well.
