Forum Discussion
Deploy dataflow gen2 using deployment pipeline
- 1 year ago
Thankyou, MasonMA, for your response.
Hi DiKi-I,We appreciate your question posted on the Microsoft Fabric Community Forum.
From what I understand, right now Dataflow Gen2 does not automatically update its Lakehouse or Data Warehouse connections when you deploy it. This creates some difficulties while moving across different environments because you need to change the connections manually.
Please follow these steps that might help solve the problem:
- Create three separate Fabric workspaces named Dev, Test, and Prod.
- Go to Deployment Pipelines in Fabric. Create a new pipeline and assign the Dev workspace to Stage 1. Then link the Test and Prod workspaces to the next stages.
- After setting up the pipeline, the Dataflow Gen2 will show up as an artifact. Click on Deploy to Test, and then Deploy to Prod. After deployment, update the Lakehouse or Data Warehouse connections manually in each stage.
Also, please check these links for more information:
The Microsoft Fabric deployment pipelines process - Microsoft Fabric | Microsoft Learn
Dataflow Gen2 with CI/CD and Git integration - Microsoft Fabric | Microsoft Learn
Overview of Fabric deployment pipelines - Microsoft Fabric | Microsoft LearnIf you find this response helpful, please mark it as the accepted solution and give kudos. This will help other community members who may have the same question.
If you have any more questions, feel free to reach out to the Microsoft Fabric community.
Thank you.
Thankyou, MasonMA, for your response.
Hi DiKi-I,
We appreciate your question posted on the Microsoft Fabric Community Forum.
From what I understand, right now Dataflow Gen2 does not automatically update its Lakehouse or Data Warehouse connections when you deploy it. This creates some difficulties while moving across different environments because you need to change the connections manually.
Please follow these steps that might help solve the problem:
- Create three separate Fabric workspaces named Dev, Test, and Prod.
- Go to Deployment Pipelines in Fabric. Create a new pipeline and assign the Dev workspace to Stage 1. Then link the Test and Prod workspaces to the next stages.
- After setting up the pipeline, the Dataflow Gen2 will show up as an artifact. Click on Deploy to Test, and then Deploy to Prod. After deployment, update the Lakehouse or Data Warehouse connections manually in each stage.
Also, please check these links for more information:
The Microsoft Fabric deployment pipelines process - Microsoft Fabric | Microsoft Learn
Dataflow Gen2 with CI/CD and Git integration - Microsoft Fabric | Microsoft Learn
Overview of Fabric deployment pipelines - Microsoft Fabric | Microsoft Learn
If you find this response helpful, please mark it as the accepted solution and give kudos. This will help other community members who may have the same question.
If you have any more questions, feel free to reach out to the Microsoft Fabric community.
Thank you.
- mike99991 year agoAdvocate I
Just to add to this answer
Step 4 - every time you re-deploy the dataflow gen2 you will need to re-update the Lakehouse or Data Warehouse connection manually (since it is overwritten by the deployment pipeline).
This wasn't the case with dataflow gen1+datamarts, so hopefully Microsoft will look to streamline or parametrize the outputs in the future.