Forum Discussion
Dataflow Gen2 still points to DEV environment after deployment to TEST - Can this be automated?
Hello everyone,
I'm looking for best practices around automating environment promotion for Dataflow Gen2 in Microsoft Fabric.
Current Scenario
I have 4 Dataflow Gen2 pipelines that were developed and tested in our DEV workspace. I recently moved them to our TEST workspace using our deployment process through deployment Pipelines.
After deployment, I noticed that the dataflows are still pointing to the DEV environment rather than automatically switching to the corresponding TEST resources.
What is currently happening?
- Dataflows were created in DEV.
- Dataflows were deployed/moved to TEST.
- In TEST, the dataflows are still referencing DEV resources (source and/or destination).
- The only workaround I currently see is:
- Manually removing the destination.
- Re-adding the TEST destination.
- Republishing the dataflow.
What I would like to achieve
I would like to automate the environment switching process so that when a Dataflow Gen2 is promoted from DEV to TEST:
- DEV connections automatically become TEST connections.
- DEV Lakehouse/Warehouse destinations automatically become TEST destinations.
- No manual reconfiguration is required after deployment.
- The process can be managed consistently across multiple dataflows.
Questions
- Is there a recommended approach for automating this in Microsoft Fabric?
- Are deployment rules, parameters, or environment variables supported for Dataflow Gen2 connections and destinations?
- Can this be achieved through:
- Fabric Deployment Pipelines?
- Git integration / CI-CD?
- Fabric REST APIs?
- VS Code-based configuration management?
- How are other teams managing DEV → TEST → PROD promotions for Dataflow Gen2 while avoiding manual destination remapping?
Any guidance, examples, or recommended architecture patterns would be greatly appreciated.
Thanks in advance!
1 Reply
- Adityakumar
Helper II
This is generally related to how Dataflow Gen2 handles source and destination references across deployment stages.
Deploying a Dataflow Gen2 from Dev to Test does not automatically guarantee that all its source and destination references will point to the Test environment.
Microsoft currently recommends the following approaches for Dataflow Gen2 CI/CD:
Use Public Parameters
Parameterize environment-dependent references such as Workspace IDs, Lakehouse IDs, and Warehouse IDs. Pass the appropriate values when executing the Dataflow through a Fabric Pipeline's Dataflow activity.
Use Variable Libraries
Maintain environment-specific values in a Variable Library and reference them in your Dataflow so the appropriate values are used at runtime.
Use Relative References
Where supported, use relative references with Fabric connectors instead of hardcoded workspace and item GUIDs.
Important limitation: Dataflow Gen2 does not support dynamically reconfiguring external data source connections using parameters or workspace variables. Deployment rules also cannot modify Dataflow Gen2 connections or mashup logic.
My recommendation: Review the Dataflow's Power Query M code and identify any hardcoded Dev workspace or item IDs. For Fabric Lakehouse and Warehouse references, consider parameterizing these IDs and passing the Test or Production values through your orchestration pipeline.
Also, after deploying a Dataflow Gen2 with CI/CD support, Microsoft currently recommends opening and saving the deployed Dataflow to trigger its background publishing process before refreshing it.
Relevant Microsoft Learn documentation:
CI/CD and ALM solution architectures for Dataflow Gen2
Dataflow Gen2 with CI/CD and Git integration
Use public parameters in Dataflow Gen2
If this resolves your issue, please mark the response as Accepted Solution to help other community members facing the same challenge.
