Forum Discussion

AnkitaSaxena-82's avatar
AnkitaSaxena-82
Frequent Visitor
7 hours ago

Dataflow Gen2 still points to DEV environment after deployment to TEST - Can this be automated?

Hello everyone,

I'm looking for best practices around automating environment promotion for Dataflow Gen2 in Microsoft Fabric.

Current Scenario

I have 4 Dataflow Gen2 pipelines that were developed and tested in our DEV workspace. I recently moved them to our TEST workspace using our deployment process through deployment Pipelines.

After deployment, I noticed that the dataflows are still pointing to the DEV environment rather than automatically switching to the corresponding TEST resources.

What is currently happening?

  • Dataflows were created in DEV.
  • Dataflows were deployed/moved to TEST.
  • In TEST, the dataflows are still referencing DEV resources (source and/or destination).
  • The only workaround I currently see is:
    • Manually removing the destination.
    • Re-adding the TEST destination.
    • Republishing the dataflow.

What I would like to achieve

I would like to automate the environment switching process so that when a Dataflow Gen2 is promoted from DEV to TEST:

  • DEV connections automatically become TEST connections.
  • DEV Lakehouse/Warehouse destinations automatically become TEST destinations.
  • No manual reconfiguration is required after deployment.
  • The process can be managed consistently across multiple dataflows.

Questions

  1. Is there a recommended approach for automating this in Microsoft Fabric?
  2. Are deployment rules, parameters, or environment variables supported for Dataflow Gen2 connections and destinations?
  3. Can this be achieved through:
    • Fabric Deployment Pipelines?
    • Git integration / CI-CD?
    • Fabric REST APIs?
    • VS Code-based configuration management?
  4. How are other teams managing DEV → TEST → PROD promotions for Dataflow Gen2 while avoiding manual destination remapping?

Any guidance, examples, or recommended architecture patterns would be greatly appreciated.

Thanks in advance!

dataflow
General Comment
Interesting Links
Need Help
power bi
Show and Tell
Tips and Tricks
Tutorial Requests

2 Replies

  • benibix's avatar
    benibix
    Icon for Helper V rankHelper V
    6 hours ago

    Hi AnkitaSaxena-82

    Deployment rules won't do this. The docs say they can change some item properties, but not Dataflow connections or the mashup logic. The fix is to make the dataflow stage-neutral before you deploy, and that only works for Dataflow Gen2 with CI/CD, so check your 4 are that type. For a Lakehouse or Warehouse, pick the item under the !(Current Workspace) node in the connector. The script then holds the item name instead of a workspace ID, and it resolves in TEST as long as TEST has items with the same names. For values that change per stage, use a variable library in the same workspace, or public parameters passed from a pipeline Dataflow activity. Two limits. Variables can't change connection information or destination mappings, and connections to other sources stay fixed as authored, so those need their own setup in TEST. I can't see your dataflows, so I don't know which references are absolute right now.

    https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-relative-references

  • ShivekMaharaj's avatar
    ShivekMaharaj
    Icon for Community Champion rankCommunity Champion
    3 hours ago

    Hi AnkitaSaxena-82​,

    There is an important distinction between Dataflow Gen2 sources and destinations when using deployment pipelines.

    For Fabric sources such as Lakehouse or Warehouse, Microsoft now supports relative references. If you select the source through the !(Current Workspace) node when authoring the Dataflow, the query stores the item by name instead of the DEV workspace/item IDs. After deployment, it resolves the same-named item in the TEST workspace automatically.

    Microsoft documents this behaviour in the Dataflow Gen2 relative reference guidance.

    The limitation is the destination side. Microsoft's cross-workspace dependency binding guidance states that Dataflow Gen2 output destinations use absolute references and do not automatically rebind during deployment.

    Deployment rules do not currently solve this either. Microsoft's deployment rules documentation supports rules for items such as semantic models and Dataflow Gen1, but not Dataflow Gen2 connections or destinations.

    Variable libraries are useful for environment-specific values inside the mashup, but Microsoft also documents that they cannot change connection information or destination mappings.

    So for DEV -> TEST -> PROD, I would design it as follows:

    • use !(Current Workspace) for Fabric source references
    • keep matching Lakehouse/Warehouse names across environments where possible
    • use variable libraries for configuration values inside the Dataflow
    • treat destination rebinding as a separate deployment concern because it is not currently automatic


    That means the source side can be made environment-neutral today, but fully automatic destination switching for Dataflow Gen2 is still a limitation you need to design around.

    AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.