Deployment pipelines - Dataflows in deployed workspaces output to dev

Hi dolphinantonym,

When deploying Dataflows Gen2 using deployment pipelines, the dataflows keep their original connection and destination settings. As a result, even after moving from Dev to Test or Prod, they continue to write to the Lakehouse in the Dev workspace. Currently, Fabric does not provide an automated way to update all dataflow outputs in bulk during deployment.

To handle this, you can use Variable Libraries to maintain environment-specific configurations. Each stage in the pipeline (Dev, Test, Prod) can have its own active value set in the same variable library. These values let you reference environment-specific parameters—like workspace IDs, Lakehouse names, or table identifiers—without manual changes after each deployment. During runtime, the dataflow uses the active value set for the environment, ensuring the correct configuration is applied.

If you use Git-based source control, Variable Libraries are Git-enabled and versioned with your dataflows. When you create a workspace from a feature branch, you can select the active value set for that branch or environment, allowing flexible testing and management of configurations.

In the dataflow’s mashup.pq file, you can reference variables using Power Query functions like Variable.Value() or Variable.ValueOrDefault(). For example, Variable.Value("$(/**/My Library/Workspace ID)") gets the workspace ID dynamically. However, Variable Libraries can only control parameters used inside the query logic and cannot rebind the output sink set in the dataflow UI.

For larger environments with many dataflows, you can use Fabric REST APIs to automate rebinding. A script can loop through dataflows in the target workspace and update their destination properties, providing scalable automation to ensure deployed dataflows write to the correct Lakehouses after deployment.

Lifecycle Management of the Microsoft Fabric Variable library - Microsoft Fabric | Microsoft Learn
Use Fabric variable libraries in Dataflow Gen2 (Preview) - Microsoft Fabric | Microsoft Learn

Thank you.