Forum Discussion
Automating Dataflow Gen2 destination changes in deployment pipelines
Thankyou, blopez11, for your response.
Hi mustaphaben,Thank you for reaching out via the Microsoft Fabric Community Forum.
Based on my understanding, Dataflow Gen2 destinations such as Lakehouse or workspace IDs are currently stored within the dataflow definition (mashup.pq) and are not supported by deployment rules. Therefore, after deploying from DEV to PROD, the dataflow continues to point to the DEV Lakehouse unless the destination is updated manually.
Please consider the following approaches that may help resolve the issue:
- Store the target workspaceId and lakehouseId in a Variable Library and reference these variables in the dataflow using Variable.ValueOrDefault("$(/**/LibraryName/VariableName)"). Assign different values for DEV and PROD. During deployment, the correct Lakehouse will be applied automatically.
- After deployment, call the Get Definition API to download the dataflow definition. Replace the DEV workspaceId or lakehouseId with the PROD values, and then push the changes back using the Update Definition API.
Please refer to the links below for additional information:
Use public parameters in Dataflow Gen2 (Preview) - Microsoft Fabric | Microsoft Learn
Use Fabric variable libraries in Dataflow Gen2 (Preview) - Microsoft Fabric | Microsoft Learn
Items - Get Dataflow Definition - REST API (Dataflow) | Microsoft Learn
Items - Update Dataflow Definition - REST API (Dataflow) | Microsoft Learn
Dataflow definition - Microsoft Fabric REST APIs | Microsoft LearnWe hope the above information helps to resolve the issue. If you have any further queries, please feel free to contact the Microsoft Fabric Community.
Thank you.
Hi mustaphaben,
Thank you for the follow up.
Based on my understanding, since Fabric deployment pipelines currently do not automatically switch Dataflow Gen2 destinations from the DEV Lakehouse to the PROD Lakehouse, full automation in Azure DevOps can be achieved by using one of the following methods:
- Install the Fabric CLI in the Azure DevOps pipeline and use the Fabric CI/CD toolkit to publish items from the Git repository to the target workspace, with parameter substitution for PROD values.
- Use the Microsoft Fabric Terraform provider in the pipeline to declaratively provision and manage workspaces, dataflows, and destinations. Execute terraform plan and terraform apply as part of the release so that PROD dataflows automatically point to the correct Lakehouse.
Both approaches eliminate the need for manual edits and enable integration of Fabric deployment into a standard CI/CD process.
For further reference, please see the following links:
fabric-cli
GitHub - microsoft/fabric-cicd: Jumpstart CICD deployments in Microsoft Fabric
Terraform Provider for Microsoft Fabric (Generally Available) | Microsoft Fabric Blog | Microsoft FabricWe hope that the information provided will help to resolve the issue. Should you have any further queries, please feel free to contact the Microsoft Fabric community.
Thank you.
Hi mustaphaben,
We have faced the exact same challenge when moving from DEV → PROD using Deployment Pipelines with Dataflows Gen2.
Here’s what we found:
Deployment rules today support notebooks, pipelines, semantic models, and connections — but not Dataflows Gen2.
The source part can be parameterized in Power Query M (using query parameters), which lets you switch between DEV/PROD data sources automatically.
The destination (sink) of a Dataflow Gen2 is not currently configurable via deployment rules, so after deployment it still writes to the DEV Lakehouse.
What we did as a workaround:
We manually update the destination Lakehouse in PROD after each deployment,
orWe keep Dataflows Gen2 in a separate workspace that is not included in the deployment pipeline, to avoid overwriting PROD configuration with DEV configuration.
Hope it will be added in the next updates, since it’s a common need when using Fabric at scale with multiple environments.
Best regards,
Antoine