Forum Discussion
Pipeline source control issue: Dataflow Gen2 pipelines overwrite dataflowId with logicalId
Hi, I’m experiencing an issue with Dataflow Gen2 and source control.
In short, the pipeline commits typeProperties with the Dataflow’s logicalId instead of the dataflowId.
Also, workspaceId is set to a placeholder value of "00000000-0000-0000-0000-000000000000".
Steps to reproduce:
1. Create a Dataflow Gen2.
2. In a pipeline, add a Dataflow activity and select the Dataflow created in the previous step.
3. Commit changes to source control.
4. Deploy changes to another workspace.
5. Try to run the pipeline - you’ll get an UnknownException.
6. Manually update workspaceId and dataflowId in pipeline-content.json to the correct values. Now the pipeline works fine.
7. Commit the corrected values to source control.
8. After this commit, Fabric forces another commit, again replacing the dataflowId with logicalId and resetting the workspaceId to the placeholder value.
(Note: logicalId comes from the Dataflow’s .platform file.)
This issue messes up my deployment process.
Is there any known workaround, or do you know when this issue might be fixed?
7 Replies
Did you try the RESTAPI? The Dataflow Gen2 pipeline activity doesn't support parameterized or dynamic inputs for workspaceId and dataflowId. It expects selections via dropdown only. If you manually edit those IDs (even with correct ones), Fabric will override them with the logicalId and zeroed workspaceId on the next commit, garbage in, garbage out. That’s by design, not a glitch.
Re: Unable to run Dataflow Gen 2 (CI/CD) by giving... - Microsoft Fabric Community
No timeline or ETA on a fix, this hasn’t acknowledged as urgent. So far, no roadmap or release notes hint that parameterized Dataflow activities for Gen2 will work any differently soon.
Please 'Kudos' and 'Accept as Solution' if this answered your query.
I worked around the issue by using Notebooks instead of Dataflow Gen2.
It’s disappointing to hear this. It basically means Dataflows are unusable in multi-environment scenarios. In their current state, I can’t rely on them.
Hi DarSz ,
Thank you for reaching out to the Microsoft fabric community forum. We understand the challenge you’re facing this behavior significantly impacts CI/CD workflows across multiple environments.
At present, Dataflow Gen2 pipeline activities do not support parameterized or dynamic values for workspaceId and dataflowId. When changes are committed to source control, Fabric replaces these values with the logicalId and a placeholder workspace ID by design. This limits the ability to use Dataflows reliably in multi-environment CI/CD deployments.
Your workaround of using Notebooks is a valid approach, and many customers adopt this path for scenarios where Dataflow Gen2 is not yet fully supported. Currently, there is no published ETA or roadmap commitment on enabling parameterized deployment of Dataflow Gen2 pipeline activities as mentioned by Vinodh247 .
As a further steps, I’d recommend:
- Submitting and upvoting feedback on the Fabric Ideas - Microsoft Fabric Communityso that this scenario can be prioritized.
- Tracking the Microsoft Fabric Roadmap for updates on Dataflow Gen2 pipeline activity improvements.
Dataflow Gen2 activities may not fit well in CI/CD processes today, and Notebooks or alternative orchestration methods are the recommended option until more flexible deployment support is introduced.
Hope this helps, please feeel free to reach out for any further questions.
Thank you.
Hi DarSz ,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.
Thank you.
Hi DarSz ,
Could you please confirm if you've submitted this as an idea in the Ideas Forum? If so, sharing the link here would be helpful for other community members who may have similar feedback.
For any further discussions or questions, please reach out to fabric community forum.
Thank you.
Sorry, got distracted with other projects.
I submitted the idea here:
https://community.fabric.microsoft.com/t5/Fabric-Ideas/Dataflow-Gen2-on-multi-environment-scenarios/idi-p/4824498#M163682Hi DarSz ,
Thank you for letting us know. Hopefully, Microsoft will include this in future updates. If you have any further questions, please feel free to reach out we're always here to help.
