Please enable Dataflow Gen2 destination parametrization.
We would like to parametrize the path/table in destination Lakehouse.
11 Comments
This is a huge issue for us. We are rolling out dataflow gen 2 to over 40 workspaces. Each workflow has a SQL source with 41 tables, which we have those connections parameterized. We are loading these into a lakehouse. We use the dataflow Gen 2 template to deploy these and we just have to set the parameters and the source is taken care of. The destination is a different story. We have to go into all 41 tables and set the destination to the workspace and lakehouse that we need for this workspace. Please add this feature soon!!!!
- Status added:Needs Votes
- Dear This issue has been reported back in 2023. I understand a recent development allows to pass data destination as a pipeline parameter, ideally it is a deployment parameter. We can see Dataflows everywhere on Fabric slides, yet they are seemingly impossible to deploy via the Workspace Deployment Pipelines. Only works with ADO and Git. This is ridiculous. I've already switched my whole environment to notebooks. I still want to leverage dataflows though, for the correct use case. I don't understand how this is not a known issue or development. And why this idea is not sprawling with votes.
This is sorely needed and should be feasible to implement now that public parameters and CI/CD in Dataflows gen2 are becoming GA as of Q2 2025. Please include functionality for parameterized DF destinations so that we can have finally an end-to-end automated Deployment Pipelines solution for DF.
- Status changed:Needs VotestoCompleted
This is currently available and we have some guides to help you with how to use parameters or variables to dynamically pass values to use in your data destination:
CI/CD and ALM solution architectures for Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
- I don't consider this issue COMPLETED. The only is a workaround editing the dataflow source using git. That's not what I would call a solution. Please reopen this issue.
GertjanIdema do feel free to open a new idea that better describes the functionality that you'd like to see added.
From a functional standpoint, we added support for parameterization and for such evaluations to happen and be fully supported by the Dataflow engine via public parameters and/or Fabric variable libraries.
- I feel the same. It can not be assumed that every Fabric user uses git and therefore this is a limited workaround. GertjanIdema, this idea specifically mentions that the parametrization shall be in the UI, feel free to add your vote: https://community.fabric.microsoft.com/t5/Fabric-Ideas/Fabric-gt-Dataflow-Gen-2-gt-Destination-Allow-this-to-be-a/idi-p/5318667
- Mauro89, For me, this problem was solved by the introduction of the !(Current workspace) feature. (Workspace ID = "."). I keep my related dataflows and lakehouses (Bronze, Silver, Gold) in one workspace and use !(Current workspace) everywhere in the Gen2 dataflows for source and destination. Implementation from Development to Acceptance to Production in the implementation pipeline now works without a problem and I never need to make changes in git anymore.
Mauro89 completely agree with you. There shouldn't be a need to use git at all and that's why we introduced the "advanced editor" for data destinations which can be found here:
https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-advanced-edit-data-destinations
The full end to end guide for using parameters in Dataflow can be found here:
https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-parameterized-dataflow
hope this helps!
