Forum Discussion
Fails - pass variables from pipelines as parameters into a Dataflow Gen2
Super UserHi DoesItMatter,
This should be supported, but only with Dataflow Gen2 with CI/CD support and with public parameter mode enabled.
In the Dataflow, make sure the option "Enable parameters to be discovered and override for execution" is enabled. Then, in the pipeline Dataflow activity, you should be able to pass values in the Dataflow parameters section.
That said, the behavior you describe sounds more like a pipeline UI serialization bug than a parameter design issue. If the dynamic expression is visible in the JSON but disappears after reopening the pipeline or switching tabs, the activity JSON may be saved correctly at first, but the UI is overwriting or rehydrating it back to the default parameter values.
A few things I would try :
- Save the pipeline after editing the JSON and run it without reopening the activity UI.
- Check whether the pipeline run actually uses the dynamic value, even if the UI later shows the default value.
- Make sure the activity is using the Fabric Dataflow type, not an older/legacy dataflow activity definition.
- Try with a simple text parameter first, for example @pipeline().parameters.MyParam.
- If the JSON keeps reverting, I would raise it as a Fabric support issue and include the before/after pipeline JSON.
So yes, the scenario is expected to work, but your BOOM, values disappear behavior really looks like a product/UI bug.
Useful docs :
- https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-parameters
- https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-activity
- https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-parameterized-dataflow
Hope this helps narrow it down. If this helped, feel free to mark it as a solution so it can help others facing the same issue.
Note: This response was enhanced with the assistance of an AI tool. The information has been verified against the official Microsoft Fabric documentation.
Thanks for your answer, but it does not sound you read my steps, more of a human completed AI generated text.
- 1st step tells it is gen2.
- 2nd step tells parameters are exposed.
- 3rd step tells text, and other types were tried.
Paramter set expression is visible when the pipeline is saved. However, it disappears:
- when pipeline opened
- when pipeline saved but not closed, simply switching between the dataflow parameter set tabs clears the parameter set from the json code.
Pipeline save and run: same issue, no parameters passed.
I think there are multiple bugs in the entire functionality, does not seem tested properly.
- tayloramy
Hi DoesItMatter,
Would Variable Libraries be able to meet your needs? I have successfully referenced variables from variable libraries inside dataflow gen 2 before. Microsoft has some documentation on that here: https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-variable-library-integration
No, I already mentioned that it is not working. I highly appreaciate if you could try before commenting. This way the community can avoid non-value added and untested comments.
I created a support ticket and Microsoft verified that the entire functionality contains bugs. A fix is expected somewhere in July.
- Tamanchu
I completely understand the frustration this kind of thing is really draining when you've done everything right and it still doesn't work. I took the time to read through your problem properly, and you've clearly tested it thoroughly. Opening the ticket with Microsoft was the right move that's genuinely the best way to get this looked at and confirmed.
