Issue with parameterised Dataflow Gen2 activity

[Anonymous]

Problem Statement:

A published Dataflow Gen2 that contains parameters is not exposing those parameters to the Dataflow Gen2 activity within a Fabric Pipeline. The pipeline editor shows 'This dataflow has no parameters', even though they are defined and saved in the Dataflow.

Error Message When Pipeline is Run:

There was a problem refreshing the dataflow: 'Refresh with parameters is not supported for non-parametric dataflows'. Error code: ParametersNotSupported.

Troubleshooting Steps Already Performed:

• The Dataflow (df_Update_KQL_Watermark) runs successfully when refreshed manually from the Dataflow editor itself. The manual run correctly writes data to the KQL database.
• The Dataflow has been repeatedly saved and published after the parameters (pPipelineName, pNewWatermarkValue) were added.
• We have waited for significant time (more than 5 minutes) after saving to account for any service latency.
• The Dataflow Gen2 activity was deleted from the pipeline and a brand new one was created. The new activity still fails to detect the parameters when the Dataflow is selected.

Has anyone faced this issue?

[DoesItMatter]

Fabric pipelines simply do not save the changes when dataflow parameters are set. CI/CD enabled or not it is still not working. Also, no solution with Library variables.

[DoesItMatter]

Over a year, the issue is still there. It does not work with CI/CD enabled pipelines. Is there any ambition or timeline to fix this?

[anon97242]

When you specify your DataflowGen2 with parameters  in the Pipeline Dataflow Activity. you have to manually add the parameters. They do not automatically appear unfortunately.

[v-aatheeque · Accepted solution]

Hi @Anonymous ,
@anon97242 @wardy912  Thanks for your quick response , In addition to that ,

• Parameters in Dataflow Gen2 are part of a preview feature called public parameters. This mode only works with dataflows that have CI/CD support enabled. If your dataflow isn’t set up this way, the parameters won’t be exposed to the pipeline.

Reference : https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-parameters

Hope this helps!!

If this post was helpful, please consider marking Accept as solution to assist other members in finding it more easily.

If you continue to face issues, feel free to reach out to us for further assistance!

[wardy912]

I have faced this issue before and couldn't fix unfortunately, but as a workaround I used a notebook and I've never looked back.
With notebooks you can obtain a much more granular and deliberate set of data transformations.
You can also add base parameters to the notebook pipeline item which forces the pipeline to use it once referenced in your notebook, this is not available in dataflow gen2.
If notebooks are an option for you, I'd take that approach to avoid this issue.

To troubleshoot your dataflow gen2, even though your parameters are defined, make sure that the parameters are referenced in at least one of the queries, without this the pipeline may ignore it.