Forum Discussion

DoesItMatter's avatar
DoesItMatter
Regular Visitor
4 months ago

Fails - pass variables from pipelines as parameters into a Dataflow Gen2

Hi,

 

I am trying to pass pipeline parameters/variables into a dataflow gen2 in Fabric. I could not find a way to have it working. Does anyone have the same issue or have already found a workaround? Below the steps I followed:

  1. Create a dataflow gen2
  2. Enable parameters expose in the dataflow
  3. Define a new parameter in the dataflow (I tried various data types, text, decimal, any)
  4. Save and close the dataflow
  5. Create a pipeline
  6. Define parameters in the pipeline (also defined variables as an alternative)
  7. Add the dataflow to the pipeline
  8. Set the dataflow parameters dynamically from the pipeline parameters (also tried with pipeline variables and variable library in Fabric)
  9. Save and close the pipeline
  10. Reopen the pipeline
  11. BOOM! Dynamically set dataflow parameters in the pipeline are gone, default dataflow parameter values are shown.

When I check the JSON code, I see that the dataflow parameters are set dynamically. After reopening or simply switching between tabs (main, settings, parameters), they disappear.

Various options did not work: CI/CD devops enabled workspace, Fabric library item variables, pipeline parameters, pipeline variables, set variable activity.
Has anyone managed to pass pipeline parameters/variables into a dataflow gen2 ? 

Need Help
pipelines

9 Replies

  • Tamanchu's avatar
    Tamanchu
    Icon for Super User rankSuper User
    4 months ago

    Hi DoesItMatter,

    This should be supported, but only with Dataflow Gen2 with CI/CD support and with public parameter mode enabled.

    In the Dataflow, make sure the option "Enable parameters to be discovered and override for execution" is enabled. Then, in the pipeline Dataflow activity, you should be able to pass values in the Dataflow parameters section.

    That said, the behavior you describe sounds more like a pipeline UI serialization bug than a parameter design issue. If the dynamic expression is visible in the JSON but disappears after reopening the pipeline or switching tabs, the activity JSON may be saved correctly at first, but the UI is overwriting or rehydrating it back to the default parameter values.

    A few things I would try :

    • Save the pipeline after editing the JSON and run it without reopening the activity UI.
    • Check whether the pipeline run actually uses the dynamic value, even if the UI later shows the default value.
    • Make sure the activity is using the Fabric Dataflow type, not an older/legacy dataflow activity definition.
    • Try with a simple text parameter first, for example @pipeline().parameters.MyParam.
    • If the JSON keeps reverting, I would raise it as a Fabric support issue and include the before/after pipeline JSON.

    So yes, the scenario is expected to work, but your BOOM, values disappear behavior really looks like a product/UI bug.

    Useful docs :

     

    Hope this helps narrow it down. If this helped, feel free to mark it as a solution so it can help others facing the same issue.

     

    Note: This response was enhanced with the assistance of an AI tool. The information has been verified against the official Microsoft Fabric documentation.

    • DoesItMatter's avatar
      DoesItMatter
      Regular Visitor
      4 months ago

      Thanks for your answer, but it does not sound you read my steps, more of a human completed AI generated text.

      • 1st step tells it is gen2.
      • 2nd step tells parameters are exposed.
      • 3rd step tells text, and other types were tried.

      Paramter set expression is visible when the pipeline is saved. However, it disappears:

      • when pipeline opened
      • when pipeline saved but not closed, simply switching between the dataflow parameter set tabs clears the parameter set from the json code.

      Pipeline save and run: same issue, no parameters passed.

       

      I think there are multiple bugs in the entire functionality, does not seem tested properly.

  • Soaps's avatar
    Soaps
    Regular Visitor
    4 months ago

    I have the same problem. I have had data pipelines running dataflows (gen2 ci/cd) with parameters for a long time now, but in the last couple of weeks it just stopped working. First it said that it could not find the parameters and I had to manually add them every time I wanted to run the pipeline, since they disappeared every time the pipeline was closed. 
    Now it finds the parameters from the dataflow but the correct values still disappear every time the pipeline is closed. I sure hope it will be fixed soon because this is really frustrating to me and my users.