Dataflow Gen2 parameter behavior differs between direct execution and Pipeline runs

[CSDataEng]

When using Dataflow Gen2 in Microsoft Fabric, parameters behave differently depending on how the Dataflow is executed, even when the same parameters and values are used.

After detailed investigation and confirmation with Microsoft Support, we observed the following inconsistent behavior:

• Direct Dataflow execution (via Dataflow UI):
  When a query or query-related logic is passed as a Dataflow parameter, the Dataflow correctly re-evaluates the query and adapts the schema accordingly.

• Pipeline-triggered Dataflow execution:
  When the same parameter is passed from a Pipeline to the Dataflow, the schema is not re-evaluated or adapted, even though the parameter values are identical.

Internally, the Pipeline appears to execute the Dataflow using a different execution path / mashup handling mechanism than a direct Dataflow run. This leads to schema mismatches, mashup document errors, or an inability to support dynamic query logic when triggered via a Pipeline.

Why This Is a Problem

• From a user perspective, Dataflow parameters should behave consistently, regardless of whether the Dataflow is executed directly or via a Pipeline.
• Currently, solutions that work perfectly in a direct Dataflow run break when orchestrated via Pipelines, even though Pipelines are the recommended orchestration mechanism in Fabric.
• This limits the ability to build generic, parameter-driven Dataflows that are orchestrated by Pipelines.
• It also creates confusion, as there is no clear documentation explaining that schema changes are only supported in direct Dataflow runs but not in Pipeline-triggered runs.

Expected Behavior

• Parameter handling and schema evaluation should be harmonized between:
  • Direct Dataflow Gen2 runs
  • Pipeline-triggered Dataflow Gen2 runs
• If a Dataflow supports schema adaptation based on parameters, this behavior should be identical regardless of the execution method.
• Alternatively, if there are technical limitations, these should be explicitly documented, including guidance on supported design patterns.

[miguel]

The path that the pipeline uses is actually the one exposed through the REST API:

Background Jobs - Run On Demand Execute - REST API (Dataflow) | Microsoft Learn

Whenever you click "Save and run" within the Dataflow UI, you are effectively not passing any parameters override to your execution. You can also check this in the "Recent runs" as to any "Save and run" triggered runs will not show a Parameters section, whereas any execution that does have parameters override, only available through the API which is what Pipelines leverage, will in fact show the section.

Whenever you change the parameter value within the actual Dataflow, there are some potential changes that the UI is doing behind the scenes because of this. It's difficult to say if this might be what's impacting the difference that you're seeing as we would need more information about your particular scenario and what expectation you may have.

In other words:

- Dataflow triggered within the UI  = No parameters override passed

- Dataflow triggered via REST API with parameters override payload = Parameters override passed

The best way to compare the behavior from Pipelines is to use the REST API and send you parameters override payload. The example request from the documentation also outlines the payload for the parameters override:

POST https://api.fabric.microsoft.com/v1/workspaces/a0a0a0a0-bbbb-cccc-dddd-e1e1e1e1e1e1/dataflows/dddddddd-9999-0000-1111-eeeeeeeeeeee/jobs/execute/instances

{
  "executionData": {
    "executeOption": "ApplyChangesIfNeeded",
    "parameters": [
      {
        "parameterName": "OrderKey",
        "type": "Automatic",
        "value": 25
      },
      {
        "parameterName": "Threshold",
        "type": "Automatic",
        "value": "start"
      }
    ]
  }
}

Given the way that the idea is written, I'll have to decline it as the architecture expressed in the idea is not the actual one that's implemented today as explained above as the Dataflow UI does not pass a parameters override payload.

Do feel free to create a forum post so we can go deeper into your scenario and why it might be behaving in a particular way and what expectations you may have. You can also reach out to me directly via a private message and we can go deeper. I believe it might be better for us to focus on the scenario with some details of it and what might be going wrong so we can discover the root cause.