CSDataEng's avatar
CSDataEng
Regular Visitor
4 months ago
Status:
Abandoned

Dataflow Gen2 parameter behavior differs between direct execution and Pipeline runs

When using Dataflow Gen2 in Microsoft Fabric, parameters behave differently depending on how the Dataflow is executed, even when the same parameters and values are used. After detailed investigation...
data engineering
Data Factory | Dataflow
fabric platform
miguel's avatar
miguel
Community Admin
3 months ago
Status changed:
New
to
Abandoned

The path that the pipeline uses is actually the one exposed through the REST API:

Background Jobs - Run On Demand Execute - REST API (Dataflow) | Microsoft Learn

 

Whenever you click "Save and run" within the Dataflow UI, you are effectively not passing any parameters override to your execution. You can also check this in the "Recent runs" as to any "Save and run" triggered runs will not show a Parameters section, whereas any execution that does have parameters override, only available through the API which is what Pipelines leverage, will in fact show the section.

 

Whenever you change the parameter value within the actual Dataflow, there are some potential changes that the UI is doing behind the scenes because of this. It's difficult to say if this might be what's impacting the difference that you're seeing as we would need more information about your particular scenario and what expectation you may have.

 

In other words:

- Dataflow triggered within the UI  = No parameters override passed

- Dataflow triggered via REST API with parameters override payload = Parameters override passed

 

The best way to compare the behavior from Pipelines is to use the REST API and send you parameters override payload. The example request from the documentation also outlines the payload for the parameters override:

 

POST https://api.fabric.microsoft.com/v1/workspaces/a0a0a0a0-bbbb-cccc-dddd-e1e1e1e1e1e1/dataflows/dddddddd-9999-0000-1111-eeeeeeeeeeee/jobs/execute/instances

{
  "executionData": {
    "executeOption": "ApplyChangesIfNeeded",
    "parameters": [
      {
        "parameterName": "OrderKey",
        "type": "Automatic",
        "value": 25
      },
      {
        "parameterName": "Threshold",
        "type": "Automatic",
        "value": "start"
      }
    ]
  }
}

 

Given the way that the idea is written, I'll have to decline it as the architecture expressed in the idea is not the actual one that's implemented today as explained above as the Dataflow UI does not pass a parameters override payload.

 

Do feel free to create a forum post so we can go deeper into your scenario and why it might be behaving in a particular way and what expectations you may have. You can also reach out to me directly via a private message and we can go deeper. I believe it might be better for us to focus on the scenario with some details of it and what might be going wrong so we can discover the root cause.

Recent ideas