Forum Discussion
Using DataflowGen2 CI/CD parameterized, parallel execution, with pipelines
Internally, Dataflow Gen2 jobs may hit compute or metadata contention when too many parallel refreshes are triggered in a short interval. The Dataflow execution engine sometimes fails to forward exceptions back to the parent pipeline. Also the Fabric UI does not yet provide robust debugging/logging for concurrent Dataflow executions. Pls note that dataflow Gen2 is still evolving support for robust parameter handling in parallel executions, especially when resource reuse or contention is present.
As a temp workaround try:
- Switch to sequential execution for critical paths until Microsoft enhances Dataflow Gen2's support for robust parallel execution.
- Alternatively, use Notebook-based logic (Fabric Notebooks) for parallel operations where concurrency is more stable and transparent.
But always it is ideal to follow the best practices as listed below:
- Throttle parallelism: Instead of completely disabling parallelism, set Batch count = 2 and introduce a Wait activity (2–5 seconds) between executions to reduce race conditions.
- Isolate Dataflows per iteration: If possible, clone the dataflow for testing and assign different names for each execution path to test whether the issue is caused by shared state or metadata locks.
- Enable verbose logging (if possible via REST API): Capture logs at the workspace or Dataflow job level to see if backend logs show failures.
- Log Feedback to Microsoft (with Session ID)
Use the “Feedback” button inside the Fabric UI and submit:- Session ID from your screenshot: b087d12b-cdc8-4a07-8a3a-33a6c9f34504
- Pipeline run ID and workspace details
- Avoid Parameter Binding in Highly Parallel Jobs: Parameters in Dataflow Gen2 often get lost or mismatched when triggered concurrently. For now, externalize transformations to Notebooks or Spark job definitions if possible.
Please Kudos & 'Accept' as solution if the reply was helpful. This will be benefitting other community members who face the same issue.
Internally, Dataflow Gen2 jobs may hit compute or metadata contention when too many parallel refreshes are triggered in a short interval. The Dataflow execution engine sometimes fails to forward exceptions back to the parent pipeline. Also the Fabric UI does not yet provide robust debugging/logging for concurrent Dataflow executions. Pls note that dataflow Gen2 is still evolving support for robust parameter handling in parallel executions, especially when resource reuse or contention is present.
As a temp workaround try:
- Switch to sequential execution for critical paths until Microsoft enhances Dataflow Gen2's support for robust parallel execution.
- Alternatively, use Notebook-based logic (Fabric Notebooks) for parallel operations where concurrency is more stable and transparent.
But always it is ideal to follow the best practices as listed below:
- Throttle parallelism: Instead of completely disabling parallelism, set Batch count = 2 and introduce a Wait activity (2–5 seconds) between executions to reduce race conditions.
- Isolate Dataflows per iteration: If possible, clone the dataflow for testing and assign different names for each execution path to test whether the issue is caused by shared state or metadata locks.
- Enable verbose logging (if possible via REST API): Capture logs at the workspace or Dataflow job level to see if backend logs show failures.
- Log Feedback to Microsoft (with Session ID)
Use the “Feedback” button inside the Fabric UI and submit:- Session ID from your screenshot: b087d12b-cdc8-4a07-8a3a-33a6c9f34504
- Pipeline run ID and workspace details
- Avoid Parameter Binding in Highly Parallel Jobs: Parameters in Dataflow Gen2 often get lost or mismatched when triggered concurrently. For now, externalize transformations to Notebooks or Spark job definitions if possible.
Please Kudos & 'Accept' as solution if the reply was helpful. This will be benefitting other community members who face the same issue.
Thank you Vinodh247 for the detailed response!
For others potentially running into this issue:
Throttle parallelism: Instead of completely disabling parallelism, set Batch count = 2 and introduce a Wait activity (2–5 seconds) between executions to reduce race conditions. I experimented with differernt bath sizes and wait times (even using rand() to randomize dataflow kick offs) while this did improve the issues, it never resolved it.
Isolate Dataflows per iteration: If possible, clone the dataflow for testing and assign different names for each execution path to test whether the issue is caused by shared state or metadata locks. I am actually doing this in another project and it works well, the issue is there is limitation on # of workspace items and it becomes extremely onerous when there are more than an handful of iterations.
Avoid Parameter Binding in Highly Parallel Jobs: Parameters in Dataflow Gen2 often get lost or mismatched when triggered concurrently. For now, externalize transformations to Notebooks or Spark job definitions if possible. This is unfortunate, hopefully a more robust and performant option is availble via DataflowGen2 in the future.
Will be sure to provide feedback with fabric, was not aware of that option.
Thanks again!