Forum Discussion
Poor parallel processing problem in pipelines (aka P5)
Hi v-kpoloju-msft
The update is from Microsoft:
https://learn.microsoft.com/en-us/azure/data-factory/pipeline-trigger-troubleshoot-guide#degree-of-parallelism--increase-doesnt-result-in-higher-throughput
The problem is that this is an obvious bug and they choose not to fix it, despite the fact that customers have struggled for years:Any concurrent or threaded programming language nowadays will allow tasks to be re-balanced while processing is underway. Customers of ADF will expect it to perform the dynamic load-balancing, especially given the excessive cost of the underlying compute and the underlying network components.
The workarounds can often be complex, and involve predicting how long something will take to run, before you run it. This prediction is not always accurate, and working on that prediction can take even more programming effort than the work that is done inside the loop.
Hi dbeavon3,
Since we haven't heard back from you, we wanted to follow up regarding your ticket.
Could you please provide an update on the status of your ticket ? it will be helpful for other members of the community who have similar problems as yours to solve it faster.
Thankyou.
Hi v-kpoloju-msft
The update is from Microsoft:
https://learn.microsoft.com/en-us/azure/data-factory/pipeline-trigger-troubleshoot-guide#degree-of-parallelism--increase-doesnt-result-in-higher-throughput
The problem is that this is an obvious bug and they choose not to fix it, despite the fact that customers have struggled for years:
Any concurrent or threaded programming language nowadays will allow tasks to be re-balanced while processing is underway. Customers of ADF will expect it to perform the dynamic load-balancing, especially given the excessive cost of the underlying compute and the underlying network components.
The workarounds can often be complex, and involve predicting how long something will take to run, before you run it. This prediction is not always accurate, and working on that prediction can take even more programming effort than the work that is done inside the loop.
Hi dbeavon3,
We apologize for the inconvenience. Unfortunately, we do not have an immediate solution currently. However, we will escalate this issue to our internal team to gather insights from various perspectives and resolve it as soon as possible.
Thank you.
Hi dbeavon3,
If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community. If we don’t hear back, we’ll go ahead and close this thread.
Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.
Thank you for your understanding and participation.