Forum Discussion
Poor parallel processing problem in pipelines (aka P5)
Hi v-kpoloju-msft
The update is from Microsoft:
https://learn.microsoft.com/en-us/azure/data-factory/pipeline-trigger-troubleshoot-guide#degree-of-parallelism--increase-doesnt-result-in-higher-throughput
The problem is that this is an obvious bug and they choose not to fix it, despite the fact that customers have struggled for years:Any concurrent or threaded programming language nowadays will allow tasks to be re-balanced while processing is underway. Customers of ADF will expect it to perform the dynamic load-balancing, especially given the excessive cost of the underlying compute and the underlying network components.
The workarounds can often be complex, and involve predicting how long something will take to run, before you run it. This prediction is not always accurate, and working on that prediction can take even more programming effort than the work that is done inside the loop.
Hi dbeavon3,
We regret the inconvenience you are experiencing and acknowledge your requirements. However, we are unable to raise the support ticket on your behalf.
Kindly submit the support ticket using the link provided below.
https://learn.microsoft.com/en-us/power-bi/support/create-support-ticket
Thank you for your understanding.
Hi dbeavon3,
Since we haven't heard back from you, we wanted to follow up regarding your ticket.
Could you please provide an update on the status of your ticket ? it will be helpful for other members of the community who have similar problems as yours to solve it faster.
Thankyou.
Hi v-kpoloju-msft
The update is from Microsoft:
https://learn.microsoft.com/en-us/azure/data-factory/pipeline-trigger-troubleshoot-guide#degree-of-parallelism--increase-doesnt-result-in-higher-throughput
The problem is that this is an obvious bug and they choose not to fix it, despite the fact that customers have struggled for years:Any concurrent or threaded programming language nowadays will allow tasks to be re-balanced while processing is underway. Customers of ADF will expect it to perform the dynamic load-balancing, especially given the excessive cost of the underlying compute and the underlying network components.
The workarounds can often be complex, and involve predicting how long something will take to run, before you run it. This prediction is not always accurate, and working on that prediction can take even more programming effort than the work that is done inside the loop.
Hi dbeavon3,
We apologize for the inconvenience. Unfortunately, we do not have an immediate solution currently. However, we will escalate this issue to our internal team to gather insights from various perspectives and resolve it as soon as possible.
Thank you.