Forum Discussion
Helper IISpark notebook not taking all executors, only running on 1
hi v-prasare I had to re-write my code and break it into modules so I can efficiently uses executors and reducing the no. of rows to be processed.
However this was not the most ideal use of my time.
Hoping to see more flexibility around use of executors in future.
Helper IIThanks, tayloramy, for your feedback. I’m familiar with the behavior, but the executor startup time is taking longer than expected, which is affecting overall performance.
Thank you as well, v-prasare, for your input. AQE is enabled by default in Spark, and reducing the partition size hasn't made a significant difference. Additionally, keeping a custom pool warm isn’t a cost-effective option.
I’ll explore further code optimizations to work around this issue.
Community SupportThank you for your prompt response. Please continue to share updates on the progress, as these insights will be valuable for customers who may have similar questions.