Forum Discussion
Helper IIISpark notebook not taking all executors, only running on 1
hi v-prasare I had to re-write my code and break it into modules so I can efficiently uses executors and reducing the no. of rows to be processed.
However this was not the most ideal use of my time.
Hoping to see more flexibility around use of executors in future.
Helper IIIThanks, tayloramy, for your feedback. I’m familiar with the behavior, but the executor startup time is taking longer than expected, which is affecting overall performance.
Thank you as well, v-prasare, for your input. AQE is enabled by default in Spark, and reducing the partition size hasn't made a significant difference. Additionally, keeping a custom pool warm isn’t a cost-effective option.
I’ll explore further code optimizations to work around this issue.
v-prasareCommunity Support
Thank you for your prompt response. Please continue to share updates on the progress, as these insights will be valuable for customers who may have similar questions.