Forum Discussion
Spark notebook not taking all executors, only running on 1
Dear community,
I need some guidance.
I'm using the starter pool with 1–10 executors, but when my PySpark notebook runs, it usually starts with only one executor.
Sometimes it takes almost a minute before a second (or additional) executor spins up. This becomes frustrating, because switching to a custom pool takes 4–5 minutes to start, and due to these limitations, Spark isn’t operating at its full potential.
Does anyone have recommendations or solutions to improve executor scaling or startup performance?
My use case is writing a parquet file from dataframe to a lakehouse. DF is repartitioned to 200 partitions to improve some performance however the no. of exectuors is really impacting the performance.
If I dont repartition, it takes 5-7 minutes.
hi v-prasare I had to re-write my code and break it into modules so I can efficiently uses executors and reducing the no. of rows to be processed.
However this was not the most ideal use of my time.
Hoping to see more flexibility around use of executors in future.
8 Replies
Hi Ayush05-gateway,
Starter pools scale executors reactively, so jobs often begin with a single executor and only scale after Spark detects enough pending tasks this can take 45–60 seconds. That’s why you’re seeing delays before additional executors start.
To improve performance:
-
Force parallelism early (e.g., df.repartition(80).cache(); df.count()), which triggers autoscaling sooner. Use fewer partitions to reduce shuffle overhead (e.g., 80 instead of 200).
-
Enable Adaptive Query Execution (AQE) for smarter partition sizing.
-
For consistent performance, consider using a warm custom pool, as Starter pools are better suited for lightweight or interactive workloads.
Thanks,
Prashanth
-
Thanks, tayloramy, for your feedback. I’m familiar with the behavior, but the executor startup time is taking longer than expected, which is affecting overall performance.
Thank you as well, v-prasare, for your input. AQE is enabled by default in Spark, and reducing the partition size hasn't made a significant difference. Additionally, keeping a custom pool warm isn’t a cost-effective option.
I’ll explore further code optimizations to work around this issue.
Thank you for your prompt response. Please continue to share updates on the progress, as these insights will be valuable for customers who may have similar questions.
Hi Ayush05-gateway,
This is the intended behaviour. SPark sessions don't start with all executors as that would use far more CU on the capacity. So they scale up when needed.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
Hi Ayush05-gateway,
As we haven't herad back from you, did you find any workraoun for this requirement? If you still have any questions or need more support, please feel free to let us know. We are happy to help you.
Thank you for your patience and look forward to hearing from you.
Best Regards,
Prashanth Are
MS Fabric community supporthi v-prasare I had to re-write my code and break it into modules so I can efficiently uses executors and reducing the no. of rows to be processed.
However this was not the most ideal use of my time.
Hoping to see more flexibility around use of executors in future.
Hi Ayush05-gateway,
Thanks for calling out the executor‑scaling great insights, and kudos for digging deep to diagnose the behavior. Your scenario highlights exactly where more flexibility and clearer guidance would help Fabric users and capacity admins.
If you can, please share this as an Idea on Microsoft Fabric Ideas.
The product teams actively review that channel, and well‑articulated feedback like yours directly influences prioritization. More customers raising the same topic helps accelerate improvements.
Thanks,
Prashanth
hi v-prasare I did submit an Idea, thanks. Marking this topic as resolved. though no solution was used, just re-writing logic.
