Forum Discussion
Helper IIISpark notebook not taking all executors, only running on 1
hi v-prasare I had to re-write my code and break it into modules so I can efficiently uses executors and reducing the no. of rows to be processed.
However this was not the most ideal use of my time.
Hoping to see more flexibility around use of executors in future.
Community SupportHi Ayush05-gateway,
Starter pools scale executors reactively, so jobs often begin with a single executor and only scale after Spark detects enough pending tasks this can take 45–60 seconds. That’s why you’re seeing delays before additional executors start.
To improve performance:
-
Force parallelism early (e.g., df.repartition(80).cache(); df.count()), which triggers autoscaling sooner. Use fewer partitions to reduce shuffle overhead (e.g., 80 instead of 200).
-
Enable Adaptive Query Execution (AQE) for smarter partition sizing.
-
For consistent performance, consider using a warm custom pool, as Starter pools are better suited for lightweight or interactive workloads.
Thanks,
Prashanth