Forum Discussion

Ayush05-gateway's avatar
Ayush05-gateway
Icon for Helper II rankHelper II
9 months ago
Solved

Spark notebook not taking all executors, only running on 1

Dear community, I need some guidance. I'm using the starter pool with 1–10 executors, but when my PySpark notebook runs, it usually starts with only one executor.   Sometimes it takes almost a mi...
Notebook
  • Ayush05-gateway's avatar
    Ayush05-gateway
    9 months ago

    hi v-prasare I had to re-write my code and break it into modules so I can efficiently uses executors and reducing the no. of rows to be processed.

    However this was not the most ideal use of my time.

    Hoping to see more flexibility around use of executors in future.

tayloramy's avatar
tayloramy
Icon for Super User rankSuper User
9 months ago

Hi Ayush05-gateway, 

 

This is the intended behaviour. SPark sessions don't start with all executors as that would use far more CU on the capacity. So they scale up when needed. 

 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.