Forum Discussion

Ayush05-gateway's avatar
Ayush05-gateway
Icon for Helper II rankHelper II
9 months ago
Solved

Spark notebook not taking all executors, only running on 1

Dear community, I need some guidance. I'm using the starter pool with 1–10 executors, but when my PySpark notebook runs, it usually starts with only one executor.   Sometimes it takes almost a mi...
notebook
  • Ayush05-gateway's avatar
    Ayush05-gateway
    9 months ago

    hi v-prasare I had to re-write my code and break it into modules so I can efficiently uses executors and reducing the no. of rows to be processed.

    However this was not the most ideal use of my time.

    Hoping to see more flexibility around use of executors in future.

v-prasare's avatar
v-prasare
Icon for Community Support rankCommunity Support
9 months ago

Hi Ayush05-gateway,

As we haven't herad back from you, did you find any workraoun for this requirement? If you still have any questions or need more support, please feel free to let us know. We are happy to help you.

 

 

Thank you for your patience and look forward to hearing from you.
Best Regards,
Prashanth Are
MS Fabric community support