Forum Discussion

Ayush05-gateway's avatar
Ayush05-gateway
Icon for Helper II rankHelper II
10 months ago
Solved

Spark notebook not taking all executors, only running on 1

Dear community, I need some guidance. I'm using the starter pool with 1–10 executors, but when my PySpark notebook runs, it usually starts with only one executor.   Sometimes it takes almost a mi...
Notebook
  • Ayush05-gateway's avatar
    Ayush05-gateway
    10 months ago

    hi v-prasare I had to re-write my code and break it into modules so I can efficiently uses executors and reducing the no. of rows to be processed.

    However this was not the most ideal use of my time.

    Hoping to see more flexibility around use of executors in future.

Ayush05-gateway's avatar
Ayush05-gateway
Icon for Helper II rankHelper II
10 months ago

hi v-prasare I had to re-write my code and break it into modules so I can efficiently uses executors and reducing the no. of rows to be processed.

However this was not the most ideal use of my time.

Hoping to see more flexibility around use of executors in future.

v-prasare's avatar
v-prasare
Icon for Community Support rankCommunity Support
10 months ago

Hi Ayush05-gateway,

Thanks for calling out the executor‑scaling great insights, and kudos for digging deep to diagnose the behavior. Your scenario highlights exactly where more flexibility and clearer guidance would help Fabric users and capacity admins.

If you can, please share this as an Idea on Microsoft Fabric Ideas.

 

The product teams actively review that channel, and well‑articulated feedback like yours directly influences prioritization. More customers raising the same topic helps accelerate improvements.

 

 

Thanks,

Prashanth