Forum Discussion

bruceOkallau's avatar
bruceOkallau
Regular Visitor
7 months ago
Solved

cannot run any spark code from notebook with starter pool and default workspace

hello, I have had the workspace upgraded a couple of times and currently on F64.   But I cannot run any spark from a notebook.   Even simply importing functions  "from pyspark.sql.functions imp...
data engineering
notebook
  • bruceOkallau's avatar
    bruceOkallau
    7 months ago

    MS support gave me the solution. I'll post it here in case it helps anyone in the future:

     

    In Admin Portal > Capactity Settings > Autoscale Billing for Apache Spark

    turn off autoscaling.

deborshi_nag's avatar
deborshi_nag
Super User
7 months ago

Hello bruceOkallau 

 

The root cause is probably a starter pool configuration vs. capacity limits mismatch in your Fabric workspace.
 
Starter pools in Fabric always use Medium node size; you don’t change node size for starter pools. What is configurable is the autoscale max node count and executor limits. For F64, the starter pool should allow Default max nodes = 10 and Max number of nodes = 16.
 
Your workspace appears to have a workload autoscale limit = 2 nodes, and the validation logic is (incorrectly) rejecting Medium with that limit. This looks like a misaligned starter pool or capacity setting rather than a PySpark issue.
 

1) Check the Starter pool settings in your workspace

  1. In your Fabric workspace, go to Workspace settings → Data Engineering / Data Science → StarterPool.
  2. Ensure Max nodes is set to a value compatible with F64 (e.g., 10 or higher, up to 16). If it currently shows 2, bump it up.
  3. Save, then restart the Spark session in your notebook (use the “Stop session” / “Start session” buttons).

2) Confirm the workspace is on the correct capacity and not throttled/paused

  • In Admin portal → Capacities, verify the workspace is assigned to your F64 capacity and that the capacity is running (not paused).
  • If you’ve recently changed capacities, the starter pool defaults can drift; reapplying settings or briefly restarting the capacity can clear stale limits. 

This should fix your issue. Hope this helps, kindly appreciate giving a Kudos or accepting as a Solution. 

 
 
 
 
  • bruceOkallau's avatar
    bruceOkallau
    Regular Visitor
    7 months ago

    Thank you for these tips, unfortunately this does not apply to my situation.

     

    for #1. the Autoscale value is set at 1-1 and if I change to anything else it gives me a red 'out of range' prompt and returns the value to 1. the mouse-over tool tip says 'value must be 1'

    the executors value is 0 similarly is not allowed to be changed.

     

    When I try to create a new pool it won't let me change the node size, it must be small and the nodes must be 1-1

     

    for #2 I did confirm that my workspace is assigned to the F64 capacity and that the F64 capacity's status is "active"