Forum Discussion
cannot run any spark code from notebook with starter pool and default workspace
hello, I have had the workspace upgraded a couple of times and currently on F64.
But I cannot run any spark from a notebook.
Even simply importing functions
MS support gave me the solution. I'll post it here in case it helps anyone in the future:
In Admin Portal > Capactity Settings > Autoscale Billing for Apache Spark
turn off autoscaling.
6 Replies
Hello bruceOkallau
The root cause is probably a starter pool configuration vs. capacity limits mismatch in your Fabric workspace.Starter pools in Fabric always use Medium node size; you don’t change node size for starter pools. What is configurable is the autoscale max node count and executor limits. For F64, the starter pool should allow Default max nodes = 10 and Max number of nodes = 16.Your workspace appears to have a workload autoscale limit = 2 nodes, and the validation logic is (incorrectly) rejecting Medium with that limit. This looks like a misaligned starter pool or capacity setting rather than a PySpark issue.1) Check the Starter pool settings in your workspace
- In your Fabric workspace, go to Workspace settings → Data Engineering / Data Science → StarterPool.
- Ensure Max nodes is set to a value compatible with F64 (e.g., 10 or higher, up to 16). If it currently shows 2, bump it up.
- Save, then restart the Spark session in your notebook (use the “Stop session” / “Start session” buttons).
2) Confirm the workspace is on the correct capacity and not throttled/paused
- In Admin portal → Capacities, verify the workspace is assigned to your F64 capacity and that the capacity is running (not paused).
- If you’ve recently changed capacities, the starter pool defaults can drift; reapplying settings or briefly restarting the capacity can clear stale limits.
This should fix your issue. Hope this helps, kindly appreciate giving a Kudos or accepting as a Solution.
Thank you for these tips, unfortunately this does not apply to my situation.
for #1. the Autoscale value is set at 1-1 and if I change to anything else it gives me a red 'out of range' prompt and returns the value to 1. the mouse-over tool tip says 'value must be 1'
the executors value is 0 similarly is not allowed to be changed.
When I try to create a new pool it won't let me change the node size, it must be small and the nodes must be 1-1
for #2 I did confirm that my workspace is assigned to the F64 capacity and that the F64 capacity's status is "active"
Hi bruceOkallau,
Thank you for contacting the Microsoft Fabric Community Forum.
Based on my understanding, the error SparkSettingsInvalidAutoscaleNodeSize – Invalid NodeSize value: Medium for autoscale limit 2 on Fabric F64 might be caused by a configuration mismatch in the Starter Spark pool following multiple capacity upgrades. Starter pools in Microsoft Fabric are system managed with fixed node size and autoscale settings. In rare cases, the pool metadata does not refresh automatically to reflect the current capacity SKU, causing Spark session validation to fail before any code executes. This is why even a simple import fails.
Please follow the workaround below which may help resolve the issue:
- Create and attach a custom Spark pool with an F64 compatible node size and enable autoscale.
If you prefer to continue using the Starter pool, please raise a Microsoft Fabric support ticket so that the backend Starter pool configuration can be realigned with the F64 capacity using the link:Microsoft Fabric Support and Status | Microsoft Fabric
For your reference, please find the links below:
Apache Spark compute for Data Engineering and Data Science - Microsoft Fabric | Microsoft Learn
Configure and manage starter pools in Fabric Spark. - Microsoft Fabric | Microsoft Learn
Create custom Apache Spark pools in Fabric - Microsoft Fabric | Microsoft LearnWe hope this information helps to resolve the issue. Should you have any further queries, please feel free to contact the Microsoft Fabric community.
Thank you.
it appears that the correct node size for F64 is 'medium' is that correct?
When I try to create a new pool it is defaulted to 'small' and the number of nodes must be 1-1 and 0 executors. When I try to save that pool I get this error:
MS support gave me the solution. I'll post it here in case it helps anyone in the future:
In Admin Portal > Capactity Settings > Autoscale Billing for Apache Spark
turn off autoscaling.
Hi bruceOkallau,
Thank you for the update. We are pleased to hear that your issue has been resolved. Should you have any further queries, please feel free to contact the Microsoft Fabric community.
Thank you.
