Forum Discussion
cannot run any spark code from notebook with starter pool and default workspace
- 7 months ago
MS support gave me the solution. I'll post it here in case it helps anyone in the future:
In Admin Portal > Capactity Settings > Autoscale Billing for Apache Spark
turn off autoscaling.
Hi bruceOkallau,
Thank you for contacting the Microsoft Fabric Community Forum.
Based on my understanding, the error SparkSettingsInvalidAutoscaleNodeSize – Invalid NodeSize value: Medium for autoscale limit 2 on Fabric F64 might be caused by a configuration mismatch in the Starter Spark pool following multiple capacity upgrades. Starter pools in Microsoft Fabric are system managed with fixed node size and autoscale settings. In rare cases, the pool metadata does not refresh automatically to reflect the current capacity SKU, causing Spark session validation to fail before any code executes. This is why even a simple import fails.
Please follow the workaround below which may help resolve the issue:
- Create and attach a custom Spark pool with an F64 compatible node size and enable autoscale.
If you prefer to continue using the Starter pool, please raise a Microsoft Fabric support ticket so that the backend Starter pool configuration can be realigned with the F64 capacity using the link:Microsoft Fabric Support and Status | Microsoft Fabric
For your reference, please find the links below:
Apache Spark compute for Data Engineering and Data Science - Microsoft Fabric | Microsoft Learn
Configure and manage starter pools in Fabric Spark. - Microsoft Fabric | Microsoft Learn
Create custom Apache Spark pools in Fabric - Microsoft Fabric | Microsoft Learn
We hope this information helps to resolve the issue. Should you have any further queries, please feel free to contact the Microsoft Fabric community.
Thank you.
- bruceOkallau7 months agoRegular Visitor
it appears that the correct node size for F64 is 'medium' is that correct?
When I try to create a new pool it is defaulted to 'small' and the number of nodes must be 1-1 and 0 executors. When I try to save that pool I get this error: