Forum Discussion
Advocate IIHow many sessions I could run in F2 Capacity?
Hi AnmolGan81 ,
Thanks for the detailed update and for sharing what you’ve observed, it’s great that you’ve figured out what’s happening with the sessions.
The issue seems to be:
On an F2 SKU, the limited resources (4 Spark vCores) mean you can only run one Spark session at a time, regardless of how many notebooks you’re using. So, if there’s an active session in one notebook, trying to start another session in a different notebook results in the "too many requests" error.
You’re also right that just closing the notebook or stopping the job from the Monitor doesn’t fully terminate the session unless it times out or you explicitly stop it.
A Few Suggestions:
-
Ensure the Session is Stopped:
-Adding spark.stop() to the end of your notebook is the reliable way to free up resources.
-Restarting the capacity works, but that’s more of a workaround than a long-term fix. -
Enable Bursting (with Limitations):
-Bursting temporarily increases the available Spark vCores for your capacity (e.g., an F2 SKU can scale up to 20 Spark vCores during bursts).
-This allows for better concurrency, meaning you may be able to run multiple notebooks simultaneously during a burst period, provided the combined workload does not exceed the burst limit.
-However, bursting is not a permanent solution and can only support short-term spikes in usage. If both jobs are resource-intensive, you might still run into resource contention even with bursting enabled.
-Bursting depends on resource availability, and it's important to note that it only helps with concurrency but doesn't guarantee success for all parallel workloads. -
Upgrade Your Capacity:
-If running multiple sessions in parallel is essential, upgrading to an F4 SKU or higher would provide more Spark vCores and better concurrency support.
Let me know if you need help managing sessions, enabling bursting, or exploring capacity options. Happy to assist further!
If this helps, please accept as solution to help others benefit, a kudos would be appreciated.
Best regards,
Vinay.-
Community SupportHi AnmolGan81 ,
Thanks for reaching out and sharing the details about the issue you're facing with running Spark jobs on the F2 capacity. After looking into it, I believe the problem you're encountering is a combination of resource contention and persistent Spark sessions that continue running in the background, even after the query finishes in your notebook.
When you run a query in the notebook, it may appear to complete successfully, but the Spark session might still be active in the background, holding onto resources. This can cause the capacity limits for F2 to be reached.
Possible solutions:
1. After running each query, make sure to call spark.stop() in your notebook to explicitly terminate the Spark session. This will release the resources and allow other jobs to run without hitting the capacity limit.
Example:
-
df = spark.sql("SELECT * FROM TestLakehouse.us_population_county_area LIMIT 1000")display(df)
spark.stop()
2.Bursting:
Enabling bursting will allow you to use up to 20 Spark VCores instead of the base 4, which can help if you’re running multiple lightweight queries concurrently. Bursting will give you a bit more room, it's important to manage sessions actively.
3. Optimize Your Spark Pool Configuration:
Review the Spark pool settings to make sure you’re using the right node size and max nodes. Enabling dynamic allocation could help manage resources more efficiently, scaling up or down based on the workload.
I’d suggest starting by enabling bursting and managing your sessions more carefully with spark.stop() after each query. If that doesn’t fully resolve the issue, an upgrade to a larger capacity like F4 might be necessary, especially if your jobs are more resource-intensive.
Refer the below links for better understanding:
Concurrency limits and Bursting in Microsoft Fabric
Burstable capacities
Let me know if you need any help with these changes or if you have any other questions. I'm happy to assist further!
If this helps, please accept as solution to help others benefit, a kudos would be appreciated.
Best regards,
Vinay.
Memorable Membercan't remember where I read it, but I remember someone or the doc saying
mssparkutils.session.stop()
is preferred to
spark.stop()
Do you happen to know whether this is true? And if yes, why, that is, what could possibly be the drawback of the latter vs the former?