How many sessions I could run in F2 Capacity?

[AnmolGan81]

I was exploring Fabric capacity and currently I am on F2, but I was only able to run 1 spark job, on F2 and as soon as I started to run another job it wont let me do it and give me too many requests error, wanted to know how many jobs I can con-currently run on and F2 and is there any documentation which stats spark job limitations?

[emihle_mr]

silence, that is what has found you retorting at the empty void of information. it used to be about helping others first before resorting to gameplay to escape fun. I'm new to all of this, we'll have to discuss it further in our private messages, friendly reminder to use the F6 first before the L.

[emihle_mr]

doctors for life international v speaker of the national assembly and others [2006] ZACC 11

[v-veshwara-msft]

Hi @AnmolGan81 ,
Thanks for reaching out and sharing the details about the issue you're facing with running Spark jobs on the F2 capacity. After looking into it, I believe the problem you're encountering is a combination of resource contention and persistent Spark sessions that continue running in the background, even after the query finishes in your notebook.

When you run a query in the notebook, it may appear to complete successfully, but the Spark session might still be active in the background, holding onto resources. This can cause the capacity limits for F2 to be reached.

Possible solutions: 

1. After running each query, make sure to call spark.stop() in your notebook to explicitly terminate the Spark session. This will release the resources and allow other jobs to run without hitting the capacity limit.

         Example:

1. df = spark.sql("SELECT * FROM TestLakehouse.us_population_county_area LIMIT 1000")

   display(df)
   spark.stop()

2.Bursting:

Enabling bursting will allow you to use up to 20 Spark VCores instead of the base 4, which can help if you’re running multiple lightweight queries concurrently. Bursting will give you a bit more room, it's important to manage sessions actively.

3. Optimize Your Spark Pool Configuration:

Review the Spark pool settings to make sure you’re using the right node size and max nodes. Enabling dynamic allocation could help manage resources more efficiently, scaling up or down based on the workload.

I’d suggest starting by enabling bursting and managing your sessions more carefully with spark.stop() after each query. If that doesn’t fully resolve the issue, an upgrade to a larger capacity like F4 might be necessary, especially if your jobs are more resource-intensive.

Refer the below links for better understanding:
Concurrency limits and Bursting in Microsoft Fabric 
Burstable capacities 

Let me know if you need any help with these changes or if you have any other questions. I'm happy to assist further!

If this helps, please accept as solution to help others benefit, a kudos would be appreciated.

Best regards,
Vinay.

[AnmolGan81]

When I try to get into the notebook and try to run the same query I get below error, and nothing is running in job monitor as I checked before running any queries.

JSON { "type": "close", "timeStamp": 108670.20000001788, "code": 1000, "reason": "{\"reason\":\"Session error or stopped.\",\"state\":\"session-completed\"}", "wasClean": true, "target": { "url": "wss://6dc6322c24a042a2828f8e2aa68f9b82.pbidedicated.windows.net/webapi/capacities/6DC6322C-24A0-42A2-828F-8E2AA68F9B82/workloads/Notebook/Data/Direct/api/workspaces/cf8788ae-d2a9-4176-8900-5299acf0cce7/artifacts/0465d3f7-5fa7-4b86-a091-1395eb545a70/jupyterApi/versions/1/api/kernels/abd3f4ed-1da3-4124-a965-ae10fc877312/channels?token=dummy_token&session_id=c9350f12-d2b1-470a-ab6c-a33f1394c6aa", "readyState": 3, "protocolsProfile": [ 7, 3975 ] }, "currentTarget": { "url": "wss://6dc6322c24a042a2828f8e2aa68f9b82.pbidedicated.windows.net/webapi/capacities/6DC6322C-24A0-42A2-828F-8E2AA68F9B82/workloads/Notebook/Data/Direct/api/workspaces/cf8788ae-d2a9-4176-8900-5299acf0cce7/artifacts/0465d3f7-5fa7-4b86-a091-1395eb545a70/jupyterApi/versions/1/api/kernels/abd3f4ed-1da3-4124-a965-ae10fc877312/channels?token=dummy_token&session_id=c9350f12-d2b1-470a-ab6c-a33f1394c6aa", "readyState": 3, "protocolsProfile": [ 7, 3975 ] }, "isTrusted": true } Additional info: InstanceId: e529904c-127d-4fe7-bcfd-99e39c992504

[v-veshwara-msft]

Hi @AnmolGan81 ,

Thanks for sharing the error details. Based on the information, it seems that the session might have timed out, been forcibly stopped, or encountered an issue while maintaining resources.

Recommended Actions:

1. Restart the Spark Session:
   You can try restarting the session in your notebook by going to Connect > New standard or High Concurrency session. This will create a new session and re-establish the connection.
   Alternatively, you can stop all operations, close the notebook, and open it again to start with a fresh session

If the issue persists, you can try enabling Bursting and Auto-scaling as suggested in the previous response.

Hope these help. Please reach out if you face any issues.

If this helps, please accept as solution to help others benefit, a kudos would be appreciated.

Best regards,
Vinay.

[AnmolGan81]

I found what was going wrong, whenever I am terminating the job from monitor and trying to create another session it wont let me do it, so I would have to restart the capacity and actually terminate the spark session from the notebook and can easily restart it whenever needed, also closing the notebook and reopening will not fix the issue until the session has timed out or actually stopped.

funny thing is in F2 SKU if you have not terminated the session for one notebook and try to run the job in another notebook it wont let me do it and give too many requests error, I think that is due to F2 Size..seems cost is very high for an F2 SKU when I can actually run 1 single session on one notebook but seems that is the way to go about it as of now. 

Thanks for all help 😀 

[v-veshwara-msft · Accepted solution]

Hi @AnmolGan81 ,

Thanks for the detailed update and for sharing what you’ve observed, it’s great that you’ve figured out what’s happening with the sessions.

The issue seems to be:

On an F2 SKU, the limited resources (4 Spark vCores) mean you can only run one Spark session at a time, regardless of how many notebooks you’re using. So, if there’s an active session in one notebook, trying to start another session in a different notebook results in the "too many requests" error.

You’re also right that just closing the notebook or stopping the job from the Monitor doesn’t fully terminate the session unless it times out or you explicitly stop it.

A Few Suggestions:

1. Ensure the Session is Stopped:

   -Adding spark.stop() to the end of your notebook is the reliable way to free up resources.
   -Restarting the capacity works, but that’s more of a workaround than a long-term fix.

2. Enable Bursting (with Limitations):

   -Bursting temporarily increases the available Spark vCores for your capacity (e.g., an F2 SKU can scale up to 20 Spark vCores during bursts).
   -This allows for better concurrency, meaning you may be able to run multiple notebooks simultaneously during a burst period, provided the combined workload does not exceed the burst limit.
   -However, bursting is not a permanent solution and can only support short-term spikes in usage. If both jobs are resource-intensive, you might still run into resource contention even with bursting enabled.
   -Bursting depends on resource availability, and it's important to note that it only helps with concurrency but doesn't guarantee success for all parallel workloads.

3. Upgrade Your Capacity:

   -If running multiple sessions in parallel is essential, upgrading to an F4 SKU or higher would provide more Spark vCores and better concurrency support.

Let me know if you need help managing sessions, enabling bursting, or exploring capacity options. Happy to assist further!

If this helps, please accept as solution to help others benefit, a kudos would be appreciated.

Best regards,
Vinay.

[jennratten]

I'm glad it worked out for you.

[AnmolGan81]

I tried creating new sessions and also tried auto scaling and high concurrency but none of it are working and I am getting the same error as before.

[jennratten]

@AnmolGan81 - Are you specifically running the two queries in two different notebooks?  Have you tried putting them in the same notebook and then running the notebook so that both queries are executed?  The query you posted is very basic - you shouldn't need to change the capacity settings or scale up to a higher capacity.  Since you are just querying the first 1000 rows of data from a lakehouse table, you don't really need ot use spark - you can just use Python.  See the snip below.  Can you please reply with screenshots showing the language and language version being used?  Also please let us know what the Spark settings are - these appear in the workspace settings under Data Engineering.  Thanks!

[AnmolGan81]

No I am talking about running 2 queries in different notebook and not same notebook, yes I can very well run in the same notebook but I was trying to see what happens if I run two queries in different notebooks when 1 notebook query session is already active, and I have not changed the capacity setting and all this I am doing in on F2.

Well I found another blog which has faced the similar issue:
Re: Spark Sessions in MS Fabric Fail to Connect - Microsoft Fabric Community

[jennratten]

Hello @AnmolGan81 - thanks for posting.

Yes, there are limits on the number of sessions that can be created based on the capacity SKU - due to the compute resources for the capacity.  Concurrency limits are based on the number of Spark VCores for the capacity - each capacity unit (CU) provides 2 Spark VCores.  The F2 capacity has 2 capacity units and therefore 4 Spark VCores.  You can run multiple sessions, but the exact number will depend on the capacity resource available and the requirements of each session. If the capacity is temporarily fully utilized then additional sessions will be pended until capacity resources are available - if the capacity is fully utilized for a prolonged period of time then the sessions could be throttled or rejected.  This can be monitored using the Capacity Monitoring Report.

Here are some links to documentation where you can read more.

Concurrency limits and queueing in Apache Spark for Fabric - Microsoft Fabric | Microsoft Learn

Install the Microsoft Fabric capacity metrics app - Microsoft Fabric | Microsoft Learn

Please let me know if there are any other questions I can answer.

[AnmolGan81]

So I have been running basic query for a delta table that is already created, below query I am running in notebook

df = spark.sql("SELECT * FROM TestLakehouse.us_population_county_area LIMIT 1000")

display(df)

for first table it runs and suceeded but this keeps on running in monitor window, and when I run same query for another table it throws capacity issues which I posted previosuly, I dont understand if the query is completed running in the notebook then why it keeps on running in the monitor table and I have to manually cancel it to fire another query in order to not face the F2 capacity request issue, and again these are just basic queries nothing fancy.

[govindarajan_d]

Hi @AnmolGan81 ,

It is strange that the query keeps running after it finishes. Ideally, you shouldn't have to do this, but try adding the below code in your notebook at the end to see if that terminates the notebook. 

spark.stop()

You can also try using a high concurrency session for your notebooks if queries are split across different notebooks, so that you do not face the capacity exceeded issue. 

https://learn.microsoft.com/en-us/fabric/data-engineering/configure-high-concurrency-session-noteboo...

[AnmolGan81]

okay let me try it