topic Re: Concurrent spark sessions in Data Engineering

Concurrent spark sessions

Kar_c — Fri, 07 Mar 2025 22:31:02 GMT

Hi, We have a fabric capacity with F8 SKU

(medium, memory optimised), we are facing too many sessions issue very often while using spark notebooks or running pipelines. Also, if two pipelines run together both are taking so long to complete. How my concurrent spark sessions can we have with F8 SKU.

What can we do to handle this scenario?

Re: Concurrent spark sessions

nilendraFabric — Fri, 07 Mar 2025 22:56:46 GMT

Hello @Kar_c

• F8 SKU provides:
• 16 Spark vCores (base).
• Burst multiplier: 3x → up to 48 Spark vCores.
• Each Medium Starter Pool session (default) uses 16 vCores (2 nodes × 8 vCores/node).
• Without bursting: Only 1 session can run (16 vCores / 16 vCores per session).
• With bursting: Up to 3 sessions (48 vCores / 16 vCores per session

If capacity is exceeded:
• Interactive notebooks: Fail with “Too Many Requests For Capacity” errors.
• Scheduled jobs (via Pipelines/Scheduler): Queue up to 8 jobs

Each pipeline activity starts a separate session. For example:
• 2 pipelines running 2 notebooks each = 4 sessions (exceeds F8’s 3-session limit with bursting).

try implementing these for better concurrency

Use High Concurrency Mode
• Allows up to 5 notebooks to share a single Spark session

Leverage `NotebookUtils.RunMultiple`
• Run multiple notebooks in parallel within a single session to avoid capacity limits

Single-node Spark Pool:
• Configure autoscale to 1 node → 8 vCores per session.
• F8 can run 6 concurrent sessions (48 vCores / 8 vCores per session)
Custom Small Pool:
• Half the size of Medium → 4 vCores per session.

Re: Concurrent spark sessions

v-saisrao-msft — Mon, 10 Mar 2025 07:07:39 GMT

Hi @Kar_c,

Thank you for reaching out to the Microsoft Fabric Forum Community.

Thank you for your question. I wanted to check if you had a chance to review the information provided by @nilendraFabric,on optimizing Spark concurrency. Additionally, I am also including Microsoft documentation which might help you understand better.

Concurrency limits and queueing in Apache Spark for Fabric - Microsoft Fabric | Microsoft Learn

If this post helps, please give us ‘Kudos’ and consider accepting it as a solution to help others find it more easily.

Thank you

Re: Concurrent spark sessions

v-saisrao-msft — Thu, 13 Mar 2025 06:20:27 GMT

Hi @Kar_c,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

Re: Concurrent spark sessions

v-saisrao-msft — Sun, 16 Mar 2025 16:15:11 GMT

Hi @Kar_c,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

Re: Concurrent spark sessions

v-saisrao-msft — Wed, 19 Mar 2025 05:12:04 GMT

Hi @Kar_c,

We haven’t heard back from you regarding your issue. If it has been resolved, please mark the helpful response as the solution and give a ‘Kudos’ to assist others. If you still need support, let us know.

Thank you.