Forum Discussion

Ryan1304's avatar
Ryan1304
Frequent Visitor
1 year ago
Solved

Spark Session Staring Issue

Hi All, 

 

I’ve recently been experiencing significant delays when starting sessions in my notebook environment. Previously, sessions would start in under 30 seconds. However, starting from this week, I’ve noticed the following issues:

- Startup time has increased significantly, now taking at least 3 minutes and 30 seconds, even for a simple or blank notebook.

- Some notebooks take over 10 minutes to start.

- This issue occurs consistently, including with new or empty notebooks.  

 

For your information: 

Fabric Capacity: F8 SKU, Location: Southeast Asia,

Spark setting: Starter Pool (Node Family: Memory Optimized, Node Size: Medium, Number of Nodes: 1-6) 

Run time version: 1.3 (Spark 3.5, Delta 3.2)

High concurrency: On, Native execution engine: Off

 

Could anyone please assist in diagnosing and resolving the issue? I’d appreciate any guidance or checks I can perform as well.

Thank you

  • For faster start up times, Increases the Nodes ( Virtual Machines) and Native Execution Engine ) . If you are using small VMs ( For example F8s, F16..), This problem persists. You should use F64 capacity or more. This is equivalent to Power BI Premium P1 Capacity.

     

     

     

6 Replies

  • v-prasare's avatar
    v-prasare
    Community Support

    Hi Ryan1304,

     

    Can you try below troubleshooting steps and let me know if it helps?

     

     Starter Pools may take longer to warm up after periods of inactivity. If no nodes are running, startup involves provisioning VMs, which takes time. try switching to a dedicated Spark pool, even a small one, can offer more consistent and faster startup times.

    Too many simultaneous sessions starts, pipeline triggers, or data refreshes can congest your capacity. Disable High Concurrency mode temporarily to see if startup improves

    navigate to the Admin Portal and selecting Capacity settings, then go to the Monitor section and open Capacity metrics. check key indicators such as CPU and memory pressure, queue length, and overall resource utilization trends. Pay close attention to any signs of throttling or job queuing, which may indicate that your capacity is under strain or experiencing contention, leading to delays in notebook session startup

     

     

     

     

     

    Thanks,

    Prashanth Are

    MS Fabric community

     

    If this solution works, please consider marking it as Accept as Solution, this will help other community members easily find answers to similar issues.

  • v-prasare's avatar
    v-prasare
    Community Support

    Hi Ryan1304,

    As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

     

     

     

    Thanks,

    Prashanth Are

    MS Fabric community support

     

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

  • Ryan1304's avatar
    Ryan1304
    Frequent Visitor

    Hi v-prasare 

     

    Thanks for the guidance. I had try Disable High Concurrency mode and also using a dedicated Spark pool with a small size and 3 maximum of nodes. But the time of starting the spark session still remain same. 

  • v-prasare's avatar
    v-prasare
    Community Support


    We are following up once again regarding your query. Could you please confirm if the issue has been resolved through the support ticket with Microsoft?
    If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community. If we don’t hear back, we’ll go ahead and close this thread.
    Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.

    Thank you for your understanding and participation.

  • For faster start up times, Increases the Nodes ( Virtual Machines) and Native Execution Engine ) . If you are using small VMs ( For example F8s, F16..), This problem persists. You should use F64 capacity or more. This is equivalent to Power BI Premium P1 Capacity.