Forum Discussion

YashRaj5's avatar
YashRaj5
Frequent Visitor
10 months ago
Solved

Shared Sessions are taking longer to get attached while executing notebook in pipeline run in fabric

Hello, Over the past 3–4 days, we've observed an increase in the time it takes for a shared Spark session to attach when transitioning from one notebook activity to another. We've provided screens...
data engineering
data pipeline
notebook
  • Vinodh247's avatar
    Vinodh247
    10 months ago

    The screenshots show increased attach times for shared spark sessions in fabric pipelines esp., at the start and end notebook activities. This slowdown is likely due to backend resource contention/session reuse delays within the shared spark pool. When multiple notebooks run concurrently or capacity is under load, session warmup and context switch times may increase.

     

    actions that you can try:

    1. Monitor fabric capacity metrics (CPU, memory, spark concurrency).

    2. Try assigning a dedicated spark session per pipeline or using “Run in new session” for critical notebooks.

    3. Clear idle sessions periodically using the fabric REST API or capacity admin tools.

    4. Raise a fabric support ticket with timestamps for correlation as this might be a transient backend performance regression.

v-achippa's avatar
v-achippa
Icon for Community Support rankCommunity Support
10 months ago

Hi YashRaj5,

 

Thank you for reaching out to Microsoft Fabric Community.

 

Thank you Vinodh247 for the prompt response.

 

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user for the issue worked? or let us know if you need any further assistance.

 

Thanks and regards,

Anjan Kumar Chippa

YashRaj5's avatar
YashRaj5
Frequent Visitor
10 months ago

Thanks Vinodh247 , for you input, we were also suspecting the same, but the issue is kind of still there, as we are facing high runtime because of this small change from the backend.