Error while creating the livy session & Failed to create session for executing notebook

[RaivisMezis]

Hey,
Im running Fabric Pipeline, which consists of approx 10 Notebooks that is running concurrently with same Session tag (inside those notebooks ar RunMultiple DAG to run around 30 other notebooks). Pipelline successfully has been running for 9 months, and 3 days ago it started to fail random notebooks within Pipeline. When Notbooks triggerend manually outside Pipeline, evrything is fine.

Error messages:

Error while creating the livy session or Failed to create session for executing notebook.
At timepoint CU is still at 35% of utilization. No other Jobs are running. 

[v-tsaipranay · Accepted solution]

Hi @RaivisMezis ,

Thank you for reaching out to the Microsoft fabric community forum.

To troubleshoot the Livy session failures in your Microsoft Fabric pipeline, start by reviewing how session tags are used. Since your pipeline runs ~10 notebooks concurrently each triggering ~30 others via RunMultiple DAG and all use the same session tag, this can overload the Live Pool Manager (LPM), especially in the North Europe region where throttling issues have been confirmed. Assigning unique session tags or reducing concurrency can help.

Also, avoid setting spark.fabric.pools.skipStarterPools unless necessary, as skipping starter pools increases session creation time and failure risk. Use built-in notebook utilities like notebookutils.stop() and restartPython() to manage sessions cleanly.

For diagnostics, run the LPM throttling query to identify overloaded nodes and coordinate with the LPM team if needed. Since manual notebook runs work fine, the issue likely lies in how Fabric orchestrates sessions under pipeline load.

Hope this helps, feel free to reach out for any further questions.

Thank you.

[RaivisMezis]

Hey @v-tsaipranay 

Thank you for the reply.
Yes, I already started yesterday to split the notebooks in the pipeline by adding different session tags.
Also, thanks for the suggestion to utilize restartPython().
Marking this as the solution, as I also received comments from the community about the same issue.

P.S. Tonight’s (ELT) run went smoothly 🙂

[abm_chris]

Have been experiencing this also in North Europe, usually see these before big releases and FabCon Vienna is soon.

[RaivisMezis]

Yes we ar also "on" NorthEurope

[DHruvMakutu]

We've also been experiencing this a lot recently. It happens sometimes other times it's an issue. Along with a PY4J error while saving which also keeps on happening on an intermittent basis. 

[fdnavarropecci]

We have been experiencing the same for the last couple of days