topic Re: Fabric Failed to create session for executing notebook in Data Engineering

Fabric Failed to create session for executing notebook

UdaySutar — Tue, 06 May 2025 17:53:11 GMT

The pipeline contains a notebook activity, and while it runs successfully at times, it occasionally fails, generating the following error.Please provide the reason of failure and how to fix this permanantly.

"error": {
"ename": "Exception",
"evalue": "Failed to create session for executing notebook. SessionId: Notebook: Notebook_",
"traceback": [
"Exception: Failed to create session for executing notebook. SessionId: Notebook: Notebook_",
"--> SparkCoreError/UnexpectedSessionState: Livy session has failed. Error code: SparkCoreError/UnexpectedSessionState. SessionInfo.State from SparkCore is Error: Encountered an unexpected session state Error while waiting for session to become Idle. Error description: [plugins.. WorkspaceType:<> CCID:<>] Attempt=[1]/[3]Cluster was in terminal state=[Cancelled] before it reached 'Ready' state. Cluster job has WorkspaceName=[], SpecName=[], and JobId=[].. Source: SparkCoreService."
]

Re: Fabric Failed to create session for executing notebook

V-yubandi-msft — Wed, 07 May 2025 04:55:01 GMT

Hi @UdaySutar ,

Thank you for engaging with the Microsoft Fabric Community. The Spark runtime in Microsoft Fabric couldn't become ready and was shut down before execution. This happened due to limited resources, excessive job queue times, or failures in cluster setup and provisioning.

To resolve the notebook execution issue in Microsoft Fabric, make sure the Spark pool has enough resources and is properly configured.

If sessions fail, increase the timeout duration to allow more startup time.
If problems continue, restart the notebook block or terminate old sessions that might be causing conflicts.
Scaling up resources in the Spark pool can improve performance, and checking for Fabric service outages ensures the issue isn't due to system disruptions

I found similar issues in the community that might be useful for your case, so it’s worth checking them out.

Solved: In Fabric Failed to create session for executing n... - Microsoft Fabric Community

Solved: Unable to start spark session: Livy session has fa... - Microsoft Fabric Community

If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.

Re: Fabric Failed to create session for executing notebook

Srisakthi — Thu, 08 May 2025 12:35:28 GMT

Hi @UdaySutar ,

Might be it is trying to spin up to multiple sessions to run the notebooks. Have you trued session tags and high concurrency session.? It can help you saving costs, startup time and it can avoid creating multiple sessions.

Regards,

Srisakthi

Re: Fabric Failed to create session for executing notebook

UdaySutar — Thu, 08 May 2025 12:38:26 GMT

Hi @Srisakthi - can you explain in detail each steps to proceed ?

Re: Fabric Failed to create session for executing notebook

Srisakthi — Thu, 08 May 2025 12:45:31 GMT

Hi @UdaySutar ,

Step 1: Goto your Workspace settings -> Spark settings -> High Concurrency then enable for notebooks and pipeline run options.

Step 2 : In your notebook attach to high concurrency session(use when you want to run manually)

Step 3: In your data pipeline -> notebook activity -> settings -> Advanced settings -> give your session tag name(anything)

So now when you run your pipeline, if no high concurrency session available it will create high concurrency session and in one session it can execute 5 notebooks.

Reference : https://learn.microsoft.com/en-us/fabric/data-engineering/configure-high-concurrency-session-notebooks-in-pipelines

Regards,

Srisakthi

Re: Fabric Failed to create session for executing notebook

UdaySutar — Thu, 08 May 2025 16:33:46 GMT

Hi @Srisakthi
this is failing for scheduled pipeline. can we connect on this .

Re: Fabric Failed to create session for executing notebook

V-yubandi-msft — Mon, 12 May 2025 07:14:44 GMT

Hi @UdaySutar ,

The error message:

Failed to create session for executing notebook… Cluster was in terminal state=[Cancelled]"

This issue often occurs during scheduled executions because the Spark cluster is provisioned on demand. If the cluster fails to initialize in time, it may timeout or terminate unexpectedly.

To improve the reliability of scheduled pipeline runs, consider configuring the environment to use High Concurrency Mode with a session tag. This allows the pipeline to reuse an existing Spark session, reducing cold start delays that often disrupt scheduled tasks.

By using an active high concurrency session, the scheduled pipeline avoids the overhead of starting a new cluster each time, ensuring smoother execution and better stability.

I hope this helps....Please Accept as solution if this meets your needs and a Kudos would be appreciated.

Yugandhar.

Re: Fabric Failed to create session for executing notebook

Srisakthi — Mon, 12 May 2025 08:31:57 GMT

Hi @UdaySutar ,

We can connect.

Regards,

Srisakthi

Re: Fabric Failed to create session for executing notebook

V-yubandi-msft — Thu, 15 May 2025 12:42:01 GMT

Hi @UdaySutar ,

Can you please confirm if the solution worked for you. If it did, please consider marking the answer 'Accept as Solution' so others with similar queries may find it more easily. If not, please share the details.

Thank you.

Re: Fabric Failed to create session for executing notebook

V-yubandi-msft — Mon, 19 May 2025 05:09:47 GMT

Hi @UdaySutar ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

Re: Fabric Failed to create session for executing notebook

V-yubandi-msft — Wed, 21 May 2025 07:02:36 GMT

Hi @UdaySutar ,

We noticed we haven't received a response from you yet, so we wanted to follow up and ensure the solution we provided addressed your issue. If you require any further assistance or have additional questions, please let us know.

Your feedback is valuable to us, and we look forward to hearing from you soon.

Re: Fabric Failed to create session for executing notebook

V-yubandi-msft — Wed, 04 Jun 2025 09:21:44 GMT

Hi @UdaySutar ,

We’re following up on your query to check if the issue has been resolved. Could you please confirm the status?

If the issue has been resolved, we kindly request you to share your resolution or key insights in this thread. This will help other community members facing similar challenges.

If the issue persists, we recommend raising a support ticket instead of connecting via a call. You can submit a request here: Create a Support Ticket.

If we do not hear back, we’ll proceed with closing this thread. However, should you need further assistance in the future, feel free to reach out via the Microsoft Fabric Community Forum and create a new thread. Our team will be happy to assist you.

— Yugandhar
Community Support Team