Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Enhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.

Reply
UdaySutar
Frequent Visitor

Fabric Failed to create session for executing notebook

The pipeline contains a notebook activity, and while it runs successfully at times, it occasionally fails, generating the following error.Please provide the reason of failure and how to fix this permanantly.


"error": {
"ename": "Exception",
"evalue": "Failed to create session for executing notebook. SessionId: Notebook: Notebook_",
"traceback": [
"Exception: Failed to create session for executing notebook. SessionId: Notebook: Notebook_",
"--> SparkCoreError/UnexpectedSessionState: Livy session has failed. Error code: SparkCoreError/UnexpectedSessionState. SessionInfo.State from SparkCore is Error: Encountered an unexpected session state Error while waiting for session to become Idle. Error description: [plugins.. WorkspaceType:<> CCID:<>] Attempt=[1]/[3]Cluster was in terminal state=[Cancelled] before it reached 'Ready' state. Cluster job has WorkspaceName=[], SpecName=[], and JobId=[].. Source: SparkCoreService."
]

1 ACCEPTED SOLUTION
V-yubandi-msft
Community Support
Community Support

Hi @UdaySutar ,

We’re following up on your query to check if the issue has been resolved. Could you please confirm the status?

If the issue has been resolved, we kindly request you to share your resolution or key insights in this thread. This will help other community members facing similar challenges.

 

If the issue persists, we recommend raising a support ticket instead of connecting via a call. You can submit a request here: Create a Support Ticket.

 

If we do not hear back, we’ll proceed with closing this thread. However, should you need further assistance in the future, feel free to reach out via the Microsoft Fabric Community Forum and create a new thread. Our team will be happy to assist you.

 

— Yugandhar
Community Support Team

View solution in original post

11 REPLIES 11
V-yubandi-msft
Community Support
Community Support

Hi @UdaySutar ,

We’re following up on your query to check if the issue has been resolved. Could you please confirm the status?

If the issue has been resolved, we kindly request you to share your resolution or key insights in this thread. This will help other community members facing similar challenges.

 

If the issue persists, we recommend raising a support ticket instead of connecting via a call. You can submit a request here: Create a Support Ticket.

 

If we do not hear back, we’ll proceed with closing this thread. However, should you need further assistance in the future, feel free to reach out via the Microsoft Fabric Community Forum and create a new thread. Our team will be happy to assist you.

 

— Yugandhar
Community Support Team

V-yubandi-msft
Community Support
Community Support

Hi @UdaySutar ,

We noticed we haven't received a response from you yet, so we wanted to follow up and ensure the solution we provided addressed your issue. If you require any further assistance or have additional questions, please let us know.

Your feedback is valuable to us, and we look forward to hearing from you soon.

V-yubandi-msft
Community Support
Community Support

Hi @UdaySutar ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

 

Thank you.

V-yubandi-msft
Community Support
Community Support

Hi @UdaySutar ,

Can you please confirm if the solution worked for you. If it did, please consider marking the answer 'Accept as Solution' so others with similar queries may find it more easily. If not, please share the details.

 

Thank you.

Srisakthi
Super User
Super User

Hi @UdaySutar ,

 

Step 1:  Goto your Workspace settings -> Spark settings -> High Concurrency then enable for notebooks and pipeline run options.

Step 2 : In your notebook attach to high concurrency session(use when you want to run manually)

Step 3: In your data pipeline -> notebook activity -> settings -> Advanced settings -> give your session tag name(anything)

 

So now when you run your pipeline, if no high concurrency session available it will create high concurrency session and in one session it can execute 5 notebooks.

 

Reference : https://learn.microsoft.com/en-us/fabric/data-engineering/configure-high-concurrency-session-noteboo...

 

Regards,

Srisakthi

 

Hi @Srisakthi 
this is failing for scheduled pipeline. can we connect on this .

Hi @UdaySutar ,

 

We can connect.

 

Regards,

Srisakthi

Hi @UdaySutar ,

The error message:

Failed to create session for executing notebook… Cluster was in terminal state=[Cancelled]"

This issue often occurs during scheduled executions because the Spark cluster is provisioned on demand. If the cluster fails to initialize in time, it may timeout or terminate unexpectedly.

 

To improve the reliability of scheduled pipeline runs, consider configuring the environment to use High Concurrency Mode with a session tag. This allows the pipeline to reuse an existing Spark session, reducing cold start delays that often disrupt scheduled tasks.

 

By using an active high concurrency session, the scheduled pipeline avoids the overhead of starting a new cluster each time, ensuring smoother execution and better stability.

 

I hope this helps....Please Accept as solution if this meets your needs and a Kudos would be appreciated.

 

Yugandhar.

Srisakthi
Super User
Super User

Hi @UdaySutar ,

 

Might be it is trying to spin up to multiple sessions to run the notebooks. Have you trued session tags and high concurrency session.? It can help you saving costs, startup time and it can avoid creating multiple sessions.

 

Regards,

Srisakthi

Hi @Srisakthi - can you explain in detail each steps to proceed  ?

V-yubandi-msft
Community Support
Community Support

Hi @UdaySutar ,

Thank you for engaging with the Microsoft Fabric Community.  The Spark runtime in Microsoft Fabric couldn't become ready and was shut down before execution. This happened due to limited resources, excessive job queue times, or failures in cluster setup and provisioning.

To resolve the notebook execution issue in Microsoft Fabric, make sure the Spark pool has enough resources and is properly configured.

  1. If sessions fail, increase the timeout duration to allow more startup time.
  2. If problems continue, restart the notebook block or terminate old sessions that might be causing conflicts.
  3. Scaling up resources in the Spark pool can improve performance, and checking for Fabric service outages ensures the issue isn't due to system disruptions

 

I found similar issues in the community that might be useful for your case, so it’s worth checking them out.

Solved: In Fabric Failed to create session for executing n... - Microsoft Fabric Community

Solved: Unable to start spark session: Livy session has fa... - Microsoft Fabric Community

 

If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.

Helpful resources

Announcements
Fabric July 2025 Monthly Update Carousel

Fabric Monthly Update - July 2025

Check out the July 2025 Fabric update to learn about new features.

August 2025 community update carousel

Fabric Community Update - August 2025

Find out what's new and trending in the Fabric community.