The ultimate Fabric, Power BI, SQL, and AI community-led learning event. Save €200 with code FABCOMM.
Get registeredEnhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.
The pipeline contains a notebook activity, and while it runs successfully at times, it occasionally fails, generating the following error.Please provide the reason of failure and how to fix this permanantly.
"error": {
"ename": "Exception",
"evalue": "Failed to create session for executing notebook. SessionId: Notebook: Notebook_",
"traceback": [
"Exception: Failed to create session for executing notebook. SessionId: Notebook: Notebook_",
"--> SparkCoreError/UnexpectedSessionState: Livy session has failed. Error code: SparkCoreError/UnexpectedSessionState. SessionInfo.State from SparkCore is Error: Encountered an unexpected session state Error while waiting for session to become Idle. Error description: [plugins.. WorkspaceType:<> CCID:<>] Attempt=[1]/[3]Cluster was in terminal state=[Cancelled] before it reached 'Ready' state. Cluster job has WorkspaceName=[], SpecName=[], and JobId=[].. Source: SparkCoreService."
]
Solved! Go to Solution.
Hi @UdaySutar ,
We’re following up on your query to check if the issue has been resolved. Could you please confirm the status?
If the issue has been resolved, we kindly request you to share your resolution or key insights in this thread. This will help other community members facing similar challenges.
If the issue persists, we recommend raising a support ticket instead of connecting via a call. You can submit a request here: Create a Support Ticket.
If we do not hear back, we’ll proceed with closing this thread. However, should you need further assistance in the future, feel free to reach out via the Microsoft Fabric Community Forum and create a new thread. Our team will be happy to assist you.
— Yugandhar
Community Support Team
Hi @UdaySutar ,
We’re following up on your query to check if the issue has been resolved. Could you please confirm the status?
If the issue has been resolved, we kindly request you to share your resolution or key insights in this thread. This will help other community members facing similar challenges.
If the issue persists, we recommend raising a support ticket instead of connecting via a call. You can submit a request here: Create a Support Ticket.
If we do not hear back, we’ll proceed with closing this thread. However, should you need further assistance in the future, feel free to reach out via the Microsoft Fabric Community Forum and create a new thread. Our team will be happy to assist you.
— Yugandhar
Community Support Team
Hi @UdaySutar ,
We noticed we haven't received a response from you yet, so we wanted to follow up and ensure the solution we provided addressed your issue. If you require any further assistance or have additional questions, please let us know.
Your feedback is valuable to us, and we look forward to hearing from you soon.
Hi @UdaySutar ,
May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.
Thank you.
Hi @UdaySutar ,
Can you please confirm if the solution worked for you. If it did, please consider marking the answer 'Accept as Solution' so others with similar queries may find it more easily. If not, please share the details.
Thank you.
Hi @UdaySutar ,
Step 1: Goto your Workspace settings -> Spark settings -> High Concurrency then enable for notebooks and pipeline run options.
Step 2 : In your notebook attach to high concurrency session(use when you want to run manually)
Step 3: In your data pipeline -> notebook activity -> settings -> Advanced settings -> give your session tag name(anything)
So now when you run your pipeline, if no high concurrency session available it will create high concurrency session and in one session it can execute 5 notebooks.
Reference : https://learn.microsoft.com/en-us/fabric/data-engineering/configure-high-concurrency-session-noteboo...
Regards,
Srisakthi
Hi @UdaySutar ,
The error message:
Failed to create session for executing notebook… Cluster was in terminal state=[Cancelled]"
This issue often occurs during scheduled executions because the Spark cluster is provisioned on demand. If the cluster fails to initialize in time, it may timeout or terminate unexpectedly.
To improve the reliability of scheduled pipeline runs, consider configuring the environment to use High Concurrency Mode with a session tag. This allows the pipeline to reuse an existing Spark session, reducing cold start delays that often disrupt scheduled tasks.
By using an active high concurrency session, the scheduled pipeline avoids the overhead of starting a new cluster each time, ensuring smoother execution and better stability.
I hope this helps....Please Accept as solution if this meets your needs and a Kudos would be appreciated.
Yugandhar.
Hi @UdaySutar ,
Might be it is trying to spin up to multiple sessions to run the notebooks. Have you trued session tags and high concurrency session.? It can help you saving costs, startup time and it can avoid creating multiple sessions.
Regards,
Srisakthi
Hi @UdaySutar ,
Thank you for engaging with the Microsoft Fabric Community. The Spark runtime in Microsoft Fabric couldn't become ready and was shut down before execution. This happened due to limited resources, excessive job queue times, or failures in cluster setup and provisioning.
To resolve the notebook execution issue in Microsoft Fabric, make sure the Spark pool has enough resources and is properly configured.
I found similar issues in the community that might be useful for your case, so it’s worth checking them out.
Solved: In Fabric Failed to create session for executing n... - Microsoft Fabric Community
Solved: Unable to start spark session: Livy session has fa... - Microsoft Fabric Community
If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.
User | Count |
---|---|
4 | |
4 | |
2 | |
2 | |
2 |
User | Count |
---|---|
17 | |
15 | |
11 | |
6 | |
5 |