Forum Discussion
Issues running Spark Notebook
Hi I'm runing spark notebooks on Fabric trial, it's been working great and no time out issues, or severe dalys in sessions starting.
But today, notebooks do not start the spark session, returning this type of error:
"Livy session has failed. Error code: SparkCoreError/SessionDidNotEnterIdle. SessionInfo.State from SparkCore is Error: Session did not enter idle state after 10 minutes. Source: System."
I see from the forum that others have experience similar issues, with severe delays in functionality.
Can anyone shed more light on this?
Thanks,
This issue persisted for a week. I updated to runtime 1.3 at the time and it had no effect on the issue. MS stated at the time it was a resource allocation issue. Since 07-18-2024 I have not experienced this issue with my pipelines. For now it seems MS has resolved the issue. (Central Canada)
75 Replies
What is the point of this page in circumstances like this..!
At this stage, I'm certain that's just a static table and not connected to any data.
Also having the same issue, as mentioned above. No outage or awareness on the service so very frustrating that we are facing an issue for so long yet no acknowledgement by Microsoft
Same issue here (North America). Commenting so I get the solution when it is found.
Same issue here in Norway, North Europe region. Getting the same error. Weirdly, the Azure status says everything is working fine.
It's frustrating that in the status of Fabric everything is "working without any problem", while this problem has been experienced in many regions...
My organization is encountering this issue today as well, also in Canada Central.
Ive submitted a support request to Unified Support, will update if I learn more.Looks like its been identified as an issue now, I was in contact with support briefly and passed on some information, then I received a notice around 10PM MST last night. Currently there is a posting in the admin center Service health - Microsoft 365 admin center:
Latest Update:
Same error North Europe
Hey everyone,
I just had some resolution on this issue by enabling runtime 1.3 (Preview) on my workspace. I have restarted a pipeline with hundreds of Notebook executions and they all seem to be running smoothly with the newer (Preview) runtime.
Of course this could just be a convenient glitch, so I will comment back tomorrow on the progress, but it looks promising to me as I was able to reproduce the issue with a manual execution of a notebook, and that has also been working since I made the config change.
If you are able to reproduce the issue while running Notebooks manually, I recommend experimenting with a custom Notebook environment set to runtime 1.3 and see if you get any better results.
I will follow up tomorrow with a status update.
Thank you
i also have the same solution applied and data pipeline now works successfully on 1.3 spark.
I've just tried the 1.3 change and got a Pandas error. Further reading this morning suggests I'm hitting memory issues based on the size of the data frame I'm outputting... Have edited this post as my code is the issue here, not Fabric 🙂
So just to reply to myself... I've had other pipelines calling notebooks over the weekend - default is now 1.3 - and they've worked ok (and have calls to pandas to output files).
I've not yet reverted to 1.2 to test it, but can confirm that 1.3 is working.
Looks like something strange is going on with this particular notebook...
I am also facing the same issue since yesterday both using the trial capacity and a reserved capacity - both from Canada Central
Having a similar issue. It's not timing out, since I don't do a lot of complex processes, but the spark pools usually takes 3 to 5 min to start (before the issue this was 10sec). This all started 24h ago for me (time of writing is 17/07/2024, 2PM Brussels)
Also hitting this same error right now in North Europe. It has not been fixed, contrary to screenshots suggesting otherwise.
This issue persisted for a week. I updated to runtime 1.3 at the time and it had no effect on the issue. MS stated at the time it was a resource allocation issue. Since 07-18-2024 I have not experienced this issue with my pipelines. For now it seems MS has resolved the issue. (Central Canada)
It seems that this issue is happening again Microsoft Fabric Support and Status | Microsoft Fabric.
Service Outage/Degradation
Fabric Premium customers in the North Europe region may be experiencing issues with submitting Spark jobs, executing notebook operations, or rendering Python and R visuals. Engineers are investigating the issue, and an update will be provided soon.
We noticed this issue which started from 2025/05/13 8PM (UTC+2). What's strange is that we are back at a P1 ticket with Microsoft but on the Service status, its only showing as Degraded.
The other sad part is that its been over 12 hours for us and still no resolution or any ETA on the fix.Looks like its back up again
