Forum Discussion
Scheduled pipeline notebook execution fails intermittently
Hi Prasanna,
Thanks for following up. The issue hasn’t occurred again. I made a couple of changes to optimize the code in the notebook that was failing most often (although the errors happened in several notebooks, not just that one), and it seems stable now. It’s been running for over a week without any failures.
We can consider the problem resolved for now. If it happens again for any reason, I’ll reach out again to the community.
Best regards,
Carlos
Hi carlossoria,
Yes, we’ve also come across this issue a few times with scheduled pipeline notebook executions in Fabric. The error usually shows “Notebook execution failed at Notebook service with http status code 200 – Failed to create session for executing notebook.”
In our case, it happened randomly and seemed to be related to temporary backend or session initialization issues in the Fabric notebook service. The pipeline shows as failed, but no logs appear since the session never actually started.
What worked for us was:
Re-running the same pipeline manually — it usually succeeds without any changes.
Making sure the workspace and capacity are active and not in sleep mode before the scheduled run.
Adding a small delay or pre-check activity (for example, a lightweight notebook or REST call) before the main notebook execution, to ensure the session starts properly.
It seems to be an intermittent platform-side issue rather than something in the notebook code itself. Hopefully Microsoft will address this in a future update.
Best regards,
Gopi Krishna
Hello and thank you for sharing those insights, Gopi.
Regarding option 3, I have a question: since the session initialization is specific to the notebook being executed at that moment, how does running a lightweight notebook first help? Wouldn’t the main notebook still need to start its own session afterwards, which could potentially fail again?
I’m asking because, in this case, the failed notebook was not the first one running in the environment — other notebooks had executed successfully before — so it doesn’t seem to be a cold start issue then.
Thanks again!
Hi carlossoria,
The idea behind adding a lightweight or pre-check notebook isn’t to fix the session for the main notebook directly, but to make sure the Fabric capacity and Spark environment are fully warmed up and responsive before the main job runs.
In our case, the failures mostly happened when the environment had been idle for a while, and the first heavy notebook in the schedule couldn’t create a session properly — it failed at the initialization step. By triggering a small “warm-up” notebook (something that just reads a small table or executes a simple cell), the compute session starts up the Spark engine and resources, so the next notebook’s session initializes faster and more reliably.
If other notebooks are already running successfully in your environment, then yes — it might not be a cold-start issue in your case. It could instead be related to transient session creation limits or capacity-level resource contention at the exact trigger time.
We saw that re-running manually almost always succeeded, which points to this being more of a temporary backend or session allocation glitch rather than a notebook-specific problem.
Hopefully Microsoft will address this soon — since it seems to affect scheduled runs more than manual ones.
Best regards,
Gopi Krishna