Forum Discussion
Scheduled pipeline notebook execution fails intermittently
Hi Prasanna,
Thanks for following up. The issue hasn’t occurred again. I made a couple of changes to optimize the code in the notebook that was failing most often (although the errors happened in several notebooks, not just that one), and it seems stable now. It’s been running for over a week without any failures.
We can consider the problem resolved for now. If it happens again for any reason, I’ll reach out again to the community.
Best regards,
Carlos
Hello and thank you for sharing those insights, Gopi.
Regarding option 3, I have a question: since the session initialization is specific to the notebook being executed at that moment, how does running a lightweight notebook first help? Wouldn’t the main notebook still need to start its own session afterwards, which could potentially fail again?
I’m asking because, in this case, the failed notebook was not the first one running in the environment — other notebooks had executed successfully before — so it doesn’t seem to be a cold start issue then.
Thanks again!
Super UserHi carlossoria,
The idea behind adding a lightweight or pre-check notebook isn’t to fix the session for the main notebook directly, but to make sure the Fabric capacity and Spark environment are fully warmed up and responsive before the main job runs.
In our case, the failures mostly happened when the environment had been idle for a while, and the first heavy notebook in the schedule couldn’t create a session properly — it failed at the initialization step. By triggering a small “warm-up” notebook (something that just reads a small table or executes a simple cell), the compute session starts up the Spark engine and resources, so the next notebook’s session initializes faster and more reliably.
If other notebooks are already running successfully in your environment, then yes — it might not be a cold-start issue in your case. It could instead be related to transient session creation limits or capacity-level resource contention at the exact trigger time.
We saw that re-running manually almost always succeeded, which points to this being more of a temporary backend or session allocation glitch rather than a notebook-specific problem.
Hopefully Microsoft will address this soon — since it seems to affect scheduled runs more than manual ones.
Best regards,
Gopi Krishna