Forum Discussion

carlossoria's avatar
carlossoria
Frequent Visitor
9 months ago
Solved

Scheduled pipeline notebook execution fails intermittently

Hi everyone,   I'm encountering an intermittent issue with a notebook executed via a scheduled pipeline in Microsoft Fabric. Occasionally, the notebook fails to start execution entirely, and other ...
data engineering
notebook
Spark Job Definition
  • carlossoria's avatar
    carlossoria
    9 months ago

    Hi Prasanna,
    Thanks for following up. The issue hasn’t occurred again. I made a couple of changes to optimize the code in the notebook that was failing most often (although the errors happened in several notebooks, not just that one), and it seems stable now. It’s been running for over a week without any failures.
    We can consider the problem resolved for now. If it happens again for any reason, I’ll reach out again to the community.
    Best regards,
    Carlos

Anonymous's avatar
Anonymous
Not applicable
9 months ago

Hi carlossoria,

 

The pipeline notebook failed because the Livy Spark session wasn’t active at the time of submission. The error message was: requirement failed: Session isn’t active (HTTP 400). This usually happens if the Spark session hasn’t finished initializing, has timed out, or was closed before the job was submitted. Next steps:
Please restart the notebook/pipeline to reinitialize the Spark session.
Ensure the cluster is healthy and the session is in idle or running state before submitting jobs.
If this keeps happening, we may need to add retry/wait logic in the pipeline to check session status before execution.

 

Thanks & Regards,

Prasanna Kumar