Forum Discussion

carlossoria's avatar
carlossoria
Frequent Visitor
9 months ago
Solved

Scheduled pipeline notebook execution fails intermittently

Hi everyone,   I'm encountering an intermittent issue with a notebook executed via a scheduled pipeline in Microsoft Fabric. Occasionally, the notebook fails to start execution entirely, and other ...
data engineering
notebook
Spark Job Definition
  • carlossoria's avatar
    carlossoria
    9 months ago

    Hi Prasanna,
    Thanks for following up. The issue hasn’t occurred again. I made a couple of changes to optimize the code in the notebook that was failing most often (although the errors happened in several notebooks, not just that one), and it seems stable now. It’s been running for over a week without any failures.
    We can consider the problem resolved for now. If it happens again for any reason, I’ll reach out again to the community.
    Best regards,
    Carlos

Anonymous's avatar
Anonymous
Not applicable
9 months ago

Hi carlossoria,

 

Thank you for reaching out to the Microsoft Fabric Forum Community, and special thanks to Gpop13 and Ugk161610  for prompt and helpful responses.

Just following up to see if the Response provided by community members were helpful in addressing the issue. if the issue still persists Feel free to reach out if you need any further clarification or assistance.

 

Best regards,
Prasanna Kumar

 

  • carlossoria's avatar
    carlossoria
    Frequent Visitor
    9 months ago

    Hi Prasanna,

    Thank you for following up.

    The issue still persists, although it has been intermittent. On Wednesday and Thursday, all executions ran correctly without any problems. However, on Tuesday we encountered the error described in the opening post, and today (Friday) we experienced a different error during execution:

    InvalidHttpRequestToLivy: Submission failed due to error content = ["requirement failed: Session isn't active."]
HTTP status code: 400

    This time, the error occurred during the execution phase, unlike the previous case where the session failed to start and the job could not begin.

     

    Regards