Forum Discussion

carlossoria's avatar
carlossoria
Frequent Visitor
9 months ago
Solved

Scheduled pipeline notebook execution fails intermittently

Hi everyone,   I'm encountering an intermittent issue with a notebook executed via a scheduled pipeline in Microsoft Fabric. Occasionally, the notebook fails to start execution entirely, and other ...
data engineering
notebook
Spark Job Definition
  • carlossoria's avatar
    carlossoria
    9 months ago

    Hi Prasanna,
    Thanks for following up. The issue hasn’t occurred again. I made a couple of changes to optimize the code in the notebook that was failing most often (although the errors happened in several notebooks, not just that one), and it seems stable now. It’s been running for over a week without any failures.
    We can consider the problem resolved for now. If it happens again for any reason, I’ll reach out again to the community.
    Best regards,
    Carlos

KevinChant's avatar
KevinChant
Icon for Super User rankSuper User
9 months ago

Yes I have. You need to monitor your capacity to see if limits are being reached by your users. In additin, I recommend check that the notebooks that are being used in the workspaces using the same capacity are being used optimally.

carlossoria's avatar
carlossoria
Frequent Visitor
9 months ago

Hello,

 

I've checked the capacity metrics, there are no signs of high usage or saturation during the periods when the notebooks failed. Capacity usage remains well below the maximum limit, and there are no rejection events due to lack of resources. I don't think these notebook errors are capacity related.

 

Thanks.