Forum Discussion

carlossoria's avatar
carlossoria
Frequent Visitor
9 months ago
Solved

Scheduled pipeline notebook execution fails intermittently

Hi everyone,   I'm encountering an intermittent issue with a notebook executed via a scheduled pipeline in Microsoft Fabric. Occasionally, the notebook fails to start execution entirely, and other ...
data engineering
notebook
Spark Job Definition
  • carlossoria's avatar
    carlossoria
    9 months ago

    Hi Prasanna,
    Thanks for following up. The issue hasn’t occurred again. I made a couple of changes to optimize the code in the notebook that was failing most often (although the errors happened in several notebooks, not just that one), and it seems stable now. It’s been running for over a week without any failures.
    We can consider the problem resolved for now. If it happens again for any reason, I’ll reach out again to the community.
    Best regards,
    Carlos

Gpop13's avatar
Gpop13
Icon for Advocate IV rankAdvocate IV
9 months ago

Hi carlossoria , is there high concurrency ON for the notebooks? It may happen so that if some other notebooks are running using high concurrency session, this notebook may not have had sufficient resources to start execution. May be altering the spark cluster settings could help?

also usually retry option helps but you mentioned it's not viable as it could cause duplicates. I am not aware of your architecture but if the notebook has not run then why there could be duplicates if retried?

 

 

carlossoria's avatar
carlossoria
Frequent Visitor
9 months ago

Hello! I'm going to answer directly on the reply you sent:

 

Hi @carlossoria , is there high concurrency ON for the notebooks? No, none of the notebooks have high concurrency on, all of them use standard session mode. It may happen so that if some other notebooks are running using high concurrency session, this notebook may not have had sufficient resources to start execution. May be altering the spark cluster settings could help?

also usually retry option helps but you mentioned it's not viable as it could cause duplicates. I am not aware of your architecture but if the notebook has not run then why there could be duplicates if retried? It's not viable because there are some cases in which this error happens during execution (not this particular case, but the log error is more or less the same). Then, if I establish retries in a notebook that performs writing operations on some tables and the notebook starts executing, I cannot guarantee that any writing has not been done prior to the session error.