User Profile
carlossoria
Frequent Visitor
Joined 1 year ago
User Widgets
Contributions
Re: Scheduled pipeline notebook execution fails intermittently
Hi Prasanna, Thanks for following up. The issue hasn’t occurred again. I made a couple of changes to optimize the code in the notebook that was failing most often (although the errors happened in several notebooks, not just that one), and it seems stable now. It’s been running for over a week without any failures. We can consider the problem resolved for now. If it happens again for any reason, I’ll reach out again to the community. Best regards, Carlos2.3KViews1like0CommentsRe: Scheduled pipeline notebook execution fails intermittently
Hi Prasanna, Thank you for following up. The issue still persists, although it has been intermittent. On Wednesday and Thursday, all executions ran correctly without any problems. However, on Tuesday we encountered the error described in the opening post, and today (Friday) we experienced a different error during execution: InvalidHttpRequestToLivy: Submission failed due to error content = ["requirement failed: Session isn't active."] HTTP status code: 400 This time, the error occurred during the execution phase, unlike the previous case where the session failed to start and the job could not begin. Regards2.7KViews1like0CommentsRe: Scheduled pipeline notebook execution fails intermittently
Hello, I've checked the capacity metrics, there are no signs of high usage or saturation during the periods when the notebooks failed. Capacity usage remains well below the maximum limit, and there are no rejection events due to lack of resources. I don't think these notebook errors are capacity related. Thanks.2.8KViews0likes0CommentsRe: Scheduled pipeline notebook execution fails intermittently
Hello and thank you for sharing those insights, Gopi. Regarding option 3, I have a question: since the session initialization is specific to the notebook being executed at that moment, how does running a lightweight notebook first help? Wouldn’t the main notebook still need to start its own session afterwards, which could potentially fail again? I’m asking because, in this case, the failed notebook was not the first one running in the environment — other notebooks had executed successfully before — so it doesn’t seem to be a cold start issue then. Thanks again!2.8KViews0likes1CommentRe: Scheduled pipeline notebook execution fails intermittently
Hello! I'm going to answer directly on the reply you sent: Hi @carlossoria , is there high concurrency ON for the notebooks? No, none of the notebooks have high concurrency on, all of them use standard session mode. It may happen so that if some other notebooks are running using high concurrency session, this notebook may not have had sufficient resources to start execution. May be altering the spark cluster settings could help? also usually retry option helps but you mentioned it's not viable as it could cause duplicates. I am not aware of your architecture but if the notebook has not run then why there could be duplicates if retried? It's not viable because there are some cases in which this error happens during execution (not this particular case, but the log error is more or less the same). Then, if I establish retries in a notebook that performs writing operations on some tables and the notebook starts executing, I cannot guarantee that any writing has not been done prior to the session error.2.9KViews0likes0CommentsScheduled pipeline notebook execution fails intermittently
Hi everyone, I'm encountering an intermittent issue with a notebook executed via a scheduled pipeline in Microsoft Fabric. Occasionally, the notebook fails to start execution entirely, and other times it crashes during execution (error with Livy session). As this (and other) notebooks in our pipelines perform writing operations, I can't establish retry logic as it could lead to duplicates. In this specific case, the notebook failed to start, and the pipeline run shows the following error: Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Failed to create session for executing notebook. This error seems to indicate that the notebook service returned a success status code (200), but internally failed to create a session for execution. Since the notebook never started, no logs are available inside the notebook itself. Has anyone experienced similar behavior or found a reliable way to handle this kind of failure? Any insights or suggestions would be greatly appreciated! Thanks in advance, CarlosSolved2.9KViews1like13CommentsError opening a Lakehouse
Hi, I have trouble opening whichever lakehouse in Fabric. When I open one, the Fabric logo stays on the screen like Fabric is charging the content of the lakehouse but it stays like that. I have tried to open a lot of lakehouses but all of them stay the same, it keeps charging forever. It started happening like 2 hours ago and still no one is working. Has anyone encountered this bug? Thanks in advance!Solved1KViews0likes4Comments
Data Privacy
Microsoft Fabric Community and Privacy
To learn more about how we manage your data, please review the Microsoft Fabric Community Data Privacy guide.