topic Re: Notebook Run Is Failing Continuously in Data Engineering

Notebook Run Is Failing Continuously

suriaprakash — Sat, 20 Jul 2024 08:57:10 GMT

Hi,

The notebook in my pipeline is continuously failing despite multiple retries. Occasionally, it runs fine. This issue started on 07/18 at midnight PT and still persists as of 07/19 at midnight PT, even though the Fabric support page indicates that all services are running fine in all regions. Most of the time, there is no information except for execution duration in the output of the notebook activity like below.

Very rarely, I receive the following message in the output of failed notebook activity:

{"status":"Failed","result":{"runId":"XXXXXXX-YYYY-51f0fXXXXX","runStatus":"Failed","sessionId":null,"sparkPool":"XXXXXX-YYYYYY-8140XXXX","error":{"ename":"Exception","evalue":"Something went wrong before starting the session. Please rerun the notebook. If this issue persists, please contact support.","traceback":["Exception: Something went wrong before starting the session. Please rerun the notebook. If this issue persists, please contact support."]},"lastCheckedOn":"2024-07-20T07:08:37.6733333Z","metadata":null,"exitValue":null},"message":"Notebook execution is in Failed state, runId: bbae20c2-daf3-4cf3-8715-51f0fad9ae5a","SparkMonitoringURL":"https://app.fabric.microsoft.com/workloads/de-ds/sparkmonitor/XXXXXX/?trident=1&experience=power-bi&tab=data","executionDuration":48}

Below is the error I get for the failure runs,

Thanks,

Suria

Re: Notebook Run Is Failing Continuously

SaiTejaTalasila — Sat, 20 Jul 2024 15:11:10 GMT

Hi @suriaprakash ,

Check your workspace settings and check whether the number of executers are auto or limited.If it is limited change it to auto and increase executers to max.Check spark UI incase if you are using spark programming it will show more details.If it is still not working download your report and create a new one and run it and check.And check for any changes in the source data type.

I hope it will be helpful.

Thanks,

Sai Teja

Re: Notebook Run Is Failing Continuously

suriaprakash — Mon, 22 Jul 2024 04:20:38 GMT

Thanks @SaiTejaTalasila for your prompt response!

There's no changes to either the pipeline or the datatypes recently. Also, I'm using the default Spark pool and so the executors are not limited. But I'll try to create a new Spark pool with no limit on executor count and try to run again. Will keep you posted.

Re: Notebook Run Is Failing Continuously

suriaprakash — Mon, 22 Jul 2024 04:27:37 GMT

I just tried with new custom Spark pool and got the same issue.

Thanks,

Suria

Re: Notebook Run Is Failing Continuously

suriaprakash — Mon, 22 Jul 2024 05:16:08 GMT

The notebook works fine when ran individually, but fails when executed as an activity in the pipeline. The error is also not self explanatory.

Below is what I have in the first cell to configure the session,

%%configure { "defaultLakehouse": { "name": { "parameterName": "lakeHouseName", "defaultValue": "Overall_Lakehouse" }, "workspaceId": { "parameterName": "lakeHouseWSId", "defaultValue": "74cbe53b-YYYY-XXXX-yyyy-XXXXXXXXXX" } } }

Re: Notebook Run Is Failing Continuously

RobinV_ — Mon, 22 Jul 2024 14:05:34 GMT

We are facing the same issue!
Were you able to resolve?

Re: Notebook Run Is Failing Continuously

suriaprakash — Mon, 22 Jul 2024 14:57:26 GMT

It's an intermittent issue for us. Though the issue hasn't got resolved, the frequency of it's occurence has reduced a lot today (07/22).

Re: Notebook Run Is Failing Continuously

RobinV_ — Mon, 22 Jul 2024 17:37:06 GMT

i noticed this occurs whenever we run the same notebook with different parameters in a for each loop. Whenever i activate serialize, or make smaller batches it seems to go through fine. So i think there is a max amount the same notebook can run at a time.

Re: Notebook Run Is Failing Continuously

VVT — Mon, 22 Jul 2024 18:31:41 GMT

We are having exactly the same issue starting right around that time period. If running the pipeline manually or from the point of failure, it seems to be running fine. No solution yet.

Re: Notebook Run Is Failing Continuously

Anonymous — Tue, 23 Jul 2024 01:54:20 GMT

Hi @suriaprakash ,

Thank you for posting in Fabric Community.

Are you using Azure notebooks? Because I see that your log format is Azure format and there is a difference between Azure synapse and Fabric synapse.

Re: Notebook Run Is Failing Continuously

suriaprakash — Thu, 25 Jul 2024 06:38:41 GMT

No, we're using Fabric notebooks.

And, the issue wasn't occuring now.

Re: Notebook Run Is Failing Continuously

Anonymous — Fri, 26 Jul 2024 01:43:17 GMT

@suriaprakash ，
We are very glad to know that the issue has been resolved.

Did you fix it by creating a new Spark pool with no limit on executor count?

If so, please remember to accept your solution as answer.

It will do great help to those who meet the similar question in this forum.

Re: Notebook Run Is Failing Continuously

suriaprakash — Fri, 26 Jul 2024 17:25:15 GMT

The problem got sorted out automatically. New spark pool with unlimited executors didn't solve the issue. Looks like a Fabric issue which happened coninously for a day or so.