Notebook Run Is Failing Continuously

[suriaprakash]

Hi,

The notebook in my pipeline is continuously failing despite multiple retries. Occasionally, it runs fine. This issue started on 07/18 at midnight PT and still persists as of 07/19 at midnight PT, even though the Fabric support page indicates that all services are running fine in all regions. Most of the time, there is no information except for execution duration in the output of the notebook activity like below.

Very rarely, I receive the following message in the output of failed notebook activity:

{"status":"Failed","result":{"runId":"XXXXXXX-YYYY-51f0fXXXXX","runStatus":"Failed","sessionId":null,"sparkPool":"XXXXXX-YYYYYY-8140XXXX","error":{"ename":"Exception","evalue":"Something went wrong before starting the session. Please rerun the notebook. If this issue persists, please contact support.","traceback":["Exception: Something went wrong before starting the session. Please rerun the notebook. If this issue persists, please contact support."]},"lastCheckedOn":"2024-07-20T07:08:37.6733333Z","metadata":null,"exitValue":null},"message":"Notebook execution is in Failed state, runId: bbae20c2-daf3-4cf3-8715-51f0fad9ae5a","SparkMonitoringURL":"https://app.fabric.microsoft.com/workloads/de-ds/sparkmonitor/XXXXXX/?trident=1&experience=power-bi&tab=data","executionDuration":48}

Below is the error I get for the failure runs,

Thanks,

Suria

[Anonymous]

Hi @suriaprakash ,

Thank you for posting in Fabric Community.

Are you using Azure notebooks? Because I see that your log format is Azure format and there is a difference between Azure synapse and Fabric synapse.

[suriaprakash]

No, we're using Fabric notebooks.

And, the issue wasn't occuring now.

[Anonymous]

@suriaprakash ，
We are very glad to know that the issue has been resolved.

Did you fix it by creating a new Spark pool with no limit on executor count?

If so, please remember to accept your solution as answer.

It will do great help to those who meet the similar question in this forum.

[suriaprakash · Accepted solution]

The problem got sorted out automatically. New spark pool with unlimited executors didn't solve the issue. Looks like a Fabric issue which happened coninously for a day or so.

[VVT]

We are having exactly the same issue starting right around that time period. If running the pipeline manually or from the point of failure, it seems to be running fine. No solution yet. 

[RobinV_]

We are facing the same issue!
Were you able to resolve?

[suriaprakash]

It's an intermittent issue for us. Though the issue hasn't got resolved, the frequency of it's occurence has reduced a lot today (07/22).

[RobinV_]

i noticed this occurs whenever we run the same notebook with different parameters in a for each loop. Whenever i activate serialize, or make smaller batches it seems to go through fine. So i think there is a max amount the same notebook can run at a time.

[suriaprakash]

The notebook works fine when ran individually, but fails when executed as an activity in the pipeline. The error is also not self explanatory.

Below is what I have in the first cell to configure the session,

%%configure
{
    "defaultLakehouse": {
        "name": {
            "parameterName": "lakeHouseName",
            "defaultValue": "Overall_Lakehouse"
        },
        "workspaceId": {
            "parameterName": "lakeHouseWSId",
            "defaultValue": "74cbe53b-YYYY-XXXX-yyyy-XXXXXXXXXX"
        }
    }
}

[suriaprakash]

I just tried with new custom Spark pool and got the same issue.

Thanks,

Suria

[SaiTejaTalasila]

Hi @suriaprakash ,

Check your workspace settings and check whether the number of executers are auto or limited.If it is limited change it to auto and increase executers to max.Check spark UI incase if you are using spark programming it will show more details.If it is still not working download your report and create a new one and run it and check.And check for any changes in the source data type.

I hope it will be helpful.

Thanks,

Sai Teja 

[suriaprakash]

Thanks @SaiTejaTalasila for your prompt response!

There's no changes to either the pipeline or the datatypes recently. Also, I'm using the default Spark pool and so the executors are not limited. But I'll try to create a new Spark pool with no limit on executor count and try to run again. Will keep you posted.