Fabric notebook disconnecting with error message LIVY_JOB_STATE_ERROR

[henryjea]

Hello dear community. 

I have a notebook in fabric on one tennant, It needs to run for about 6h to go trough and it runs with no problem. Last time it ran for 21h (im training a ML model).

I have duplicated the environment, spark settings, imported the same public librarys in the environment etc. Everything is 1=1 in another tennant. The two tennants is also exaktly the same, same capacity etc. 

But when i run the notebook in the other tennant it runs about 50 minuter and then fails with the error message "LIVY_JOB_STATE_ERROR, Source: Unknown, Message: Job failed during run time with state=[error].

Is this a known error? What can i do to make the notebook work? 

Sorry if this is an unclear question, im new to this line of work.

All the best and thanks in advance!

[v-veshwara-msft]

Hi @henryjea ,

Just checking in to see if your query has been resolved and whether any of the responses were helpful. If so, kindly consider marking the relevant reply as the Accepted Solution to assist others who may have a similar question.

Additionally, could you please confirm whether the issue has been addressed through the support ticket with Microsoft?

If the issue is now resolved, we’d greatly appreciate it if you could share any key insights or the resolution here for the benefit of the wider community. We may consider closing this thread, if we dont hear back.

Should you need further assistance in the future, please feel free to create a new post in the Microsoft Fabric Community Forum. We’ll be glad to support you.
We are closing this thread for now.

Thank you for your time and participation.

[v-veshwara-msft]

Hi @henryjea ,

We haven’t heard back from you and wanted to follow up again. If the provided solutions resolved your issue, please consider marking it as the Accepted Solution to assist others in the community.

If you still need help, feel free to reach out.

Looking forward to your response.

Thank you!

[v-veshwara-msft · Accepted solution]

Hi @henryjea ,
Thanks for reaching out in Microsoft fabric Community,
Just wanted to check if your issue been resolved. If not, please follow the guidance below.

As mentioned by @nilendraFabric , this is a generic error, and similar issues have been faced by other users.
Solved: Issues running Spark Notebook - Microsoft Fabric Community

Since you have already tried the steps suggested, here's some additional information that might help:

While this specific error isn't listed among the current known issues in Fabric, there are related concerns that might be influencing your experience. Here's what I've found:
Spark Jobs Failing Due to Runtime Updates:

A recent known issue indicates that Spark jobs might fail following updates to Fabric Runtime 1.3, based on Apache Spark 3.5. This update included significant built-in library changes, which could affect environments where libraries were previously installed or overridden.
Recommended Action is to Reinstall Libraries: If you have custom dependencies, reinstall them to rebuild against the latest runtime updates.

More details on this can be found here: Known issue - Spark jobs might fail due to Runtime 1.3 updates for GA - Microsoft Fabric | Microsoft...

Additionally, if this persists, I would suggest to raise a Support ticket with Microsoft to get the issue addressed by PG team.
Here is the link for support ticket :Microsoft Fabric Support and Status | Microsoft Fabric

Hope this helps. Please reach out for further assistance.

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and a kudos would be appreciated.

Best Regards,
Vinay.

[nilendraFabric]

Hello @henryjea 

The error `LIVY_JOB_STATE_ERROR` is generic and often triggered by issues in Spark session initialization, resource exhaustion, or backend service instability.

Check Monitoring Hub for active jobs in your  workspace and capacity monitoring app just to make sure other jobs are not running and consuming the resources and causing resources constraints for your notebooks 

[henryjea]

Unfortinatly ive have done this, I have also tried to schedule the notebook to run during the night to prevent this issue, but always the same fail. It always runs around 50 minutes and the dissconnects.