topic Re: Notebook Error: SparkCoreError/PersonalizingFailed: Livy session has failed in Data Engineering

Notebook Error: SparkCoreError/PersonalizingFailed: Livy session has failed

mdhewitt83 — Wed, 07 Aug 2024 16:34:01 GMT

We started receiving this error when running Notebooks within the last day:

SparkCoreError/PersonalizingFailed: Livy session has failed. Error code: SparkCoreError/PersonalizingFailed. SessionInfo.State from SparkCore is Error: Encountered an unexpected error while personalizing session. Failed step: 'LM_LibraryManagementPersonalizationStatement'. Source: System.

Notebooks were previously running with no issues. We're on a reserved capacity in East US, we do have a public library installed (great-expectations) in the environment.

I'm aware of other users receiving this error in other regions and it looks like Microsoft resolved those issues, but is anyone else getting this error?

Re: Notebook Error: SparkCoreError/PersonalizingFailed: Livy session has failed

Anonymous — Thu, 08 Aug 2024 03:12:26 GMT

Hi @mdhewitt83

I searched internally and on the forums, but I haven’t found any other users reporting this error recently. I tested starting a session on a custom environment with the public library (great-expectations) installed without having any error. Sometimes, it might be a temporary problem. You could try again to see if it still occurs.

Additionally, the error you got is more similar to the one in the following thread. If current environment still doesn't work, you can try creating and using a new environment to run the notebook, or try pip install the great-expectations module in notebook directly.

Power BI Lakehouse notebook cannot start session - Microsoft Q&A

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it.

Re: Notebook Error: SparkCoreError/PersonalizingFailed: Livy session has failed

mdhewitt83 — Fri, 09 Aug 2024 16:18:14 GMT

I'm still having this issue, I've tried creating multiple environments in different workspaces. I prefer to use an environment as I'll be using the great expectations library in alot of notebooks so I don't want to install it everytime a notebook runs.

I'm finding that I cannot publish an environment with great-expectations installed in the public libraries unless I install v1.0.0a1 or above - these are prerelease versions. if i go with the current or older versions of the library i cannot publish the enviromnent and I get a generic error.

@Anonymous what configuration were you able to get working?

Re: Notebook Error: SparkCoreError/PersonalizingFailed: Livy session has failed

Anonymous — Wed, 14 Aug 2024 06:43:39 GMT

Hi @mdhewitt83

Unfortunately I failed to publish an environment with great-expectations 0.18.19 today. I got the following error. I tried PyPI and Conda sources, both failed. But for v1.0.0a1, it succeeded.

However when I published an environment with the same 0.18.19 version last week, it succeeded.

Did you get the same error notification when the publish failed?

Best Regards,
Jing

Re: Notebook Error: SparkCoreError/PersonalizingFailed: Livy session has failed

mdhewitt83 — Wed, 14 Aug 2024 12:50:00 GMT

Hey @Anonymous , that is the same error.

Re: Notebook Error: SparkCoreError/PersonalizingFailed: Livy session has failed

Anonymous — Thu, 15 Aug 2024 09:32:59 GMT

Hi @mdhewitt83

I checked the error log of my publish failure. It seems the library was installed successfully in a new environment but the environment publish failed due to some error. I haven't found any solution or workaround yet. I will escalate this Publish environment failure issue internally, and update here once I got any feedback.

For the initial "SparkCoreError/PersonalizingFailed: Livy session has failed." error, I'm not sure if it has anything to do with the great-expectations library in the environment where your notebook run in. I would suggest raising a support ticket at Microsoft Fabric Support and Status | Microsoft Fabric

Best Regards,
Jing

Re: Notebook Error: SparkCoreError/PersonalizingFailed: Livy session has failed

Anonymous — Thu, 22 Aug 2024 08:47:57 GMT

Hi @mdhewitt83

I have escalated the publish failure issue internally but the engineer couldn't reproduce it. It seems not a general issue.

I'd like to confirm with you that whether you can publish an environment with the current release (0.18.19) of great-expectations library at present? If it still failed, you can try to find the error logs. Try publishing the environment again, and you may see below messages. Click View log, then View details. It will generate the logs in a new browser tab. See if there is any error in it.

If you still cannot publish the environment, I suggest creating a support ticket to investigate this further.

Best Regards,
Jing

Re: Notebook Error: SparkCoreError/PersonalizingFailed: Livy session has failed

m98ateusz — Fri, 23 Aug 2024 06:05:58 GMT

Hi @mdhewitt83 . I got the same issue today. I have pipeline, which use environment, where we have custom library ( inside we use great expectations ). Before pipeline worked fine and today I got error :

- in pipeline : Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Failed to create Livy session for executing notebook.

- when running notebook itself : Error code: SparkCoreError/PersonalizingFailed.

Re: Notebook Error: SparkCoreError/PersonalizingFailed: Livy session has failed

SirJudzin — Fri, 23 Aug 2024 06:42:27 GMT

I got the same issue starting the last evening, nothing has changed today. I've just looked into the environment libraries, changed the version of libraries to the newer one and published it (in my case it was ms-graph). After it, it started to work. It's really annoying when such an issue is happening in your production environment.

Re: Notebook Error: SparkCoreError/PersonalizingFailed: Livy session has failed

m98ateusz — Fri, 23 Aug 2024 07:11:20 GMT

I just tried to update library ( azure_cli from version 2.61.0 to 2.63.0 ) by adding new .whl file to my environement. Now I got an issue during publishing : Unknown error occurred while publishing.

In my case your resulution doesn't work @SirJudzin

Re: Notebook Error: SparkCoreError/PersonalizingFailed: Livy session has failed

Anonymous — Fri, 23 Aug 2024 09:34:12 GMT

Hi @m98ateusz

Does your environment run on Runtime 1.3? Recently there is a known issue related to Runtime 1.3 when custom libraries are installed. Known issue - Fabric Runtime 1.3 causes invalid libraries - Microsoft Fabric | Microsoft Learn

Best Regards,
Jing

Re: Notebook Error: SparkCoreError/PersonalizingFailed: Livy session has failed

m98ateusz — Fri, 23 Aug 2024 09:38:56 GMT

Hi,

I did once again thing which @SirJudzin did.

First time I planned to do it in two steps :

1.) Add new version of library (whl file)

2.) Delete old version of library (whl file).

However it failed during first step, so I decide to do it in one step :

1.) Add new version of library and delete old old version

and it worked.

My runtime version is 1.2