Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified as a Fabric Data Engineer: Check your eligibility for a 50% exam voucher offer and join us for free live learning sessions to get prepared for Exam DP-700. Get started

Reply
mdhewitt83
Frequent Visitor

Notebook Error: SparkCoreError/PersonalizingFailed: Livy session has failed

We started receiving this error when running Notebooks within the last day:

 

SparkCoreError/PersonalizingFailed: Livy session has failed. Error code: SparkCoreError/PersonalizingFailed. SessionInfo.State from SparkCore is Error: Encountered an unexpected error while personalizing session. Failed step: 'LM_LibraryManagementPersonalizationStatement'. Source: System.

 

Notebooks were previously running with no issues. We're on a reserved capacity in East US, we do have a public library installed (great-expectations) in the environment.

 

I'm aware of other users receiving this error in other regions and it looks like Microsoft resolved those issues, but is anyone else getting this error?

11 REPLIES 11
m98ateusz
Frequent Visitor

Hi @mdhewitt83 . I got the same issue today. I have pipeline, which use environment, where we have custom library ( inside we use great expectations ). Before pipeline worked fine and today I got error : 

- in pipeline : Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Failed to create Livy session for executing notebook. 

- when running notebook itself : Error code: SparkCoreError/PersonalizingFailed.

I got the same issue starting the last evening, nothing has changed today. I've just looked into the environment libraries, changed the version of libraries to the newer one and published it (in my case it was ms-graph). After it, it started to work. It's really annoying when such an issue is happening in your production environment.

 

I just tried to update library ( azure_cli from version 2.61.0 to 2.63.0 ) by adding new .whl file to my environement. Now I got an issue during publishing : Unknown error occurred while publishing.

In my case your resulution doesn't work @SirJudzin 

Hi @m98ateusz 

 

Does your environment run on Runtime 1.3? Recently there is a known issue related to Runtime 1.3 when custom libraries are installed. Known issue - Fabric Runtime 1.3 causes invalid libraries - Microsoft Fabric | Microsoft Learn

 

Best Regards,
Jing

Hi,

I did once again thing which @SirJudzin did.

First time I planned to do it in two steps : 

1.) Add new version of library (whl file)

2.) Delete old version of library (whl file).

However it failed during first step, so I decide to do it in one step : 

1.) Add new version of library and delete old old version 

and it worked.

My runtime version is 1.2

mdhewitt83
Frequent Visitor

I'm still having this issue, I've tried creating multiple environments in different workspaces. I prefer to use an environment as I'll be using the great expectations library in alot of notebooks so I don't want to install it everytime a notebook runs.

 

I'm finding that I cannot publish an environment with great-expectations installed in the public libraries unless I install v1.0.0a1 or above - these are prerelease versions. if i go with the current or older versions of the library i cannot publish the enviromnent and I get a generic error.

 

@v-jingzhan-msft what configuration were you able to get working?

Hi @mdhewitt83 

 

Unfortunately I failed to publish an environment with great-expectations 0.18.19 today. I got the following error. I tried PyPI and Conda sources, both failed. But for v1.0.0a1, it succeeded. 

vjingzhanmsft_0-1723605444473.png

 

However when I published an environment with the same 0.18.19 version last week, it succeeded. 

vjingzhanmsft_0-1723617301057.png

 

Did you get the same error notification when the publish failed? 

 

Best Regards,
Jing

Hey @v-jingzhan-msft , that is the same error.

Hi @mdhewitt83 

I have escalated the publish failure issue internally but the engineer couldn't reproduce it. It seems not a general issue.

I'd like to confirm with you that whether you can publish an environment with the current release (0.18.19) of great-expectations library at present? If it still failed, you can try to find the error logs. Try publishing the environment again, and you may see below messages. Click View log, then View details. It will generate the logs in a new browser tab. See if there is any error in it. 

vjingzhanmsft_0-1724315628536.jpeg

 

vjingzhanmsft_1-1724315628539.jpeg

 

If you still cannot publish the environment, I suggest creating a support ticket to investigate this further. 

 

Best Regards,
Jing

Hi @mdhewitt83 

 

I checked the error log of my publish failure. It seems the library was installed successfully in a new environment but the environment publish failed due to some error. I haven't found any solution or workaround yet. I will escalate this Publish environment failure issue internally, and update here once I got any feedback.

vjingzhanmsft_0-1723711259058.png

 

For the initial "SparkCoreError/PersonalizingFailed: Livy session has failed." error, I'm not sure if it has anything to do with the great-expectations library in the environment where your notebook run in. I would suggest raising a support ticket at Microsoft Fabric Support and Status | Microsoft Fabric

 

Best Regards,
Jing

v-jingzhan-msft
Community Support
Community Support

Hi @mdhewitt83 

 

I searched internally and on the forums, but I haven’t found any other users reporting this error recently. I tested starting a session on a custom environment with the public library (great-expectations) installed without having any error. Sometimes, it might be a temporary problem. You could try again to see if it still occurs. 

 

Additionally, the error you got is more similar to the one in the following thread. If current environment still doesn't work, you can try creating and using a new environment to run the notebook, or try pip install the great-expectations module in notebook directly. 

Power BI Lakehouse notebook cannot start session - Microsoft Q&A

 

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it.

Helpful resources

Announcements
Feb2025 Sticker Challenge

Join our Community Sticker Challenge 2025

If you love stickers, then you will definitely want to check out our Community Sticker Challenge!

JanFabricDE_carousel

Fabric Monthly Update - January 2025

Explore the power of Python Notebooks in Fabric!

JanFabricDW_carousel

Fabric Monthly Update - January 2025

Unlock the latest Fabric Data Warehouse upgrades!