Microsoft Fabric Community Conference 2025, March 31 - April 2, Las Vegas, Nevada. Use code MSCUST for a $150 discount.
Register nowGet certified as a Fabric Data Engineer: Check your eligibility for a 50% exam voucher offer and join us for free live learning sessions to get prepared for Exam DP-700. Get started
We started receiving this error when running Notebooks within the last day:
SparkCoreError/PersonalizingFailed: Livy session has failed. Error code: SparkCoreError/PersonalizingFailed. SessionInfo.State from SparkCore is Error: Encountered an unexpected error while personalizing session. Failed step: 'LM_LibraryManagementPersonalizationStatement'. Source: System.
Notebooks were previously running with no issues. We're on a reserved capacity in East US, we do have a public library installed (great-expectations) in the environment.
I'm aware of other users receiving this error in other regions and it looks like Microsoft resolved those issues, but is anyone else getting this error?
Hi @mdhewitt83 . I got the same issue today. I have pipeline, which use environment, where we have custom library ( inside we use great expectations ). Before pipeline worked fine and today I got error :
- in pipeline : Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Failed to create Livy session for executing notebook.
- when running notebook itself : Error code: SparkCoreError/PersonalizingFailed.
I got the same issue starting the last evening, nothing has changed today. I've just looked into the environment libraries, changed the version of libraries to the newer one and published it (in my case it was ms-graph). After it, it started to work. It's really annoying when such an issue is happening in your production environment.
I just tried to update library ( azure_cli from version 2.61.0 to 2.63.0 ) by adding new .whl file to my environement. Now I got an issue during publishing : Unknown error occurred while publishing.
In my case your resulution doesn't work @SirJudzin
Hi @m98ateusz
Does your environment run on Runtime 1.3? Recently there is a known issue related to Runtime 1.3 when custom libraries are installed. Known issue - Fabric Runtime 1.3 causes invalid libraries - Microsoft Fabric | Microsoft Learn
Best Regards,
Jing
Hi,
I did once again thing which @SirJudzin did.
First time I planned to do it in two steps :
1.) Add new version of library (whl file)
2.) Delete old version of library (whl file).
However it failed during first step, so I decide to do it in one step :
1.) Add new version of library and delete old old version
and it worked.
My runtime version is 1.2
I'm still having this issue, I've tried creating multiple environments in different workspaces. I prefer to use an environment as I'll be using the great expectations library in alot of notebooks so I don't want to install it everytime a notebook runs.
I'm finding that I cannot publish an environment with great-expectations installed in the public libraries unless I install v1.0.0a1 or above - these are prerelease versions. if i go with the current or older versions of the library i cannot publish the enviromnent and I get a generic error.
@v-jingzhan-msft what configuration were you able to get working?
Hi @mdhewitt83
Unfortunately I failed to publish an environment with great-expectations 0.18.19 today. I got the following error. I tried PyPI and Conda sources, both failed. But for v1.0.0a1, it succeeded.
However when I published an environment with the same 0.18.19 version last week, it succeeded.
Did you get the same error notification when the publish failed?
Best Regards,
Jing
Hi @mdhewitt83
I have escalated the publish failure issue internally but the engineer couldn't reproduce it. It seems not a general issue.
I'd like to confirm with you that whether you can publish an environment with the current release (0.18.19) of great-expectations library at present? If it still failed, you can try to find the error logs. Try publishing the environment again, and you may see below messages. Click View log, then View details. It will generate the logs in a new browser tab. See if there is any error in it.
If you still cannot publish the environment, I suggest creating a support ticket to investigate this further.
Best Regards,
Jing
Hi @mdhewitt83
I checked the error log of my publish failure. It seems the library was installed successfully in a new environment but the environment publish failed due to some error. I haven't found any solution or workaround yet. I will escalate this Publish environment failure issue internally, and update here once I got any feedback.
For the initial "SparkCoreError/PersonalizingFailed: Livy session has failed." error, I'm not sure if it has anything to do with the great-expectations library in the environment where your notebook run in. I would suggest raising a support ticket at Microsoft Fabric Support and Status | Microsoft Fabric
Best Regards,
Jing
Hi @mdhewitt83
I searched internally and on the forums, but I haven’t found any other users reporting this error recently. I tested starting a session on a custom environment with the public library (great-expectations) installed without having any error. Sometimes, it might be a temporary problem. You could try again to see if it still occurs.
Additionally, the error you got is more similar to the one in the following thread. If current environment still doesn't work, you can try creating and using a new environment to run the notebook, or try pip install the great-expectations module in notebook directly.
Power BI Lakehouse notebook cannot start session - Microsoft Q&A
Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it.