Forum Discussion
Notebook Error: SparkCoreError/PersonalizingFailed: Livy session has failed
We started receiving this error when running Notebooks within the last day:
SparkCoreError/PersonalizingFailed: Livy session has failed. Error code: SparkCoreError/PersonalizingFailed. SessionInfo.State from SparkCore is Error: Encountered an unexpected error while personalizing session. Failed step: 'LM_LibraryManagementPersonalizationStatement'. Source: System.
Notebooks were previously running with no issues. We're on a reserved capacity in East US, we do have a public library installed (great-expectations) in the environment.
I'm aware of other users receiving this error in other regions and it looks like Microsoft resolved those issues, but is anyone else getting this error?
11 Replies
Hi mdhewitt83
I searched internally and on the forums, but I haven’t found any other users reporting this error recently. I tested starting a session on a custom environment with the public library (great-expectations) installed without having any error. Sometimes, it might be a temporary problem. You could try again to see if it still occurs.
Additionally, the error you got is more similar to the one in the following thread. If current environment still doesn't work, you can try creating and using a new environment to run the notebook, or try pip install the great-expectations module in notebook directly.
Power BI Lakehouse notebook cannot start session - Microsoft Q&A
Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it.I'm still having this issue, I've tried creating multiple environments in different workspaces. I prefer to use an environment as I'll be using the great expectations library in alot of notebooks so I don't want to install it everytime a notebook runs.
I'm finding that I cannot publish an environment with great-expectations installed in the public libraries unless I install v1.0.0a1 or above - these are prerelease versions. if i go with the current or older versions of the library i cannot publish the enviromnent and I get a generic error.
Anonymous what configuration were you able to get working?
Hi mdhewitt83
Unfortunately I failed to publish an environment with great-expectations 0.18.19 today. I got the following error. I tried PyPI and Conda sources, both failed. But for v1.0.0a1, it succeeded.
However when I published an environment with the same 0.18.19 version last week, it succeeded.
Did you get the same error notification when the publish failed?
Best Regards,
JingHey Anonymous , that is the same error.
Hi mdhewitt83 . I got the same issue today. I have pipeline, which use environment, where we have custom library ( inside we use great expectations ). Before pipeline worked fine and today I got error :
- in pipeline : Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Failed to create Livy session for executing notebook.
- when running notebook itself : Error code: SparkCoreError/PersonalizingFailed.
I got the same issue starting the last evening, nothing has changed today. I've just looked into the environment libraries, changed the version of libraries to the newer one and published it (in my case it was ms-graph). After it, it started to work. It's really annoying when such an issue is happening in your production environment.
