topic Failed to create Livy session for executing notebook. in Data Engineering

Failed to create Livy session for executing notebook.

willmasse — Tue, 09 Jan 2024 13:40:08 GMT

Hi,

In Microsoft Fabric, we are gettting the following types of error when running notebooks in a pipeline.

"traceback":["Exception: Failed to create Livy session for executing notebook. LivySessionId: xxxxxxxx ","--> InvalidHttpRequestToLivy: [Unauthorized] Access token validation failed. HTTP status code: 401."]},"lastCheckedOn":"2024-01-09T11:44:39.1533333Z","metadata":null,"exitValue":null},"message":"Notebook execution is in Failed state, ....."

The Access token validation failed is the strange part to me...

Re: Failed to create Livy session for executing notebook.

v-cboorla-msft — Wed, 10 Jan 2024 13:22:34 GMT

Hi @willmasse

Thanks for using Microsoft Fabric Community.

Apologies for the issue that you are facing here.

The 401 Unauthorized Error is an HTTP status code error that represented the request sent by the client to the server that lacks valid authentication credentials.

Methods to rectify the error: The 401 Unauthorized error can be fixed by using any of the following ways:

Check The URL: Due to manual errors in typing the URL, the 401 unauthorized error may occur. Hence, checking the URL and rectifying the mistakes in it will fix the 401 error status.
Flush the DNS: Errors in DNS also creates 401 error status sometimes. Therefore, clearing the DNS will also rectify this error. In Windows, the DNS can be flushed by typing ipconfig/flushdns in the command prompt and clicking on ENTER.
Clear Browser Cookie: In some situations, the cookies may not work smoothly leading to improper server authentication. Thus, by clearing the cookies, the error can be rectified.
Logging out and Logging in again: If you're authenticated via a session (like in a web app), it might have expired. Therefore, logging out and logging in again by providing the credentials may also rectify this error.

I hope this information helps. Please do let us know if you have any further questions.

Re: Failed to create Livy session for executing notebook.

willmasse — Wed, 10 Jan 2024 13:30:49 GMT

These are Notebooks running in Pipelines. None of those things are applicable.

I also get these errors as well: SparkCoreError/SessionDidNotEnterIdle: Livy session has failed. Error code: SparkCoreError/SessionDidNotEnterIdle. SessionInfo.State from SparkCore is Error: Session did not enter idle state after 10 minutes.

I'm getting really frequent errors running Notebooks in Pipelines that all revolve around Livy session.

Re: Failed to create Livy session for executing notebook.

v-cboorla-msft — Thu, 18 Jan 2024 11:45:02 GMT

Hi @willmasse

Apologies for the delay in response.

This seems to be an intermittent issue, can you please try to rerun the pipelines?

If the issue still persists please do let us know.

Thanks

Re: Failed to create Livy session for executing notebook.

v-cboorla-msft — Mon, 22 Jan 2024 05:22:12 GMT

Hi @willmasse

Thanks

Re: Failed to create Livy session for executing notebook.

v-cboorla-msft — Tue, 23 Jan 2024 13:08:43 GMT

Hi @willmasse

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread.

Thanks

Re: Failed to create Livy session for executing notebook.

alozovoy — Wed, 24 Jan 2024 00:14:13 GMT

I have run into this error on many occasions and it can often be solved by limiting the number of notebooks running in parallel. It seems to hit a limit of how many notebooks can run at once. Instead of waiting for compute to be available, it just fails.

But I've also had this happen when only running a single notebook...

Re: Failed to create Livy session for executing notebook.

Anonymous — Fri, 09 Feb 2024 07:01:49 GMT

We've been getting the Livy Session did not enter idle state after 10 minutes Error intermittently. our pipelines tend to run notebooks sequentially, but there can be an overlap once in a while. But.... our notebooks do very little processing. they normally take 3-10s to complete. so how contention is causing this is baffling me.

Re: Failed to create Livy session for executing notebook.

PeterGoertz — Mon, 24 Jun 2024 18:39:17 GMT

We face a similar problem. Notebook alone needs about 30s to successfully complete. Called in a pipeline it needs over 10 Minutes to return the error mentioned above.

Re: Failed to create Livy session for executing notebook.

PeterGoertz — Mon, 24 Jun 2024 18:51:36 GMT

As a temporarily shortcut a capacity restart works.