topic Re: Issues running Spark Notebook in Data Engineering

Issues running Spark Notebook

ronan_b6 — Mon, 15 Jul 2024 18:58:07 GMT

Hi I'm runing spark notebooks on Fabric trial, it's been working great and no time out issues, or severe dalys in sessions starting.

But today, notebooks do not start the spark session, returning this type of error:

"Livy session has failed. Error code: SparkCoreError/SessionDidNotEnterIdle. SessionInfo.State from SparkCore is Error: Session did not enter idle state after 10 minutes. Source: System."

I see from the forum that others have experience similar issues, with severe delays in functionality.

Can anyone shed more light on this?

Thanks,

Re: Issues running Spark Notebook

Anonymous — Tue, 16 Jul 2024 02:57:42 GMT

Hi @ronan_b6 ,

May I ask if you are still facing this issue?
You can try to restart the spark pool to resolve the issue and make sure that the spark is configured correctly and there are enough available resources in the spark pool.
If the issue is still not resolved, can you provide more details like in which region you are facing this issue and attach a screenshot of the issue.

Best Regards,

Ada Wang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Issues running Spark Notebook

maxmerchant — Tue, 16 Jul 2024 15:01:11 GMT

I, too, am encountering this issue since yesterday. We were able to run notebooks without any issues until last week and started seeing the error below. Tried several different notebooks in different workspaces and all give the same error.

"Livy session has failed. Error code: SparkCoreError/SessionDidNotEnterIdle. SessionInfo.State from SparkCore is Error: Session did not enter idle state after 10 minutes. Source: System"

We are using Fabric F64 instance. Region Canada Central.

Is anyone else having this issue as well?

Re: Issues running Spark Notebook

naveesupply — Tue, 16 Jul 2024 17:13:45 GMT

Hi Ada,

How would I restart the spark pool? I am facing the same issue on my trial, I am also based out of Canada

Re: Issues running Spark Notebook

naveesupply — Tue, 16 Jul 2024 17:20:41 GMT

Same issue, were you able to fix it?

Re: Issues running Spark Notebook

dmedeiros — Tue, 16 Jul 2024 17:44:11 GMT

I am also facing the same issue since yesterday both using the trial capacity and a reserved capacity - both from Canada Central

Re: Issues running Spark Notebook

maxmerchant — Tue, 16 Jul 2024 18:08:26 GMT

No. Still facing this issue. The weird thing is that it Livy session starts every once in a while, with no consistency. So can't execute a pipeline with a notebook activity.

Re: Issues running Spark Notebook

naveesupply — Tue, 16 Jul 2024 19:39:43 GMT

Ya I am just running a script using the notebook but its not connecting at all. Will have to wait to see if it works tomorrow.

Re: Issues running Spark Notebook

ToughItOut — Tue, 16 Jul 2024 20:36:09 GMT

My organization is encountering this issue today as well, also in Canada Central.
Ive submitted a support request to Unified Support, will update if I learn more.

Re: Issues running Spark Notebook

Anonymous — Wed, 17 Jul 2024 03:02:13 GMT

Hi all,

We have reported this issue and submitted it to the team.
They have been aware of the issue and the engineers will do their best to resolve it. So please be patient.

Best Regards,

Ada Wang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Issues running Spark Notebook

higgy7 — Wed, 17 Jul 2024 08:24:01 GMT

I am also having the same issue, pipelines with a notebook element fail at the notebook stage with the same error as above.

When i try to start a session within a notebook it tries to start for 10 minutes and then fails to start and disconnects. This issue only started yesterday.

Re: Issues running Spark Notebook

danieletiles — Wed, 17 Jul 2024 09:34:58 GMT

Facing the same issue, North Europe

Re: Issues running Spark Notebook

MMS_007 — Wed, 17 Jul 2024 09:39:52 GMT

Same issue for me. Cannot run pipelines with notebooks (any useful error information), but also cannot run the notebook itself - after around 10 minutes it returns:
SparkCoreError/SessionDidNotEnterIdle: Livy session has failed. Error code: SparkCoreError/SessionDidNotEnterIdle. SessionInfo.State from SparkCore is Error: Session did not enter idle state after 10 minutes. Source: System.

For me it started around 4PM CET yesterday.

In About Power BI section I can see: Your data is stored in North Europe (Ireland) so I guess that's my region (I'm not an admin). It's not a trial but paid Fabric license.

@Anonymous Ada, what is this issue, what can be done about it, when it will start working?
I cannot develop nor refresh the data in any of workspaces I'm responsible for... It needs to be fixed urgently

Re: Issues running Spark Notebook

abm_chris — Wed, 17 Jul 2024 10:52:54 GMT

North Europe, now having the same issue, seem to be running ok for me from 10-11am BST

Re: Issues running Spark Notebook

tlanza89_RWL — Wed, 17 Jul 2024 11:31:01 GMT

I'm having the same issue in the North Central US region as well. The error happens when running a notebook in a pipeline.

"Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Failed to create Livy session for executing notebook"

Re: Issues running Spark Notebook

Anonymous — Wed, 17 Jul 2024 12:05:22 GMT

Facing the same issue with a F64 Capacity in North Europe.

Re: Issues running Spark Notebook

WhiteSox77 — Wed, 17 Jul 2024 12:15:39 GMT

Issue just started last 24 hours, North Europe (UK) region

Re: Issues running Spark Notebook

dver344 — Wed, 17 Jul 2024 12:24:32 GMT

Having a similar issue. It's not timing out, since I don't do a lot of complex processes, but the spark pools usually takes 3 to 5 min to start (before the issue this was 10sec). This all started 24h ago for me (time of writing is 17/07/2024, 2PM Brussels)

Re: Issues running Spark Notebook

JFTxJ — Wed, 17 Jul 2024 12:34:14 GMT

Can you please clarify how to "restart the spark pool"?

Thank you

Re: Issues running Spark Notebook

jonocrotz — Wed, 17 Jul 2024 12:56:40 GMT

Also having the same issue, as mentioned above. No outage or awareness on the service so very frustrating that we are facing an issue for so long yet no acknowledgement by Microsoft