Instable enviroment after setting environment objects

[DennesTorres]

Hi,

I had to create an Environment object and set it to the workspaces I'm working on.

The environment is on default configurations except by a spark property I enabled (dynamic partitions).

However, after I set the environment on the workspaces, I'm getting an error message more often:

"SparkCoreError/SessionDidNotEnterIdle: Livy session has failed. Error code: SparkCoreError/SessionDidNotEnterIdle. SessionInfo.State from SparkCore is Error: Session did not enter idle state after 10 minutes. Source: SparkCoreService."

I'm not sure what this means, this usually happens when I'm starting a new execution.

I'm not sure if I should take it to support or how, because I don't know how to reproduce, it's just an error message I'm getting way more often after setting the environment on the workspaces.

Kind Regards,

Dennes

[willmasse]

I'm getting these same exact errors too. Started in the past few days and they happen very frequently. Also using a enviorment but not sure if that was the issue. I had notebooks running fine until the past few days. 

[DennesTorres]

Hi, Will,

It's interesting. The fact you are getting the same errors and you are also using environments points to a pattern. The Environment objects seems to create problems to the start of spark sessions, although we have no further evidence and we can't reproduce, it seems a pattern.

Kind Regards,

Dennes

[willmasse]

Checking in, I haven't gotten these errors in the past few days and my pipelines with notebooks are seemingly running fine now. Wondering if the issue has gone away for you to? 

I'm assuming this is entirely something MS server side they must have quietly addressed? 

[DennesTorres]

Hi,

I just got one:

SparkCoreError/SessionDidNotEnterIdle: Livy session has failed. Error code: SparkCoreError/SessionDidNotEnterIdle. SessionInfo.State from SparkCore is Error: Session did not enter idle state after 10 minutes. Source: SparkCoreService. Referenced notebook stacktrace: at Notebook NB - Load Configuration, cell 2

Kind Regards,

Dennes

[Anonymous]

Hi @DennesTorres , @willmasse  -

Apologies for the issue you have been facing.

This might require a deeper investigation from our engineering team and they can guide you better.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support
Please provide the ticket number here as we can keep an eye on it.

Hope this is helpful. Please let me know incase of further queries.

[Anonymous]

Hello @DennesTorres ,

We haven’t heard from you on the last response and was just checking back to see if you got a chance to raise a support ticket. Once created - please provide the ticket number here as we can keep an eye on it.

Thank You

[DennesTorres]

Hi,

I'm not sure if I can do that. This is not a problem I can reproduce, this happens from time to time, but there is no specific steps to reproduce the problem.

Kind Regards,

Dennes

[Anonymous]

Hi @DennesTorres ,

Apologies for the delay in reply from our side. Did you got a resolution for this?
As you said I created a new environment and added one custom spark configuration for dynamic partition.

I haven't faced any issue while working with notebook execution.

Can you please retry by creating a new environment and execute the notebook?

[Anonymous]

Hi @DennesTorres ,

Thanks for using Fabric Community.

I would be able to guide you if you can explain your issue with few more details.
I am unable to understand the issue based on your information provided. 
Can you please share the documentation link you are following, so I can get better idea over your issue?

[DennesTorres]

Hi,

The issue: 

Recently I'm receiving error messages when starting new spark sessions, running notebook code blocks. The errors are way more frequent than before. 

The error message:

"SparkCoreError/SessionDidNotEnterIdle: Livy session has failed. Error code: SparkCoreError/SessionDidNotEnterIdle. SessionInfo.State from SparkCore is Error: Session did not enter idle state after 10 minutes. Source: SparkCoreService."

Sometimes I don't receive the error, but the session takes many minutes to start.

I can't reproduce, it's just something which is happening more often.

What was changed to cause this?

I created an Environment object using the default options. The only customization on the environment object was one custom spark configuration for dynamic partition. (I doubt the kind of configuration may cause this).

The environment was linked to the workspace on the workspace settings.

After this change, the errors when starting Spark sessions became way more usual.

Kind Regards,

Dennes

[WorkFull22]

I'm also having the same issue do we have any idea on cause?

[Vido434]

I have the same issue, any updates?