topic Re: Spark Sessions failing to start in Pipelines

Spark Sessions failing to start

jbasiago — Fri, 21 Jun 2024 15:31:01 GMT

Hi All,

I just started running into an issue where my notebook activities in my data pipelines are failing to start. After awhile I will get an error that 'Failed to create Livy session for executing notebook.' I'm not sure what's going on or where to even check to see what might be happening. I've seen previous posts of others having this issue and then it being resolved but no details on what/why it's happening.

Thanks!

-Jonathan

Re: Spark Sessions failing to start

Anonymous — Mon, 24 Jun 2024 09:39:14 GMT

Hi jbasiago ,

Thanks for using Fabric Community.
Can you confirm did you attached lakehouse to your notebook?
Are you still facing this issue? Can you please try now after clearing cache and cookies?

Hope this is helpful. Please do let me know incase of further queries.

Re: Spark Sessions failing to start

Anonymous — Wed, 26 Jun 2024 14:36:23 GMT

Hello jbasiago ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .
Thanks

Re: Spark Sessions failing to start

Anonymous — Wed, 26 Jun 2024 14:36:38 GMT

Hi jbasiago ,

Re: Spark Sessions failing to start

jbasiago — Thu, 27 Jun 2024 14:03:07 GMT

Hi!

Sorry, I didn't see the original post to respond to. I did try clearing my cache and cookies, but that did not help at the time. When you say attaching the lakehouse to the notebook, do you mean having a default lakehouse in the notebook? I have done that. I can run the notebook directly, it was just when trying to run the notebook from a data pipeline that it would fail and give the error that it couldn't start a session.

Things seem to be working now though. Do you have any insight on how to prevent this in the future or at least monitor our Fabric usage more directly? I wonder if there were too many requests at the time I was trying to run my pipeline which caused Fabric's smoothing process to start refusing new sessions....but I can't seem to figure out how to determine how much processing we're using vs what our capacity would allow.

Thank you!

Re: Spark Sessions failing to start

Anonymous — Fri, 28 Jun 2024 05:12:09 GMT

Hi jbasiago ,

Glad to know that you no longer facing this issue. This looks like a temporary issue and may due to many requests at that time.

Incase if you face this issue again, the best course of action is to open a support ticket and have our support team take a closer look at it: Link

Hope this helps. Please let us know if you have any other queries.