topic Re: Slow starter pool session initialization in Fabric platform

Slow starter pool session initialization

Anonymous — Tue, 31 Oct 2023 14:48:54 GMT

I am currently experimenting with a F2 capacity. Whenever I try to run a notebook it takes somewhere between 2 and 2.5 minutes to start the spark cluster even though I'm using the starter pool. All settings related to Spark are left as default for the workspace. Tried both running things manually as well as triggering the notebook from data factory.

If I go into a trial workspace and run a notebook the spark session is started in just a few seconds. This makes me think that it might be a limitation depending on which capacity you are using? But the documentation gives no indication of this: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute

Re: Slow starter pool session initialization

Anonymous — Wed, 01 Nov 2023 06:24:31 GMT

Hi Anonymous,

Apologies for the issue you have been facing.
We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.

Re: Slow starter pool session initialization

Anonymous — Wed, 01 Nov 2023 17:56:57 GMT

Hi Anonymous ,

Apologies for the issue you have been facing. We have an update from the team.
We got impacted by this limitation on supporting 1 node starter pools mainly for F2 capacity sku. Team have added this to Known Issues section in the document and the latest version should be published end of this week. We are working on a fix to enable 5 second session start experience for F2 capacity. I will update this thread once the fix is rolled out in few weeks.

Thank you

Re: Slow starter pool session initialization

Anonymous — Thu, 02 Nov 2023 07:04:46 GMT

Glad to get a confirmation I'm not doing something wrong and that it will be fixed. Thank you!

Re: Slow starter pool session initialization

Anonymous — Thu, 02 Nov 2023 07:45:58 GMT

Hi Anonymous ,

Glad to know that you got some insights for your query.
Please continue using Fabric Community for any help regarding your queries.

Re: Slow starter pool session initialization

cheresier — Tue, 26 Dec 2023 20:00:48 GMT

Anonymous , has the fix been implemented? It has been ~8 weeks since you said that the fix would be "rolled out in few weeks" and I am still seeing slow startup times on the F2 capacity.

Thanks

Re: Slow starter pool session initialization

Anonymous — Wed, 27 Dec 2023 06:17:07 GMT

Hi cheresier ,

We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.

Re: Slow starter pool session initialization

ramonsuarez — Thu, 04 Jan 2024 10:34:55 GMT

I'm getting even slower times to be able to run a notebook, up to over 8 minutes. On average they start in about 2'45", but now I'm getting more and more startups that take much longer.

Changing to an F4 capacity did not change anything.

This issue is not solved and should be changed so that it reflects that it is still open.

Re: Slow starter pool session initialization

Anonymous — Tue, 09 Jan 2024 07:58:45 GMT

Now there's been another two weeks. Any update on this? Many thanks!

Re: Slow starter pool session initialization

Anonymous — Tue, 09 Jan 2024 11:17:59 GMT

Hi Anonymous ,

No not yet. We have a deployment freeze in December and this is still being worked on. Team is targeting by Q1 for this to be rolled out to all prod regions. Will update this thread once the support for smaller capacities is enabled.

Re: Slow starter pool session initialization

ramonsuarez — Tue, 09 Jan 2024 12:57:57 GMT

Which capacities are affected by this issue right now beyond F2 and F4?

Re: Slow starter pool session initialization

ramonsuarez — Tue, 09 Jan 2024 13:07:33 GMT

It is getting worst for me, sometimes it just fails after a long time of trying to start.

Re: Slow starter pool session initialization

ramonsuarez — Fri, 19 Jan 2024 20:08:34 GMT

I spoke with support yesterday and they claimed that 3 to 4 minutes to start running a notebook is normal... And then when I was demonstrating the start of a notebook it simply failed... 😀

Re: Slow starter pool session initialization

alozovoy — Wed, 24 Jan 2024 00:03:52 GMT

F8 appears to be affected as well

Re: Slow starter pool session initialization

arjaano — Wed, 28 Feb 2024 13:35:07 GMT

This should not be an accepted solution. The issue is still there. I'm on a F4 and facing 2+ minutes startup time.

Re: Slow starter pool session initialization

wiegelman — Wed, 28 Feb 2024 13:43:42 GMT

Agree, this should not be closed - we are also using F4 and the Spark Session is not starting, it is actually timing out for us...

Diagnostics below:

Diagnostic ID: 22a1d13c-d334-4828-8215-dfe8e65dfb60

Timestamp: 2024-02-28T13:38:08.717Z

Message: [object CloseEvent]

JSON
{
"type": "close",
"timeStamp": 731861.099999994,
"code": 1000,
"reason": "{\"reason\":\"Session error or stopped.\",\"state\":\"session-completed\"}",
"wasClean": true,
"target": {
"protocolsProfile": [
7,
4173
]
},
"currentTarget": {
"protocolsProfile": [
7,
4173
]
},
"isTrusted": true
}

Additional info: InstanceId: 2491b595-9a6b-4066-bd0d-0e41152e1754

Re: Slow starter pool session initialization

wiegelman — Wed, 28 Feb 2024 16:20:02 GMT

At least issue is acknowledged - hopefully gets fixed soon

Re: Slow starter pool session initialization

alozovoy — Wed, 28 Feb 2024 16:38:18 GMT

This is a separate issue than the rest of in this thread. If you search the forums for "Livy session has failed" you will see a number of discussions.

Re: Slow starter pool session initialization

Anonymous — Wed, 28 Feb 2024 17:45:01 GMT

I have unmarked previous reply as solution. For me, the main issue here is that we have no proper way of tracking these type of issues. I have reported multiple issues that have been confirmed by a Microsoft but nothing is added to the known issues page: https://support.fabric.microsoft.com/en-us/known-issues - as the page is not updated we cannot really know if the issue has been acknowledged. The page also gives a false impression of few or no issues in Fabric. It is difficult to maintain credibility towards our customers when we don't have transparency from Microsoft.

Re: Slow starter pool session initialization

alozovoy — Wed, 28 Feb 2024 18:02:43 GMT

Agreed! A Microsoft support engineer told me last week that an issue was in their internal known issues tracker, but that it would not be added to the public known issues page. Very frustrating.