topic Re: startup times of the Spark cluster in notebooks in Data Engineering

startup times of the Spark cluster in notebooks

Nikos83 — Wed, 18 Dec 2024 06:21:53 GMT

Dear Community,

Since last Monday, we have noticed a significant delay in the startup times of the Spark cluster in our notebooks. I have conducted two tests to investigate the issue:

Notebook without a custom package: The startup time has increased to over 2 minutes, whereas it used to be very fast (maximum 8 seconds).
Notebook using our custom package: The startup time has increased to over 5 minutes.

We are using Runtime 1.3 and have not made any changes to the default settings. Additionally, we are using the trial version with F64 capacity.

Do you recognize this issue, and is there anything that can be done to resolve it?

Looking forward to your response.

Best regards,

Niek

Re: startup times of the Spark cluster in notebooks

Srisakthi — Wed, 18 Dec 2024 10:03:08 GMT

Hi Nikos83 ,

Starter pool will start instantly. If you have added any extra spark properties then session start time will be delayed.

Custom pool will take 3 mins to start and if you have added spark properties then again your session start time will be delayed further.

Use custom spark pool with high concurrency session to achieve better resource utilization when same session needs to be shared between users

Are you using starter pool or custom pool?

Regards,

Srisakthi

Re: startup times of the Spark cluster in notebooks

Nikos83 — Wed, 18 Dec 2024 10:16:38 GMT

I found the root cause: we have configured private endpoints

I hope this is a temporary limitation. Does anyone know?

Re: startup times of the Spark cluster in notebooks

Nikos83 — Wed, 18 Dec 2024 10:16:23 GMT

I hope this is a temporary limitation. Does anyone know?

Re: startup times of the Spark cluster in notebooks

Srisakthi — Wed, 18 Dec 2024 10:28:31 GMT

Hi Nikos83 ,

Yes, private endpoint delay session start time. we have also faced simialr issue.

Are you connecting to data sources which is behind firewall? Because of this reason youhave created private endpoints?

Regards,

Srisakthi

Re: startup times of the Spark cluster in notebooks

Nikos83 — Wed, 18 Dec 2024 10:30:30 GMT

Yes, that's the main reason. We tried using Trusted Workspace Access, but we couldn't get it to work

Re: startup times of the Spark cluster in notebooks

Anonymous — Wed, 18 Dec 2024 10:58:31 GMT

Hi Nikos83

Thanks for reaching out in Microsoft Community Forum

To achieve this issue, following step will be helpful

please check the cluster settings in autoscaling if enabled, please disable it

If you need any further assistance, please feel free to reach out.

If you found this post helpful, please consider marking it as "Accept as Solution" to help other members find it more easily.

Thanks,

Pavan.

Re: startup times of the Spark cluster in notebooks

Srisakthi — Thu, 19 Dec 2024 03:39:48 GMT

Hi Nikos83 ,

If possible can you share me the details of error while using TrustedWorkspace ?

Regards,

Srisakthi

Re: startup times of the Spark cluster in notebooks

Nikos83 — Thu, 19 Dec 2024 06:51:37 GMT

Yes of course: see: https://community.fabric.microsoft.com/t5/Data-Pipeline/Cross-tenant-trusted-workspace/td-p/4267089

I worked together with FabianSchut on that issue

Re: startup times of the Spark cluster in notebooks

Anonymous — Tue, 24 Dec 2024 09:30:45 GMT

Hi Nikos83

I wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.
If the community member's answer your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.
If you need any further assistance, feel free to reach out.
Thank you,
Pavan.

Re: startup times of the Spark cluster in notebooks

Anonymous — Fri, 27 Dec 2024 04:26:41 GMT

Re: startup times of the Spark cluster in notebooks

Anonymous — Tue, 31 Dec 2024 12:30:39 GMT

Hi Nikos83,

I wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.
If the community member's answer your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.
If you need any further assistance, feel free to reach out.

Thank you,
Pavan.