startup times of the Spark cluster in notebooks

[Nikos83]

Dear Community,

Since last Monday, we have noticed a significant delay in the startup times of the Spark cluster in our notebooks. I have conducted two tests to investigate the issue:

1. Notebook without a custom package: The startup time has increased to over 2 minutes, whereas it used to be very fast (maximum 8 seconds).
2. Notebook using our custom package: The startup time has increased to over 5 minutes.

We are using Runtime 1.3 and have not made any changes to the default settings. Additionally, we are using the trial version with F64 capacity.

Do you recognize this issue, and is there anything that can be done to resolve it?

Looking forward to your response.

Best regards,

Niek

[Anonymous]

Hi @Nikos83,

I wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.
If the community member's answer your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.
If you need any further assistance, feel free to reach out.

Thank you,
Pavan.

[Anonymous]

Hi @Nikos83 

I wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.
If the community member's answer your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.
If you need any further assistance, feel free to reach out.

Thank you,
Pavan.

[Anonymous]

Hi @Nikos83 

Thanks for reaching out in Microsoft Community Forum

To achieve this issue, following step will be helpful

1. please check the cluster settings in autoscaling if enabled, please disable it

If you need any further assistance, please feel free to reach out.

If you found this post helpful, please consider marking it as "Accept as  Solution" to help other members find it more easily.

Thanks,

Pavan.

[Anonymous]

Hi @Nikos83 

I wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.
If the community member's answer your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.
If you need any further assistance, feel free to reach out.
Thank you,
Pavan.

[Srisakthi]

Hi @Nikos83 ,

Yes, private endpoint delay session start time. we have also faced simialr issue.

Are you connecting to data sources which is behind firewall? Because of this reason youhave created private endpoints?

Regards,

Srisakthi

[Nikos83]

Yes, that's the main reason. We tried using Trusted Workspace Access, but we couldn't get it to work

[Srisakthi]

Hi @Nikos83 ,

If possible can you share me the details of error while using TrustedWorkspace ?

Regards,

Srisakthi

[Nikos83]

Yes of course: see: https://community.fabric.microsoft.com/t5/Data-Pipeline/Cross-tenant-trusted-workspace/td-p/4267089

I worked together with @FabianSchut on that issue

[Nikos83]

I hope this is a temporary limitation. Does anyone know?

[Nikos83 · Accepted solution]

I found the root cause: we have configured private endpoints

I hope this is a temporary limitation. Does anyone know?

[Srisakthi]

Hi @Nikos83 ,

Starter pool will start instantly. If you have added any extra spark properties then session start time will be delayed.

Custom pool will take 3 mins to start and if you have added spark properties then again your session start time will be delayed further.

Use custom spark pool with high concurrency session to achieve better resource utilization when same session needs to be shared between users

Are you using starter pool or custom pool?

Regards,

Srisakthi