Reply
Nikos83
Frequent Visitor

startup times of the Spark cluster in notebooks

Dear Community,

Since last Monday, we have noticed a significant delay in the startup times of the Spark cluster in our notebooks. I have conducted two tests to investigate the issue:

  1. Notebook without a custom package: The startup time has increased to over 2 minutes, whereas it used to be very fast (maximum 8 seconds).
  2. Notebook using our custom package: The startup time has increased to over 5 minutes.

We are using Runtime 1.3 and have not made any changes to the default settings. Additionally, we are using the trial version with F64 capacity.

Do you recognize this issue, and is there anything that can be done to resolve it?

Looking forward to your response.

Best regards,

 

Niek

1 ACCEPTED SOLUTION
Nikos83
Frequent Visitor

Nikos83_0-1734516773419.png

I found the root cause: we have configured private endpoints

I hope this is a temporary limitation. Does anyone know?

 

View solution in original post

11 REPLIES 11
v-pbandela-msft
Community Support
Community Support

Hi @Nikos83,

I wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.
If the community member's answer your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.
If you need any further assistance, feel free to reach out.

Thank you,
Pavan.

v-pbandela-msft
Community Support
Community Support

Hi @Nikos83 

I wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.
If the community member's answer your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.
If you need any further assistance, feel free to reach out.

Thank you,
Pavan.

v-pbandela-msft
Community Support
Community Support

Hi @Nikos83 

 

Thanks for reaching out in Microsoft Community Forum

 

To achieve this issue, following step will be helpful

  1. please check the cluster settings in autoscaling if enabled, please disable it

vpbandelamsft_0-1734519359208.png

 


If you need any further assistance, please feel free to reach out.

If you found this post helpful, please consider marking it as "Accept as  Solution" to help other members find it more easily.

 

Thanks,

Pavan.

Hi @Nikos83 

I wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.
If the community member's answer your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.
If you need any further assistance, feel free to reach out.
Thank you,
Pavan.

Srisakthi
Resolver II
Resolver II

Hi @Nikos83 ,

 

Yes, private endpoint delay session start time. we have also faced simialr issue.

Are you connecting to data sources which is behind firewall? Because of this reason youhave created private endpoints?

 

Regards,

Srisakthi

Yes, that's the main reason. We tried using Trusted Workspace Access, but we couldn't get it to work

Hi @Nikos83 ,

 

If possible can you share me the details of error while using TrustedWorkspace ?

 

Regards,

Srisakthi

Nikos83
Frequent Visitor

I hope this is a temporary limitation. Does anyone know?

Nikos83
Frequent Visitor

Nikos83_0-1734516773419.png

I found the root cause: we have configured private endpoints

I hope this is a temporary limitation. Does anyone know?

 

Srisakthi
Resolver II
Resolver II

Hi @Nikos83 ,

 

Starter pool will start instantly. If you have added any extra spark properties then session start time will be delayed.

Custom pool will take 3 mins to start and if you have added spark properties then again your session start time will be delayed further.

Use custom spark pool with high concurrency session to achieve better resource utilization when same session needs to be shared between users

 

Are you using starter pool or custom pool?

 

Regards,

Srisakthi

avatar user

Helpful resources

Announcements
MarchFBCvideo - carousel

Fabric Monthly Update - March 2025

Check out the March 2025 Fabric update to learn about new features.

March2025 Carousel

Fabric Community Update - March 2025

Find out what's new and trending in the Fabric community.

Top Solution Authors (Last Month)
Top Kudoed Authors (Last Month)