- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
startup times of the Spark cluster in notebooks
Dear Community,
Since last Monday, we have noticed a significant delay in the startup times of the Spark cluster in our notebooks. I have conducted two tests to investigate the issue:
- Notebook without a custom package: The startup time has increased to over 2 minutes, whereas it used to be very fast (maximum 8 seconds).
- Notebook using our custom package: The startup time has increased to over 5 minutes.
We are using Runtime 1.3 and have not made any changes to the default settings. Additionally, we are using the trial version with F64 capacity.
Do you recognize this issue, and is there anything that can be done to resolve it?
Looking forward to your response.
Best regards,
Niek
Solved! Go to Solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I found the root cause: we have configured private endpoints
I hope this is a temporary limitation. Does anyone know?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Nikos83,
I wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.
If the community member's answer your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.
If you need any further assistance, feel free to reach out.
Thank you,
Pavan.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Nikos83
I wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.
If the community member's answer your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.
If you need any further assistance, feel free to reach out.
Thank you,
Pavan.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Nikos83
Thanks for reaching out in Microsoft Community Forum
To achieve this issue, following step will be helpful
- please check the cluster settings in autoscaling if enabled, please disable it
If you need any further assistance, please feel free to reach out.
If you found this post helpful, please consider marking it as "Accept as Solution" to help other members find it more easily.
Thanks,
Pavan.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Nikos83
I wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.
If the community member's answer your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.
If you need any further assistance, feel free to reach out.
Thank you,
Pavan.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Nikos83 ,
Yes, private endpoint delay session start time. we have also faced simialr issue.
Are you connecting to data sources which is behind firewall? Because of this reason youhave created private endpoints?
Regards,
Srisakthi
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes, that's the main reason. We tried using Trusted Workspace Access, but we couldn't get it to work
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Nikos83 ,
If possible can you share me the details of error while using TrustedWorkspace ?
Regards,
Srisakthi
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes of course: see: https://community.fabric.microsoft.com/t5/Data-Pipeline/Cross-tenant-trusted-workspace/td-p/4267089
I worked together with @FabianSchut on that issue
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I hope this is a temporary limitation. Does anyone know?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I found the root cause: we have configured private endpoints
I hope this is a temporary limitation. Does anyone know?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Nikos83 ,
Starter pool will start instantly. If you have added any extra spark properties then session start time will be delayed.
Custom pool will take 3 mins to start and if you have added spark properties then again your session start time will be delayed further.
Use custom spark pool with high concurrency session to achieve better resource utilization when same session needs to be shared between users
Are you using starter pool or custom pool?
Regards,
Srisakthi

Helpful resources
Subject | Author | Posted | |
---|---|---|---|
07-15-2024 11:58 AM | |||
02-01-2024 12:09 AM | |||
01-14-2025 05:18 AM | |||
01-27-2025 08:27 PM | |||
04-07-2024 05:17 PM |
User | Count |
---|---|
11 | |
6 | |
4 | |
2 | |
2 |
User | Count |
---|---|
10 | |
9 | |
7 | |
7 | |
5 |