Forum Discussion
Start spark session instantly in private link
- 8 months ago
Hi Elavarasu,
Even with the new Proactive Resource Provisioning feature described in the Microsoft article:
- Fabric automatically pre-warms Spark resources, but…
- You cannot control it, force it, or create your own custom pool
- Private Link scenarios still add latency, because resources must be validated inside your private network.So:
You cannot start Spark fully instantly in Private Link, but you can leverage the new Fabric optimizations to reduce cold-start times — automatically.The blog describes Fabric’s internal capability called:
- Proactive Resource Provisioning
Fabric now analyzes historical usage of Spark in each workspace and proactively allocates compute before you run your next notebook.
This means:
If you frequently run Spark at 8AM — Fabric will pre-provision compute around 7:55.
If you run Spark ad-hoc all day — Fabric may keep resources warmer for longer.
If you stop using Spark — Fabric will stop pre-provisioning to save cost.
This makes Spark notebooks “feel more instant” — but NOT instant for all users and not guaranteed.
Important: This system is fully automated.
You cannot:
manually create a pre-warmed pool
reserve Spark compute
configure warm-up windows
request an always-on cluster
Behavior in Private Link Tenants
The new proactive provisioning still works, but:
Private Link tenants add:Network isolation validation
DNS routing inside the VNet
Container traffic restrictions
These extra steps cannot be pre-warmed.
"Customizing compute configurations will result in on-demand session start experience which could take from 2 - 5 minutes"
References :
- https://learn.microsoft.com/en-us/fabric/security/security-private-links-overview
- https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute
Hope it can help you!
Best regards,
Antoine
Hi Elavarasu,
Even with the new Proactive Resource Provisioning feature described in the Microsoft article:
- Fabric automatically pre-warms Spark resources, but…
- You cannot control it, force it, or create your own custom pool
- Private Link scenarios still add latency, because resources must be validated inside your private network.
So:
You cannot start Spark fully instantly in Private Link, but you can leverage the new Fabric optimizations to reduce cold-start times — automatically.
The blog describes Fabric’s internal capability called:
- Proactive Resource Provisioning
Fabric now analyzes historical usage of Spark in each workspace and proactively allocates compute before you run your next notebook.
This means:
If you frequently run Spark at 8AM — Fabric will pre-provision compute around 7:55.
If you run Spark ad-hoc all day — Fabric may keep resources warmer for longer.
If you stop using Spark — Fabric will stop pre-provisioning to save cost.
This makes Spark notebooks “feel more instant” — but NOT instant for all users and not guaranteed.
Important: This system is fully automated.
You cannot:
manually create a pre-warmed pool
reserve Spark compute
configure warm-up windows
request an always-on cluster
Behavior in Private Link Tenants
The new proactive provisioning still works, but:
Private Link tenants add:
Network isolation validation
DNS routing inside the VNet
Container traffic restrictions
These extra steps cannot be pre-warmed.
"Customizing compute configurations will result in on-demand session start experience which could take from 2 - 5 minutes"
References :
- https://learn.microsoft.com/en-us/fabric/security/security-private-links-overview
- https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute
Hope it can help you!
Best regards,
Antoine
- v-dineshya8 months ago
Community Support
Hi Elavarasu ,
Thank you for reaching out to the Microsoft Community Forum.
Hi AntoineW , Thank you for your prompt response.
Hi Elavarasu could you please try the proposed solution shared by AntoineW ? Let us know if you’re still facing the same issue we’ll be happy to assist you further.
Regards,
Dinesh