Forum Discussion
Start spark session instantly in private link
Is there any way we can use custom pools , to create a notebook spark session instanlty (inside privatelink),
we are in a organization that uses privatelink in fabric tenent, but we need to spark notebooks to start with less time. Is there any way, we can create a session ready using custom pools or some other mechanism to start it within seconds when pipeline is triggered.
Hi Elavarasu,
Even with the new Proactive Resource Provisioning feature described in the Microsoft article:
- Fabric automatically pre-warms Spark resources, but…
- You cannot control it, force it, or create your own custom pool
- Private Link scenarios still add latency, because resources must be validated inside your private network.So:
You cannot start Spark fully instantly in Private Link, but you can leverage the new Fabric optimizations to reduce cold-start times — automatically.The blog describes Fabric’s internal capability called:
- Proactive Resource Provisioning
Fabric now analyzes historical usage of Spark in each workspace and proactively allocates compute before you run your next notebook.
This means:
If you frequently run Spark at 8AM — Fabric will pre-provision compute around 7:55.
If you run Spark ad-hoc all day — Fabric may keep resources warmer for longer.
If you stop using Spark — Fabric will stop pre-provisioning to save cost.
This makes Spark notebooks “feel more instant” — but NOT instant for all users and not guaranteed.
Important: This system is fully automated.
You cannot:
manually create a pre-warmed pool
reserve Spark compute
configure warm-up windows
request an always-on cluster
Behavior in Private Link Tenants
The new proactive provisioning still works, but:
Private Link tenants add:Network isolation validation
DNS routing inside the VNet
Container traffic restrictions
These extra steps cannot be pre-warmed.
"Customizing compute configurations will result in on-demand session start experience which could take from 2 - 5 minutes"
References :
- https://learn.microsoft.com/en-us/fabric/security/security-private-links-overview
- https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute
Hope it can help you!
Best regards,
Antoine
3 Replies
- AntoineWSuper User
Hi Elavarasu,
Even with the new Proactive Resource Provisioning feature described in the Microsoft article:
- Fabric automatically pre-warms Spark resources, but…
- You cannot control it, force it, or create your own custom pool
- Private Link scenarios still add latency, because resources must be validated inside your private network.So:
You cannot start Spark fully instantly in Private Link, but you can leverage the new Fabric optimizations to reduce cold-start times — automatically.The blog describes Fabric’s internal capability called:
- Proactive Resource Provisioning
Fabric now analyzes historical usage of Spark in each workspace and proactively allocates compute before you run your next notebook.
This means:
If you frequently run Spark at 8AM — Fabric will pre-provision compute around 7:55.
If you run Spark ad-hoc all day — Fabric may keep resources warmer for longer.
If you stop using Spark — Fabric will stop pre-provisioning to save cost.
This makes Spark notebooks “feel more instant” — but NOT instant for all users and not guaranteed.
Important: This system is fully automated.
You cannot:
manually create a pre-warmed pool
reserve Spark compute
configure warm-up windows
request an always-on cluster
Behavior in Private Link Tenants
The new proactive provisioning still works, but:
Private Link tenants add:Network isolation validation
DNS routing inside the VNet
Container traffic restrictions
These extra steps cannot be pre-warmed.
"Customizing compute configurations will result in on-demand session start experience which could take from 2 - 5 minutes"
References :
- https://learn.microsoft.com/en-us/fabric/security/security-private-links-overview
- https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute
Hope it can help you!
Best regards,
Antoine
- v-dineshyaCommunity Support
Hi Elavarasu ,
Thank you for reaching out to the Microsoft Community Forum.
Hi AntoineW , Thank you for your prompt response.
Hi Elavarasu could you please try the proposed solution shared by AntoineW ? Let us know if you’re still facing the same issue we’ll be happy to assist you further.
Regards,
Dinesh