Start spark session instantly in private link

[Elavarasu]

Is there any way we can use custom pools , to create a notebook spark session instanlty (inside privatelink), 
we are in a organization that uses privatelink in fabric tenent, but we need to spark notebooks to start with less time. Is there any way, we can create a session ready using custom pools or some other mechanism to start it within seconds when pipeline is triggered.

[Elavarasu]

Thank you for your response @AntoineW , seems for now we cannot have custom pre-warmed pools. So there will always adds up start time when private link enabled.

[AntoineW · Accepted solution]

Hi @Elavarasu,

Even with the new Proactive Resource Provisioning feature described in the Microsoft article:

- Fabric automatically pre-warms Spark resources, but…
- You cannot control it, force it, or create your own custom pool
- Private Link scenarios still add latency, because resources must be validated inside your private network.

So:
You cannot start Spark fully instantly in Private Link, but you can leverage the new Fabric optimizations to reduce cold-start times — automatically.

The blog describes Fabric’s internal capability called:

- Proactive Resource Provisioning

Fabric now analyzes historical usage of Spark in each workspace and proactively allocates compute before you run your next notebook.

This means:

• If you frequently run Spark at 8AM — Fabric will pre-provision compute around 7:55.

• If you run Spark ad-hoc all day — Fabric may keep resources warmer for longer.

• If you stop using Spark — Fabric will stop pre-provisioning to save cost.

This makes Spark notebooks “feel more instant” — but NOT instant for all users and not guaranteed.

Important: This system is fully automated.

You cannot:

• manually create a pre-warmed pool

• reserve Spark compute

• configure warm-up windows

• request an always-on cluster

Behavior in Private Link Tenants

The new proactive provisioning still works, but:
Private Link tenants add:

1. Network isolation validation

2. DNS routing inside the VNet

3. Container traffic restrictions

These extra steps cannot be pre-warmed.

"Customizing compute configurations will result in on-demand session start experience which could take from 2 - 5 minutes"

References : 

- https://learn.microsoft.com/en-us/fabric/security/security-private-links-overview

- https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute

Hope it can help you!

Best regards,

Antoine

[v-dineshya]

Hi @Elavarasu ,

Thank you for reaching out to the Microsoft Community Forum.

Hi @AntoineW , Thank you for your prompt response.

Hi @Elavarasu  could you please try the proposed solution shared by  @AntoineW   ? Let us know if you’re still facing the same issue we’ll be happy to assist you further.

Regards,

Dinesh