Forum Discussion

Elavarasu's avatar
Elavarasu
New Member
8 months ago
Solved

Start spark session instantly in private link

Is there any way we can use custom pools , to create a notebook spark session instanlty (inside privatelink),  we are in a organization that uses privatelink in fabric tenent, but we need to spark n...
  • AntoineW's avatar
    8 months ago

    Hi Elavarasu,

     

    Even with the new Proactive Resource Provisioning feature described in the Microsoft article:

    Fabric automatically pre-warms Spark resources, but…
    You cannot control it, force it, or create your own custom pool
    Private Link scenarios still add latency, because resources must be validated inside your private network.

    So:
    You cannot start Spark fully instantly in Private Link, but you can leverage the new Fabric optimizations to reduce cold-start times — automatically.

     

    The blog describes Fabric’s internal capability called:

    Proactive Resource Provisioning

    Fabric now analyzes historical usage of Spark in each workspace and proactively allocates compute before you run your next notebook.

    This means:

    • If you frequently run Spark at 8AM — Fabric will pre-provision compute around 7:55.

    • If you run Spark ad-hoc all day — Fabric may keep resources warmer for longer.

    • If you stop using Spark — Fabric will stop pre-provisioning to save cost.

    This makes Spark notebooks “feel more instant” — but NOT instant for all users and not guaranteed.

    Important: This system is fully automated.

     

    You cannot:

    • manually create a pre-warmed pool

    • reserve Spark compute

    • configure warm-up windows

    • request an always-on cluster

    Behavior in Private Link Tenants

    The new proactive provisioning still works, but:
    Private Link tenants add:

    1. Network isolation validation

    2. DNS routing inside the VNet

    3. Container traffic restrictions

    These extra steps cannot be pre-warmed.

     

    "Customizing compute configurations will result in on-demand session start experience which could take from 2 - 5 minutes"

     

    References : 

    https://learn.microsoft.com/en-us/fabric/security/security-private-links-overview

    https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute

     

    Hope it can help you!

    Best regards,

    Antoine