Forum Discussion

Elavarasu's avatar
Elavarasu
New Member
8 months ago
Solved

Start spark session instantly in private link

Is there any way we can use custom pools , to create a notebook spark session instanlty (inside privatelink)
we are in a organization that uses privatelink in fabric tenent, but we need to spark notebooks to start with less time. Is there any way, we can create a session ready using custom pools or some other mechanism to start it within seconds when pipeline is triggered.

  • Hi Elavarasu,

     

    Even with the new Proactive Resource Provisioning feature described in the Microsoft article:

    Fabric automatically pre-warms Spark resources, but…
    You cannot control it, force it, or create your own custom pool
    Private Link scenarios still add latency, because resources must be validated inside your private network.

    So:
    You cannot start Spark fully instantly in Private Link, but you can leverage the new Fabric optimizations to reduce cold-start times — automatically.

     

    The blog describes Fabric’s internal capability called:

    Proactive Resource Provisioning

    Fabric now analyzes historical usage of Spark in each workspace and proactively allocates compute before you run your next notebook.

    This means:

    • If you frequently run Spark at 8AM — Fabric will pre-provision compute around 7:55.

    • If you run Spark ad-hoc all day — Fabric may keep resources warmer for longer.

    • If you stop using Spark — Fabric will stop pre-provisioning to save cost.

    This makes Spark notebooks “feel more instant” — but NOT instant for all users and not guaranteed.

    Important: This system is fully automated.

     

    You cannot:

    • manually create a pre-warmed pool

    • reserve Spark compute

    • configure warm-up windows

    • request an always-on cluster

    Behavior in Private Link Tenants

    The new proactive provisioning still works, but:
    Private Link tenants add:

    1. Network isolation validation

    2. DNS routing inside the VNet

    3. Container traffic restrictions

    These extra steps cannot be pre-warmed.

     

    "Customizing compute configurations will result in on-demand session start experience which could take from 2 - 5 minutes"

     

    References : 

    https://learn.microsoft.com/en-us/fabric/security/security-private-links-overview

    https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute

     

    Hope it can help you!

    Best regards,

    Antoine

3 Replies

  • Hi Elavarasu,

     

    Even with the new Proactive Resource Provisioning feature described in the Microsoft article:

    Fabric automatically pre-warms Spark resources, but…
    You cannot control it, force it, or create your own custom pool
    Private Link scenarios still add latency, because resources must be validated inside your private network.

    So:
    You cannot start Spark fully instantly in Private Link, but you can leverage the new Fabric optimizations to reduce cold-start times — automatically.

     

    The blog describes Fabric’s internal capability called:

    Proactive Resource Provisioning

    Fabric now analyzes historical usage of Spark in each workspace and proactively allocates compute before you run your next notebook.

    This means:

    • If you frequently run Spark at 8AM — Fabric will pre-provision compute around 7:55.

    • If you run Spark ad-hoc all day — Fabric may keep resources warmer for longer.

    • If you stop using Spark — Fabric will stop pre-provisioning to save cost.

    This makes Spark notebooks “feel more instant” — but NOT instant for all users and not guaranteed.

    Important: This system is fully automated.

     

    You cannot:

    • manually create a pre-warmed pool

    • reserve Spark compute

    • configure warm-up windows

    • request an always-on cluster

    Behavior in Private Link Tenants

    The new proactive provisioning still works, but:
    Private Link tenants add:

    1. Network isolation validation

    2. DNS routing inside the VNet

    3. Container traffic restrictions

    These extra steps cannot be pre-warmed.

     

    "Customizing compute configurations will result in on-demand session start experience which could take from 2 - 5 minutes"

     

    References : 

    https://learn.microsoft.com/en-us/fabric/security/security-private-links-overview

    https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute

     

    Hope it can help you!

    Best regards,

    Antoine

    • v-dineshya's avatar
      v-dineshya
      Community Support

      Hi Elavarasu ,

      Thank you for reaching out to the Microsoft Community Forum.

       

      Hi AntoineW , Thank you for your prompt response.

       

      Hi Elavarasu  could you please try the proposed solution shared by  AntoineW   ? Let us know if you’re still facing the same issue we’ll be happy to assist you further.

       

      Regards,

      Dinesh

  • Thank you for your response AntoineW , seems for now we cannot have custom pre-warmed pools. So there will always adds up start time when private link enabled.