Forum Discussion

Elavarasu's avatar
Elavarasu
New Member
9 months ago
Solved

Start spark session instantly in private link

Is there any way we can use custom pools , to create a notebook spark session instanlty (inside privatelink), 
we are in a organization that uses privatelink in fabric tenent, but we need to spark notebooks to start with less time. Is there any way, we can create a session ready using custom pools or some other mechanism to start it within seconds when pipeline is triggered.

  • Hi Elavarasu,

     

    Even with the new Proactive Resource Provisioning feature described in the Microsoft article:

    - Fabric automatically pre-warms Spark resources, but…
    - You cannot control it, force it, or create your own custom pool
    - Private Link scenarios still add latency, because resources must be validated inside your private network.

    So:
    You cannot start Spark fully instantly in Private Link, but you can leverage the new Fabric optimizations to reduce cold-start times — automatically.

     

    The blog describes Fabric’s internal capability called:

    - Proactive Resource Provisioning

    Fabric now analyzes historical usage of Spark in each workspace and proactively allocates compute before you run your next notebook.

    This means:

    • If you frequently run Spark at 8AM — Fabric will pre-provision compute around 7:55.

    • If you run Spark ad-hoc all day — Fabric may keep resources warmer for longer.

    • If you stop using Spark — Fabric will stop pre-provisioning to save cost.

    This makes Spark notebooks “feel more instant” — but NOT instant for all users and not guaranteed.

    Important: This system is fully automated.

     

    You cannot:

    • manually create a pre-warmed pool

    • reserve Spark compute

    • configure warm-up windows

    • request an always-on cluster

    Behavior in Private Link Tenants

    The new proactive provisioning still works, but:
    Private Link tenants add:

    1. Network isolation validation

    2. DNS routing inside the VNet

    3. Container traffic restrictions

    These extra steps cannot be pre-warmed.

     

    "Customizing compute configurations will result in on-demand session start experience which could take from 2 - 5 minutes"

     

    References : 

    - https://learn.microsoft.com/en-us/fabric/security/security-private-links-overview

    - https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute

     

    Hope it can help you!

    Best regards,

    Antoine

3 Replies

  • Hi Elavarasu,

     

    Even with the new Proactive Resource Provisioning feature described in the Microsoft article:

    - Fabric automatically pre-warms Spark resources, but…
    - You cannot control it, force it, or create your own custom pool
    - Private Link scenarios still add latency, because resources must be validated inside your private network.

    So:
    You cannot start Spark fully instantly in Private Link, but you can leverage the new Fabric optimizations to reduce cold-start times — automatically.

     

    The blog describes Fabric’s internal capability called:

    - Proactive Resource Provisioning

    Fabric now analyzes historical usage of Spark in each workspace and proactively allocates compute before you run your next notebook.

    This means:

    • If you frequently run Spark at 8AM — Fabric will pre-provision compute around 7:55.

    • If you run Spark ad-hoc all day — Fabric may keep resources warmer for longer.

    • If you stop using Spark — Fabric will stop pre-provisioning to save cost.

    This makes Spark notebooks “feel more instant” — but NOT instant for all users and not guaranteed.

    Important: This system is fully automated.

     

    You cannot:

    • manually create a pre-warmed pool

    • reserve Spark compute

    • configure warm-up windows

    • request an always-on cluster

    Behavior in Private Link Tenants

    The new proactive provisioning still works, but:
    Private Link tenants add:

    1. Network isolation validation

    2. DNS routing inside the VNet

    3. Container traffic restrictions

    These extra steps cannot be pre-warmed.

     

    "Customizing compute configurations will result in on-demand session start experience which could take from 2 - 5 minutes"

     

    References : 

    - https://learn.microsoft.com/en-us/fabric/security/security-private-links-overview

    - https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute

     

    Hope it can help you!

    Best regards,

    Antoine

    • v-dineshya's avatar
      v-dineshya
      Icon for Community Support rankCommunity Support

      Hi Elavarasu ,

      Thank you for reaching out to the Microsoft Community Forum.

       

      Hi AntoineW , Thank you for your prompt response.

       

      Hi Elavarasu  could you please try the proposed solution shared by  AntoineW   ? Let us know if you’re still facing the same issue we’ll be happy to assist you further.

       

      Regards,

      Dinesh

  • Thank you for your response AntoineW , seems for now we cannot have custom pre-warmed pools. So there will always adds up start time when private link enabled.