Microsoft Fabric Notebook Alternatives to Private Endpoints and Custom Live Pools?

Hi venkatasuresh_g​,

The trade-off you're seeing is by design: once a workspace has managed private endpoints (managed VNet), Spark can't use the shared starter pools, so every session spins up a dedicated cluster and you pay the few-minute startup. As far as I know there's no supported way to get starter-pool startup times with MPEs today, and trusted workspace access covers ADLS Gen2 storage rather than Key Vault.

So rather than trying to make each session start faster, the more effective lever is to start fewer sessions. Pay the startup once per run and run all the notebooks inside that one session.

Inside a single driver notebook, notebookutils.notebook.runMultiple executes the others in the same session, with dependencies:

dag = {
    "activities": [
        {"name": "load_bronze",  "path": "nb_load_bronze",  "timeoutPerCellInSeconds": 1800},
        {"name": "load_silver",  "path": "nb_load_silver",  "dependencies": ["load_bronze"]},
        {"name": "build_gold",   "path": "nb_build_gold",   "dependencies": ["load_silver"]},
        {"name": "export_audit", "path": "nb_export_audit", "dependencies": ["load_bronze"]}
    ],
    "concurrency": 3
}
notebookutils.notebook.runMultiple(dag)

Each child notebook can still call notebookutils.credentials.getSecret against the private vault through the MPE, so nothing secret moves between notebooks. Avoid fetching once and passing secrets as parameters, since parameter values show up in run snapshots.

For pipelines, enable high concurrency for pipelines in the workspace Spark settings and give the notebook activities the same session tag. Notebooks in the same pipeline run then share a session instead of each paying the startup.

With this, a schedule of eight short notebooks goes from eight cluster startups to one, without the always-on cost of a custom live pool.

So my overall approach would be:

Key Vault private + MPE (keep) -> accept one cluster start per run
Many short notebooks -> one driver notebook with runMultiple (shared session)
Pipelines -> high concurrency for pipelines + same session tag on notebook activities
Secrets -> getSecret inside each notebook, never passed as parameters
Consolidate schedules -> fewer, larger runs instead of many tiny ones

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Hi venkatasuresh_g​,

Have you had a chance to review the solution shared by jubinsoni​, ShivekMaharaj​ ? If the issue persists, feel free to reply so we can help further.

Thank you.

HI venkatasuresh_g​,

Checking in to see if your issue has been resolved. let us know if you still need any assistance.

Thank you.

Hi venkatasuresh_g​,

The main trade-off you are seeing is currently by design.

Microsoft’s Fabric Private Link guidance states that once a managed virtual network is provisioned for a workspace, Starter Pools are disabled because Starter Pools are prewarmed clusters hosted in a shared virtual network.

In that configuration, Spark sessions use custom compute created inside the dedicated managed VNet, which is why Microsoft documents startup times in the multi-minute range rather than the normal seconds-level Starter Pool experience.

So I would not expect Managed Private Endpoints and Starter Pools to coexist in the same workspace today.

For the Key Vault side, keeping the vault private and using a Managed Private Endpoint is the right security pattern if public network access must remain disabled.

Microsoft’s NotebookUtils credentials guidance supports retrieving Azure Key Vault secrets with notebookutils.credentials.getSecret(), while the network path to the private vault still needs to exist.

The identity and network concerns are separate:

Managed Private Endpoint
→ provides the private network path to Key Vault

user / service principal / workspace identity
→ provides authentication and Key Vault authorization

Microsoft’s Fabric connection guidance for notebooks also supports workspace identity or service principal authentication for Azure Key Vault connections, provided that identity has the required Key Vault permissions.

One current option that may fit your startup-cost concern better than keeping compute continuously available is a Custom Live Pool.

Microsoft’s Custom Live Pools guidance describes these as prehydrated custom Spark clusters that can provide roughly 5-second notebook startup during a scheduled active window.

The important distinction is that they are schedule-based.

For example:

scheduled notebooks run from 06:00–08:00
→ configure the live-pool window around that period
→ Fabric hydrates compute ahead of the workload
→ notebooks get the fast startup benefit during that window
→ outside the window, Fabric falls back to standard Spark provisioning

So this does not necessarily mean paying for warm compute 24/7.

Microsoft also recommends starting the live-pool schedule 60-90 minutes before a latency-sensitive scheduled workload so that at least one compatible cluster is already hydrated.

If you have many short notebooks, I would also consolidate them where practical.

For example:

one parent notebook/session
→ run multiple child notebooks using notebookutils.notebook.runMultiple()

rather than:

eight short notebooks
→ eight separate Spark session startups

Microsoft’s NotebookUtils orchestration guidance supports chaining and running notebooks, including parallel DAG execution.

So the architecture I would consider is:

Private Azure Key Vault
→ Managed Private Endpoint from the Fabric workspace

Authentication
→ workspace identity or dedicated service identity with least-privilege Key Vault access

Compute
→ Custom Live Pool only during known production windows

Notebook design
→ consolidate short jobs into fewer Spark sessions where possible

Secrets
→ each notebook retrieves its own secret from Key Vault rather than passing the secret value between notebooks

I would not replace Key Vault with ADLS simply to avoid the startup penalty, because they solve different security problems and Microsoft explicitly recommends Azure Key Vault for secret storage.

So the main options today appear to be:

Starter Pool + very fast startup
→ shared networking, therefore not suitable for your private-only Key Vault requirement

Managed VNet + Managed Private Endpoint
→ private connectivity, but normal Spark cold-start overhead

Managed VNet + Custom Live Pool
→ private connectivity plus much faster startup during scheduled windows, at additional capacity cost only for the periods you choose to keep the pool hydrated

I am not aware of a currently documented trusted-service or workspace-identity exception that allows Fabric Spark to reach a public-network-disabled Key Vault while keeping the shared Starter Pool path.

So if private-only Key Vault is a hard requirement, I would optimize around fewer sessions or scheduled Custom Live Pools rather than dynamic outbound IP allowlisting.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.