Overhead time / performance: what to expect

Hi nullabee​,

Thanks for the detailed description. I cannot see your tenant, so I will not claim that these numbers are or are not normal. What I can say is that a few seconds of orchestration overhead per activity is common in Fabric, and 15 to 20 seconds around a 5 second child run is large enough to be worth investigating. Below is how I would break the problem down, from the cheapest checks to the deeper ones.

1. Find out where the time actually goes

Each activity run has start, queue and end information. Before changing anything, measure the gap between the parent starting the call and the child actually starting:

Monitor hub -> pipeline run -> activity runs: compare activity start vs. child run start vs. child end
Gap A (parent start -> child start)  = queueing / scheduling / network or credential setup
Gap B (child end -> parent resume)   = result collection and orchestration
Repeat the same run at different times of day to see whether it varies with load

2. Capacity pressure

If capacity is running hot, Fabric delays or throttles work instead of failing it, which looks exactly like random slowness and occasional very long waits. Check the Capacity Metrics app for utilization, throttling and overage around the slow runs:

Look at -> CU utilization vs. capacity size, smoothed usage, throttling / delay indicators
Compare -> timestamps of the slow pipeline calls against the utilization chart
If correlated -> reduce concurrency, stagger schedules, or size the capacity up for a test

3. Network path (VNets, vault, storage)

Your storage, SQL and key vault sit in other VNETs. Each hop that goes through a gateway, private endpoint or credential retrieval can add seconds per activity, and they add up in nested pipelines:

- Is each connection going through an on-premises or VNet data gateway? Gateways have their own startup and queue time
- Are vault lookups done per activity? Fetching the same secret in every child pipeline repeats that cost
- Prefer connections that reference the vault once, and pass values down as pipeline parameters
- Check whether managed private endpoints are used and healthy for the data sources involved

4. Pipeline design

Nested pipelines   -> each Invoke pipeline call has fixed overhead; flatten where a child only does one small step
Notebooks           -> Spark session start dominates short jobs; use a starter pool or a custom pool, reuse sessions
                       across notebook activities with a session tag / high concurrency mode where it fits
Get Metadata checks -> 10 seconds to check a file is slow; test replacing it with a Lookup or a small notebook cell
                       that reads the file list once, rather than one activity per file

5. The occasional hour-long hang

A call that normally takes 15 seconds and suddenly takes an hour should never be allowed to run unbounded. Set a timeout and a retry on every activity so a hang becomes a visible failure that alerts, and capture the run IDs for support:

"policy": {
  "timeout": "0.00:05:00",
  "retry": 2,
  "retryIntervalInSeconds": 30
}

When one of those hangs happens again, note the run ID, activity ID, region and exact timestamps and open a support request, because that is the evidence needed to tell a platform issue from a capacity or network cause. The same applies to deployment pipelines that take minutes for a single notebook: record the deployment ID and the time, since progress is not visible in the UI.

6. Monitoring beyond the Monitor hub

Capacity Metrics app -> utilization, throttling, per-item consumption
Monitor hub          -> run history, durations, error details
Workspace monitoring -> detailed logs you can query, if enabled for your workspace
Own tracking         -> log start/end per pipeline into a control table to chart overhead over time

So my overall approach would be:

Measure   -> split each slow call into queue gap, run time and collection gap
Correlate -> compare slow runs with capacity utilization and throttling
Isolate   -> test a network-light copy of the pipeline (no vault, no gateway) to see what the baseline overhead is
Protect   -> timeouts and retries on every activity; support case with run IDs for the real hangs

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Hi nullabee

The notebook gap is most likely Spark session start, which the per-cell times don't include. A starter pool with no libraries or custom Spark properties normally starts in 5 to 10 seconds. Libraries from an environment add 30 seconds to 5 minutes. If the workspace uses private links or managed VNets, starter pools aren't supported and a session takes 2 to 5 minutes to start. Half a minute for 10 quick cells is the kind of gap that explains.

If one pipeline runs several notebooks, turn on high concurrency for pipelines under Workspace settings › Data Engineering/Science › Spark settings › High concurrency. Notebooks then share one session when they have the same user, default lakehouse, Spark settings and libraries, with up to 5 per session tag.

I can't see your setup, so I can't say anything on the pipeline to pipeline and SQL lookup gaps. Do your notebooks use an environment with custom libraries, or does the workspace have a managed VNet or private link?

I think workspace-wide concurrency is worth checking before treating these delays as normal. Fabric limits concurrent Lookup/Get Metadata/Delete activities to 100 per workspace, including those in child pipelines. A stored procedure called through Lookup counts as Lookup. Enable workspace monitoring, check overlapping executions in FabricDataPipelinesActivityRunsLogs, then rerun the same Lookup with other pipelines idle. 

If it's still slow, send the run IDs and UTC timestamps to Fabric support. The logs alone won't show the queue time.