Overhead time / performance: what to expect

Hi nullabee​,

Thanks for the detailed description. I cannot see your tenant, so I will not claim that these numbers are or are not normal. What I can say is that a few seconds of orchestration overhead per activity is common in Fabric, and 15 to 20 seconds around a 5 second child run is large enough to be worth investigating. Below is how I would break the problem down, from the cheapest checks to the deeper ones.

1. Find out where the time actually goes

Each activity run has start, queue and end information. Before changing anything, measure the gap between the parent starting the call and the child actually starting:

Monitor hub -> pipeline run -> activity runs: compare activity start vs. child run start vs. child end
Gap A (parent start -> child start)  = queueing / scheduling / network or credential setup
Gap B (child end -> parent resume)   = result collection and orchestration
Repeat the same run at different times of day to see whether it varies with load

2. Capacity pressure

If capacity is running hot, Fabric delays or throttles work instead of failing it, which looks exactly like random slowness and occasional very long waits. Check the Capacity Metrics app for utilization, throttling and overage around the slow runs:

Look at -> CU utilization vs. capacity size, smoothed usage, throttling / delay indicators
Compare -> timestamps of the slow pipeline calls against the utilization chart
If correlated -> reduce concurrency, stagger schedules, or size the capacity up for a test

3. Network path (VNets, vault, storage)

Your storage, SQL and key vault sit in other VNETs. Each hop that goes through a gateway, private endpoint or credential retrieval can add seconds per activity, and they add up in nested pipelines:

- Is each connection going through an on-premises or VNet data gateway? Gateways have their own startup and queue time
- Are vault lookups done per activity? Fetching the same secret in every child pipeline repeats that cost
- Prefer connections that reference the vault once, and pass values down as pipeline parameters
- Check whether managed private endpoints are used and healthy for the data sources involved

4. Pipeline design

Nested pipelines   -> each Invoke pipeline call has fixed overhead; flatten where a child only does one small step
Notebooks           -> Spark session start dominates short jobs; use a starter pool or a custom pool, reuse sessions
                       across notebook activities with a session tag / high concurrency mode where it fits
Get Metadata checks -> 10 seconds to check a file is slow; test replacing it with a Lookup or a small notebook cell
                       that reads the file list once, rather than one activity per file

5. The occasional hour-long hang

A call that normally takes 15 seconds and suddenly takes an hour should never be allowed to run unbounded. Set a timeout and a retry on every activity so a hang becomes a visible failure that alerts, and capture the run IDs for support:

"policy": {
  "timeout": "0.00:05:00",
  "retry": 2,
  "retryIntervalInSeconds": 30
}

When one of those hangs happens again, note the run ID, activity ID, region and exact timestamps and open a support request, because that is the evidence needed to tell a platform issue from a capacity or network cause. The same applies to deployment pipelines that take minutes for a single notebook: record the deployment ID and the time, since progress is not visible in the UI.

6. Monitoring beyond the Monitor hub

Capacity Metrics app -> utilization, throttling, per-item consumption
Monitor hub          -> run history, durations, error details
Workspace monitoring -> detailed logs you can query, if enabled for your workspace
Own tracking         -> log start/end per pipeline into a control table to chart overhead over time

So my overall approach would be:

Measure   -> split each slow call into queue gap, run time and collection gap
Correlate -> compare slow runs with capacity utilization and throttling
Isolate   -> test a network-light copy of the pipeline (no vault, no gateway) to see what the baseline overhead is
Protect   -> timeouts and retries on every activity; support case with run IDs for the real hangs

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.