Minimising Pipeline Overhead

Hi iShaymus ,

The overhead you're seeing is expected behavior with nested pipelines. Each Execute Pipeline activity introduces scheduling latency typically 1–3 minutes per level due to activity state polling, metadata persistence, and checkpoint writes between parent and child runs. This cost is partially fixed: it doesn't scale with your data volume, but it does compound with nesting depth.

In your case, with 3 levels of nesting (orange - red - blue/green), you're accumulating that latency at each hop. That's likely where most of your 10-minute gap is coming from.

A few approaches that can help reduce it :

• Reduce nesting depth
  Flatten your orchestration where possible. Instead of parent - child - grandchild, bring Copy Jobs and dbt execution into a single pipeline level using ForEach (parallel = true). Fewer hops = less overhead.
• Group related activities into parameterized child pipelines
  Instead of many nearly identical pipelines, use a small number of parameterized ones called from one orchestrator. Reduces both nesting and maintenance overhead.
• Use metadata-driven orchestration
  A config table can replace multiple similar pipelines. We've used this pattern to onboard dozens of sources while keeping the orchestration layer flat and fast.
• Monitor before optimizing
  Check the Monitoring Hub first sometimes the real bottleneck is Spark startup time, source latency, or SQL workload rather than pipeline scheduling itself.

Useful reference : https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-runs 

Out of curiosity are you mainly trying to reduce execution time, capacity consumption, or operational complexity? The answer could lead to different recommendations.