Pipeline Duration Mismatch – Parent Pipeline Shows 3mins but Child Activities Total Only 1Min

[Deera]

I have two pipelines.

The first pipeline contains two activities: Lookup and ForEach. Inside the ForEach activity, I’m invoking a second pipeline.

The second (invoked) pipeline contains 6 activities, which connect to source SQL Servers via a gateway and load data into a Data Warehouse in Fabric.

Here’s the issue:
From the parent (driver) pipeline view, it shows that Pipeline 2 takes about 3 minutes to complete. However, when I open Pipeline 2 and sum the duration of the individual activities, the total comes to only 1 minute.

Where is the remaining 2 minutes being spent?

[Deera · Accepted solution]

Thanks everyone for your insights. I have changed to invoke legacy activity instead of invoke preview activity which reduces my timing to 1.5 minutes. Based on your comments i understand that the extra time taken for gateway connectivity. Thanks 

[v-aatheeque]

Hi @Deera 

Glad to hear that the performance has improved after switching to the legacy activity.
Based on our findings, the remaining time is expected due to gateway connectivity overhead.
Please let me know if you face any further issues. If this solution helped, kindly acknowledge it, as it may benefit others in the community facing similar issues.

[v-aatheeque]

Hi @Deera 

Have you had a chance to look through the responses shared earlier? If anything is still unclear, we’ll be happy to provide additional support.

[v-aatheeque]

Hi @Deera

We wanted to follow up to check if you’ve had an opportunity to review the previous responses. If you require further assistance, please don’t hesitate to let us know.

[R1k91]

use the monitor , as others say you could see Queue or something else before the actual transfer

if you're importing from supported sources you could also see "time at first byte" (SQLServer should be) that basically tells you how much time the copy waited for the first byte and it should be equal to the queue.

[rizalard0684]

@Deera in short the remaining 2 minutes are spent on getting things ready and cleaning up.
Let me elaborate what these are:
1. Pipeline startup overhead
When Pipeline 1 calls Pipeline 2, Fabric doesn’t instantly start the first activity. Before any activity runs, Fabric has to spin up the pipeline runtime, resolve parameters and build the execution graph. This time counts as pipeline duration, but does not belong to any activity, so you won’t see it inside Pipeline 2.

2. Gateway + data movement “warm‑up”

Because you’re using on‑prem SQL Servers and connecting through a gateway, Fabric needs to wake up the data movement engine, establish gateway connections, validate connectivity and allocate compute for copy. 

All this happens before the first copy activity actually starts.
3. Pipeline invocation overhead inside ForEach

Every time Execute Pipeline runs there is an orchestration time, context switching between pipelines and tracking state between parent and child. Might not be long but still count a bit.
In summary, Your activities really do take only about 1 minute but the extra ~2 minutes are orchestration + gateway + runtime overhead.

[lbendlin]

They are spent spinning up the Spark instance.

[Deera]

But i haven't use any spark or notebok activity in it. 

[pallavi_r]

Hi @Deera ,

The extra minutes, could be mostly coming from overhead outside the actual activity execution things like pipeline initialization, some queueing before it starts, and gateway-related delays when establishing connections to on-prem SQL.

Thanks,

Pallavi