Invoke Pipeline Performance

[pellitteris]

I have a scenario where I've developed several pipelines that perform common operations. These pipelines are called by other pipelines that need to wait for the called pipeline to finish executing (wait on completion is on). In this case, the called pipeline takes a few seconds, while the caller waits at least a minute and 20 seconds. What's causing this discrepancy? The point is that if a pipeline performs several invoke pipeline operations, dead time accumulates.

[tayloramy · Accepted solution]

Hi @pellitteris, 

Are you using the legacy pipeline activity or the new one? 
The legacy one performs better but only works for fabric pipelines in the same workspace. 

The new activity uses the Job Scheduler APIs, which adds some overhead. A microsoft employee confirmed that they are working on improving this on this reddit thread: Why is the new Invoke Pipeline activity GA when it’s 12× slower than the legacy version? : r/Microso...

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.