Invoke Pipeline Performance

Question

I have a scenario where I've developed several pipelines that perform common operations. These pipelines are called by other pipelines that need to wait for the called pipeline to finish executing (wait on completion is on). In this case, the called pipeline takes a few seconds, while the caller waits at least a minute and 20 seconds. What's causing this discrepancy? The point is that if a pipeline performs several invoke pipeline operations, dead time accumulates.

tayloramy · Accepted Answer

Hi&nbsp;pellitteris,&nbsp;&nbsp;Are you using the legacy pipeline activity or the new one?&nbsp;The legacy one performs better but only works for fabric pipelines in the same workspace.&nbsp;&nbsp;The new activity uses the Job Scheduler APIs, which adds some overhead. A microsoft employee confirmed that they are working on improving this on this reddit thread:&nbsp;Why is the new Invoke Pipeline activity GA when it’s 12× slower than the legacy version? : r/MicrosoftFabric&nbsp;&nbsp;If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.&nbsp;&nbsp;&nbsp;

Forum Discussion

Invoke Pipeline Performance

1 Reply

Related content

Invoke Pipeline vs Invoke Pipeline (Legacy)

Invoke Pipeline

Invoke pipeline error

Data Pipeline : Invoke Pipeline Issue

Fabric Pipeline Performance