Fabric Pipeline Excutting Waiting Time

[ericjo]

Hello,

I have a question regarding pipeline execution behavior in Fabric.

I am currently building a pipeline using the "Invoke Pipeline" activity. I noticed that even after the child pipeline shows as "Completed," the parent pipeline does not finish immediately and seems to remain in a waiting state for some additional time before completing.

I would like to ask:

• Is this expected behavior in Fabric pipelines?
• If so, is there any way to reduce or optimize this waiting time?

Thank you in advance for your help.

[v-veshwara-msft · Accepted solution]

Hi @ericjo ,

Thanks for reaching out to the Microsoft Fabric Community.

Yes, the behavior you are seeing can be expected when using the newer Invoke Pipeline activity.

The newer activity provides additional orchestration and monitoring capabilities, including child pipeline tracking and integration with Fabric monitoring experiences. As part of this process, the pipeline execution goes through additional orchestration layers before the final completion status is returned to the parent pipeline. Because of this, it is possible to observe a delay between the child pipeline showing as completed and the parent pipeline activity finishing.

Reference: Invoke pipeline activity - Microsoft Fabric | Microsoft Learn

If minimizing execution overhead is important for your scenario, you may consider testing with Invoke Pipeline (Legacy). The legacy activity uses a more direct pipeline invocation mechanism and, in many scenarios, can complete with lower orchestration overhead.

Improvements to the newer Invoke Pipeline experience are being made over time, so the observed latency may improve in future updates.

Similar discussion: Solved: Invoke Pipeline Performance - Microsoft Fabric Community

Hope this helps. Please let us know if you have any further questions.

Thank you.

[v-veshwara-msft]

Hi @ericjo ,

We wanted to kindly follow up regarding your query. If you need any further assistance, please reach out.
Thank you.

[v-veshwara-msft]

Hi @ericjo ,

Just wanted to check if the responses provided were helpful in resolving your query. If further assistance is needed, please reach out.
Thank you.

[tayloramy · Accepted solution]

Hi @ericjo, 

Yes, this is expected. There is a couple of minutes of delay between the child pipeline finishing, and all the metadata updating in the Scheduler API to indicate that the job is complete. That API is what the pipeline connector is using, so that delay affects it. 

[v-veshwara-msft · Accepted solution]

Hi @ericjo ,

Thanks for reaching out to the Microsoft Fabric Community.

Yes, the behavior you are seeing can be expected when using the newer Invoke Pipeline activity.

The newer activity provides additional orchestration and monitoring capabilities, including child pipeline tracking and integration with Fabric monitoring experiences. As part of this process, the pipeline execution goes through additional orchestration layers before the final completion status is returned to the parent pipeline. Because of this, it is possible to observe a delay between the child pipeline showing as completed and the parent pipeline activity finishing.

Reference: Invoke pipeline activity - Microsoft Fabric | Microsoft Learn

If minimizing execution overhead is important for your scenario, you may consider testing with Invoke Pipeline (Legacy). The legacy activity uses a more direct pipeline invocation mechanism and, in many scenarios, can complete with lower orchestration overhead.

Improvements to the newer Invoke Pipeline experience are being made over time, so the observed latency may improve in future updates.

Similar discussion: Solved: Invoke Pipeline Performance - Microsoft Fabric Community

Hope this helps. Please let us know if you have any further questions.

Thank you.

[suparnababu8]

Hello @ericjo 

yes, It might be expected behavior in fabric pipelines. As mentioned by you, When you're using invoke pipeline activuity, the main pipeline actuivity waits till Child pipeline finshes the run status, fully confirmed and logged it. 

While coming to answer your questions, as of now there is no direct way to eliminate/reduce this waiting time. may be u can enable 'wait on completion'  

If this is unchecked, the parent pipleine  continues immediately, runing subsequent activities in parallel. Just give try and pls let me know. Hope this helps you.

Thank you!!

Did I answer your question? Mark my post as a solution!

Proud to be a Super User!