topic Re: Schedule is launched even if the preceding excusion has not been completed in Pipelines

Schedule is launched even if the preceding excusion has not been completed

vivien57 — Fri, 02 Feb 2024 14:30:32 GMT

Hello everyone,

I have a parent pipeline that calls and executes different dataflow gen2. (5 exactly)

I have scheduled this pipeline every 20 minutes.

If the execution time of the pipeline exceeds 20 minutes, the schedule runs while the previous one is not finished.

If out of 5 dataflows, 3 had finished after 20 minutes, these 3 would be restarted via the schedule while the other two had not yet finished the previous execution.

Is there any way to change this?

Thanks in advance for your feedback

Have a nice day,

Vivien

Re: Schedule is launched even if the preceding excusion has not been completed

NandanHegde — Fri, 02 Feb 2024 15:59:32 GMT

Hey,

There is a concurrency setting at the pipeline level :

This ensures that in case if a pipeline is executed when it is already in progress, the new execution goes in queue till the time the initial execution is completed.

Hope this helps your use case!!!

Re: Schedule is launched even if the preceding excusion has not been completed

vivien57 — Tue, 06 Feb 2024 06:15:15 GMT

Thank you very much for your help

That's exactly it. I thought the default was 1. But by default there is no value, what does the absence of value mean?

Have a nice day

Re: Schedule is launched even if the preceding excusion has not been completed

NandanHegde — Tue, 06 Feb 2024 06:25:20 GMT

Absence of value is the default case wherein there can be 'N' nbr of parallel executions