Run the Pipeline in Every Hour

[MKV]

Hi All,

Hope all is well.

1) An ADF pipeline runs in every hour from Mon to Fri between 7:00 AM to 9:00 PM using scheduled TRIGGER1

2) Same ADF pipeline runs every Saturday between 7:00 AM to 4:00 PM using scheduled TRIGGER2

3) No runs on Sunday.

Now issue is that Trigger is kicking off the pipeline even if previous run has not completed and as concurrency is set to 1, this new run is always in the QUEUE. So suppose current pipeline is running for 4 hours and trigger is kicking off the new execution in every hour, 4 new execution are in the QUEUE.

did someone face the same scenario? Can you please suggest how to solve this problem, I want not to run the next pipeline or next pipeline should be completed successfully as soon as it finds prev run is still running.

Regards,

MKV

[NandanHegde · Accepted solution]

The main purpose of the concurrency functionality is to move the next iterations to queue. In case if you want the consecutive runs to close if prev runs are in progress, then remove the concurrency property. Let the next iteration trigger and in the 1st step of pipeline add an activity to check whether there is already an existing run of that pipeline ( either from ADF rest api Or your internal logging framework table wherein you can have a table to mark whether the job is done or in progress status flag) 

In case of the prev run is in progress, then cancel the current run or skip the current flow and exit