Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Don't miss out! 2025 Microsoft Fabric Community Conference, March 31 - April 2, Las Vegas, Nevada. Use code MSCUST for a $150 discount. Prices go up February 11th. Register now.

Reply
MKV
Frequent Visitor

Run the Pipeline in Every Hour

Hi All,

 

Hope all is well.

 

1) An ADF pipeline runs in every hour from Mon to Fri between 7:00 AM to 9:00 PM using scheduled TRIGGER1

2) Same ADF pipeline runs every Saturday between 7:00 AM to 4:00 PM using scheduled TRIGGER2

3) No runs on Sunday.

 

Now issue is that Trigger is kicking off the pipeline even if previous run has not completed and as concurrency is set to 1, this new run is always in the QUEUE. So suppose current pipeline is running for 4 hours and trigger is kicking off the new execution in every hour, 4 new execution are in the QUEUE.

 

did someone face the same scenario? Can you please suggest how to solve this problem, I want not to run the next pipeline or next pipeline should be completed successfully as soon as it finds prev run is still running.

 

Regards,

MKV

 

1 ACCEPTED SOLUTION
NandanHegde
Super User
Super User

The main purpose of the concurrency functionality is to move the next iterations to queue. In case if you want the consecutive runs to close if prev runs are in progress, then remove the concurrency property. Let the next iteration trigger and in the 1st step of pipeline add an activity to check whether there is already an existing run of that pipeline ( either from ADF rest api Or your internal logging framework table wherein you can have a table to mark whether the job is done or in progress status flag) 

In case of the prev run is in progress, then cancel the current run or skip the current flow and exit




----------------------------------------------------------------------------------------------
Nandan Hegde (MSFT Data MVP)
LinkedIn Profile : www.linkedin.com/in/nandan-hegde-4a195a66
GitHUB Profile : https://github.com/NandanHegde15
Twitter Profile : @nandan_hegde15
MSFT MVP Profile : https://mvp.microsoft.com/en-US/MVP/profile/8977819f-95fb-ed11-8f6d-000d3a560942
Topmate : https://topmate.io/nandan_hegde
Blog :https://datasharkx.wordpress.com

View solution in original post

1 REPLY 1
NandanHegde
Super User
Super User

The main purpose of the concurrency functionality is to move the next iterations to queue. In case if you want the consecutive runs to close if prev runs are in progress, then remove the concurrency property. Let the next iteration trigger and in the 1st step of pipeline add an activity to check whether there is already an existing run of that pipeline ( either from ADF rest api Or your internal logging framework table wherein you can have a table to mark whether the job is done or in progress status flag) 

In case of the prev run is in progress, then cancel the current run or skip the current flow and exit




----------------------------------------------------------------------------------------------
Nandan Hegde (MSFT Data MVP)
LinkedIn Profile : www.linkedin.com/in/nandan-hegde-4a195a66
GitHUB Profile : https://github.com/NandanHegde15
Twitter Profile : @nandan_hegde15
MSFT MVP Profile : https://mvp.microsoft.com/en-US/MVP/profile/8977819f-95fb-ed11-8f6d-000d3a560942
Topmate : https://topmate.io/nandan_hegde
Blog :https://datasharkx.wordpress.com

Helpful resources

Announcements
Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount! Prices go up Feb. 11th.

JanFabricDE_carousel

Fabric Monthly Update - January 2025

Explore the power of Python Notebooks in Fabric!

JanFabricDW_carousel

Fabric Monthly Update - January 2025

Unlock the latest Fabric Data Warehouse upgrades!

JanFabricDF_carousel

Fabric Monthly Update - January 2025

Take your data replication to the next level with Fabric's latest updates!