Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
MKV
Regular Visitor

Run the Pipeline in Every Hour

Hi All,

 

Hope all is well.

 

1) An ADF pipeline runs in every hour from Mon to Fri between 7:00 AM to 9:00 PM using scheduled TRIGGER1

2) Same ADF pipeline runs every Saturday between 7:00 AM to 4:00 PM using scheduled TRIGGER2

3) No runs on Sunday.

 

Now issue is that Trigger is kicking off the pipeline even if previous run has not completed and as concurrency is set to 1, this new run is always in the QUEUE. So suppose current pipeline is running for 4 hours and trigger is kicking off the new execution in every hour, 4 new execution are in the QUEUE.

 

did someone face the same scenario? Can you please suggest how to solve this problem, I want not to run the next pipeline or next pipeline should be completed successfully as soon as it finds prev run is still running.

 

Regards,

MKV

 

1 ACCEPTED SOLUTION
NandanHegde
Super User
Super User

The main purpose of the concurrency functionality is to move the next iterations to queue. In case if you want the consecutive runs to close if prev runs are in progress, then remove the concurrency property. Let the next iteration trigger and in the 1st step of pipeline add an activity to check whether there is already an existing run of that pipeline ( either from ADF rest api Or your internal logging framework table wherein you can have a table to mark whether the job is done or in progress status flag) 

In case of the prev run is in progress, then cancel the current run or skip the current flow and exit

View solution in original post

1 REPLY 1
NandanHegde
Super User
Super User

The main purpose of the concurrency functionality is to move the next iterations to queue. In case if you want the consecutive runs to close if prev runs are in progress, then remove the concurrency property. Let the next iteration trigger and in the 1st step of pipeline add an activity to check whether there is already an existing run of that pipeline ( either from ADF rest api Or your internal logging framework table wherein you can have a table to mark whether the job is done or in progress status flag) 

In case of the prev run is in progress, then cancel the current run or skip the current flow and exit

Helpful resources

Announcements
Europe Fabric Conference

Europe’s largest Microsoft Fabric Community Conference

Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.

RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

MayFBCUpdateCarousel

Fabric Monthly Update - May 2024

Check out the May 2024 Fabric update to learn about new features.