Run a pipeline at time intervals within a specific time range

[rampie]

Hi everyone,

Pipelines can be scheduled to run at minute or hourly frequencies, but I’d like to know how to configure a pipeline to run every 15 minutes only during specific hours of the day.

For example, I need the process to run repeatedly between 8:00 AM and 11:00 AM, but not outside that range. Currently, when I set the recurrence to 15 minutes, it runs perpetually throughout the day, consuming unnecessary capacity for a non-continuous process.

Is there a recommended approach or workaround to achieve this behavior?

Thanks in advance for your help!

[sChalvet]

The best would be for Fabric Scheduler to support Time windows. This is a pretty common request when refreshing data to save on API calls.
For example: I would love to run a pipeline every 15 minutes between 8am and 5pm since that is when the data is consumed. No need to run up an API bill when no one is looking at the data.

[tayloramy]

Hi @rampie, 

This isn't possible with the default scheudling features. 

You can get this sort of behaviour with the API inside a ntoebook however, you can use the Scheduler API inside of a notebook to run your pipeline on a much more customizable schedule. 

See Job Scheduler - Run On Demand Item Job - REST API (Core) | Microsoft Learn

I do think this would be a very useful native feature however, so I recommend posting an idea for it: Fabric Ideas - Microsoft Fabric Community

If you post an idea, comment back on this thread with it and I will vote for it. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[rampie]

Hi @tayloramy , thanks for your response.

Using the APIs sounds like a solid approach; however, I’m not quite sure how to implement that specific one in my scenario, since it only performs a “Run On Demand” action.

How could I make it run the API every few minutes but only for a few hours?

Also, is there any API that allows enabling or disabling a pipeline (the entire pipeline or a specific activity) or its schedule?

If so, I could use the native scheduling (with 15min frequency) and simply activate the pipeline when it needs to start running, and disable it once it’s outside the desired time range.

[tayloramy · Accepted solution]

Hi @rampie, 

You would need to have a notebook issue a new RUn on demand API request every few minuites during the hour. 

You can have a notebook scheduled to start at the beginning of your run window, then have the notebook run the API request, and wait for 15 minutes, and loop that until the the time is outside of your run window. 

Hope this makes sense. 

There is this API endpoint to update an existing schedule: Job Scheduler - Update Item Schedule - REST API (Core) | Microsoft Learn

You should be able to endable/disable the schedule with this, but I have not tested it myself. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.