topic Re: Check if a pipeline is currently running in Pipelines

Check if a pipeline is currently running

BW_RFA — Mon, 19 Feb 2024 09:55:04 GMT

Hi,

I have a scheduled pipeline that runs every 15 minutes.

Once a day it may ingest a lotof data and the pipeline runs for over 15 minutes, meaning the next run tries to process the same data etc.

Is there a way to automatically check whether the pipeline is currently running (from within the pipeline) so I don't run it multiple times if it currently ongoing? I would check to see if the pipeline is running and stop the current run if it is.

Thanks

Re: Check if a pipeline is currently running

Anonymous — Mon, 19 Feb 2024 09:34:16 GMT

Hi BW_RFA
Thanks for using Fabric Community.
You can go to the monitoring hub in a pipeline to check the status of pipeline runs.

For more information please refer to this link:
https://learn.microsoft.com/en-us/fabric/data-factory/monitoring-hub-pipeline-runs

Hope this helps. Please let me know if you have any further queries.

Re: Check if a pipeline is currently running

BW_RFA — Mon, 19 Feb 2024 09:54:16 GMT

Hi,

Thank you for your reply. I think you have misunderstood my question.

I want to do this automatically from within a pipeline so that if the same pipeline is still running it will stop.

Thanks

Re: Check if a pipeline is currently running

Anonymous — Mon, 19 Feb 2024 10:38:45 GMT

Hi BW_RFA
Apologies for the misunderstanding. Directly checking the pipeline's status within an ongoing pipeline in Microsoft Fabric isn't currently available. You can set the concurrency limit to 1 in the pipeline settings. The maximum number of concurrent runs the pipeline can have is called Concurrency. By default, there is no maximum. If the concurrency limit is reached, additional pipeline runs are queued until earlier ones complete.

Hope this helps. Please let me know if you have any further queries.

Re: Check if a pipeline is currently running

NandanHegde — Mon, 19 Feb 2024 10:45:04 GMT

Unfortunately As of today Fabric data pipelines do not have any REST API because of which we cannot check the status of the pipeline at run time dynamically.

But you can enable the concurrency property wherein making it 1 would ensure only 1 instnace of pipeline is running and have others in queue.

Re: Check if a pipeline is currently running

BW_RFA — Mon, 19 Feb 2024 13:05:49 GMT

Thanks,

Does this work with scheduled triggers of the same pipeline?

Re: Check if a pipeline is currently running

Anonymous — Mon, 19 Feb 2024 15:58:28 GMT

Hi BW_RFA
Yes the concurrency works for scheduled triggers also.