Triggers of Fabric Pipelines-Concurrency

[AmruthaVarshini]

Hi Team,

We have a requirement as described below. 

There is a pipeline which will be triggered based on a file arrival in ADLS Container. When a file-1 is arrived and trigger starts the pipeline , there is possibility of file-2 arriving when first one is still in progress. The file-2 also should also trigger the same pipeline. Is there possibility of both of them being run parallely in fabric?Also please explain the option of concurrency and what are it's limitations.

Please provide your insights, it would be very helpful in deciding our solution.

[NandanHegde]

Under Pipeline settings,

there is a concurrency option.

By default, there is no limitation on cincurrent runs but you can define what can be the max concurrent runs.

In case if no concurrency is set up, based on Event trigger of 'N' number of files, there would be 'N' number of parallel executions.

[AmruthaVarshini]

Hi @NandanHegde ,

Thanks for the inputs, they are very helpful.

[Anonymous · Accepted solution]

Hi @AmruthaVarshini ,

Thanks for using Fabric Community.
As @NandanHegde  mentioned you can configue the concurrent settings in pipeline in order to make 'N' number of parallel executions.

Just want add one more point here, limitations to concurrent operations -

Docs to refer - Data Factory limitations overview - Microsoft Fabric | Microsoft Learn

Hope this is helpful. Please do let me know incase of further queries.

[AmruthaVarshini]

Hi @Anonymous ,

Thanks for the info, currently we are testing this feature on our pipelines, will get back to you if any further help is required.

[Anonymous]

Hi @AmruthaVarshini ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .

Thanks