topic Re: Microsoft Fabric Prevent multiple pipeline triggers when multiple files arrive simultaneously. in Data Engineering

Microsoft Fabric Prevent multiple pipeline triggers when multiple files arrive simultaneously.

imtiazali — Fri, 26 Dec 2025 05:06:34 GMT

Hi Team,

I am working with Microsoft Fabric event-based triggers in a data pipeline scenario.

Currently, when multiple files arrive at the same time in the source location, Fabric creates one event per file, which causes the pipeline to be triggered multiple times. Each pipeline run processes a single file independently.

However, my business requirement is slightly different:

I want the pipeline to trigger only once

Then process all newly arrived files together in a single execution

I would like to understand:

Is there any native functionality in Microsoft Fabric to:

Group multiple file-created events into a single trigger, or

Prevent multiple pipeline runs when several files arrive simultaneously?

If native support is not available, what is the recommended or best-practice approach in Fabric to handle this use case?

For example: batching logic, scheduling pipelines, file existence checks, or any design pattern commonly used by the community.

Any guidance, architectural suggestions, or official recommendations would be highly appreciated.

Thank you in advance for your support.

Re: Microsoft Fabric Prevent multiple pipeline triggers when multiple files arrive simultaneously.

nielsvdc — Fri, 26 Dec 2025 09:15:02 GMT

Hi @imtiazali, currently in Fabric event-based triggers are per event. A run is created for each file event that matches your filter. There isn’t a native "group multiple file-created events into one" trigger.

Here are some options you can consider.

Schedule trigger every x minutes
- Maybe not what you are looking for, but you can schedule the pipeline to run every 1-5 minutes between a given time frame and process all the available files. If the processing of files takes longer than the interval, move the files to a processing folder first to ensure idempotency.
- The trade-off is that you will still see runs for all files, but that do nothing, because there are now files left to process.
Concurrency = 1+"first-file" guard
- Leave the event trigger enabled, set pipeline concurrency to 1 so only one run executes at a time. All other trigger events are queued and processed after the first run.
- In the pipeline implement a guard/lease with a check a lock file or control table.
  - If no active batch, acquire the lock, list all new files, process them together, release the lock.
  - If another run is already active (or just completed the batch), skip and succeed.
- Multiple triggers can fire, but only one run actually performs work; others become quick no‑ops.
- Optionally, add a Wait activity to the pipeline for waiting until all files have been written to location that you are monitoring.
- The trade-off is that you will still see runs for all files, but that do nothing, because there are now files left to process.
Use Data Activitor for files on OneLake/Blob?
- I haven't played around with this option yet and I currently don't have time to try it, but maybe it could work.
- Configure OneLake/Blob events in Data Activator to start a pipeline when an event happens. For Blob storage you might need to configure Azure Event Hub.
- Configure the Wait time for late-arriving events option to wait for all the data to be available. Play around with finding the correct condition.

Hope this helps. If so, please give kudos ‌‌👍 and mark as Accepted Solution ‌‌✔️ to help others. If you resolved your question, let us know what worked for you.

Re: Microsoft Fabric Prevent multiple pipeline triggers when multiple files arrive simultaneously.

deborshi_nag — Sat, 27 Dec 2025 15:45:42 GMT

Hi @imtiazali

Native “group/aggregate events into one trigger” isn’t available today. Fabric’s event-based triggers (via Real‑Time hub/Activator) create a separate event—and therefore a separate pipeline run—for each file event that matches your filter.

Best‑practice is to design for batching inside your pipeline (or via a small orchestrator pattern) so only one run does the work and any concurrently triggered runs quickly no‑op.

You can use the following pattern:

Allow event triggers to fire, but ensure only one pipeline run actually processes the batch. Others detect the lock and exit fast.

How to implement:

In Pipeline settings, set Concurrency to 1 so runs queue; only one run is active. (This setting is available in ADF/Synapse and behaves similarly in Fabric.)
At the start of the pipeline:
- Try to acquire a lease/lock, e.g., write a lock file in a control folder or set a flag row in a control table (Lakehouse or SQL). If a lock exists → skip (return success/no‑op).
Once locked:
- Wait briefly (e.g., 30–120 seconds) to allow the full file burst to land.
- List files in the inbound folder, build a batch, and process them together.
- Move/archive processed files (so subsequent queued runs see “nothing new” and exit).
Release the lock.

Here's the control flow you can use in your Fabric pipeline:

Get/Set Lock
- If Condition: lock_exists() → true → Set variable “SkippedDueToLock” = true → End
- Else → Create lock (e.g., write /control/ingest.lock or INSERT INTO control_batch(status='running'))
Wait (optional)
- Wait activity: 60–120 seconds (tune to your source uploader behavior)
Enumerate batch
- Get Metadata/List → enumerate /raw/inbound/YYYY/MM/DD
- Filter to only new files (since last watermark or not in archive)
Process
- ForEach files → copy/transform (or load to staging and Spark job to batch merge)
Archive + Watermark
- Move files to /raw/archive/...
- Update control table last_processed_timestamp / last_processed_file
Release lock
- Delete /control/ingest.lock or set control row status='completed'

If this helps please mark this as a solution.

Re: Microsoft Fabric Prevent multiple pipeline triggers when multiple files arrive simultaneously.

v-hashadapu — Mon, 29 Dec 2025 09:18:28 GMT

Hi @imtiazali , Thank you for reaching out to the Microsoft Community Forum.

We find the answers shared by @deborshi_nag & @nielsvdc are appropriate. Can you please confirm if they worked for you. It will help others with similar issues find the answer easily.

Thank you @deborshi_nag & @nielsvdc for your valuable responses.

Re: Microsoft Fabric Prevent multiple pipeline triggers when multiple files arrive simultaneously.

v-hashadapu — Fri, 02 Jan 2026 11:38:24 GMT

Hi @imtiazali , Hope you are doing well. Kindly let us know if the issue has been resolved or if further assistance is needed. Your input could be helpful to others in the community.