Forum Discussion
Solved
Microsoft Fabric – Event trigger fires multiple times when multiple files arrive
Hi Team, I am using Microsoft Fabric event-based triggers with pipelines. When multiple files arrive at the same time, Fabric creates one event per file, so the pipeline is triggered multiple times...
Hello PAVAN_111
Native “group/aggregate events into one trigger” isn’t available today. Fabric’s event-based triggers (via Real‑Time hub/Activator) create a separate event—and therefore a separate pipeline run—for each file event that matches your filter.Best‑practice is to design for batching inside your pipeline (or via a small orchestrator pattern) so only one run does the work and any concurrently triggered runs quickly no‑op.You can use the following pattern:Allow event triggers to fire, but ensure only one pipeline run actually processes the batch. Others detect the lock and exit fast.How to implement:
- In Pipeline settings, set Concurrency to 1 so runs queue; only one run is active. (This setting is available in ADF/Synapse and behaves similarly in Fabric.)
- At the start of the pipeline:
- Try to acquire a lease/lock, e.g., write a lock file in a control folder or set a flag row in a control table (Lakehouse or SQL). If a lock exists → skip (return success/no‑op).
- Once locked:
- Wait briefly (e.g., 30–120 seconds) to allow the full file burst to land.
- List files in the inbound folder, build a batch, and process them together.
- Move/archive processed files (so subsequent queued runs see “nothing new” and exit).
- Release the lock.
Here's the control flow you can use in your Fabric pipeline:
Get/Set Lock
- If Condition: lock_exists() → true → Set variable “SkippedDueToLock” = true → End
- Else → Create lock (e.g., write /control/ingest.lock or INSERT INTO control_batch(status='running'))
Wait (optional)
- Wait activity: 60–120 seconds (tune to your source uploader behavior)
Enumerate batch
- Get Metadata/List → enumerate /raw/inbound/YYYY/MM/DD
- Filter to only new files (since last watermark or not in archive)
Process
- ForEach files → copy/transform (or load to staging and Spark job to batch merge)
Archive + Watermark
- Move files to /raw/archive/...
- Update control table last_processed_timestamp / last_processed_file
Release lock
- Delete /control/ingest.lock or set control row status='completed'
If this helps please mark this as a solution or a thumbs up!
AntoineW
Super User
Super UserNative grouping event based triggers is not supported today.
The event-based trigger with pipelines is based on a specific event.
Maybe you should use schedule one, to ingest data batchly.