Forum Discussion
Microsoft Fabric – Event trigger fires multiple times when multiple files arrive
Hi Team,
I am using Microsoft Fabric event-based triggers with pipelines.
When multiple files arrive at the same time, Fabric creates one event per file, so the pipeline is triggered multiple times.
My requirement is to trigger the pipeline only once and then process all the files together.
Is there any native way in Microsoft Fabric to:
group multiple file-created events into a single trigger, or
prevent multiple pipeline runs when multiple files arrive?
If this is not supported, what is the recommended approach in Fabric to handle this scenario?
Thanks in advance.
Hello PAVAN_111
Native “group/aggregate events into one trigger” isn’t available today. Fabric’s event-based triggers (via Real‑Time hub/Activator) create a separate event—and therefore a separate pipeline run—for each file event that matches your filter.Best‑practice is to design for batching inside your pipeline (or via a small orchestrator pattern) so only one run does the work and any concurrently triggered runs quickly no‑op.You can use the following pattern:Allow event triggers to fire, but ensure only one pipeline run actually processes the batch. Others detect the lock and exit fast.How to implement:
- In Pipeline settings, set Concurrency to 1 so runs queue; only one run is active. (This setting is available in ADF/Synapse and behaves similarly in Fabric.)
- At the start of the pipeline:
- Try to acquire a lease/lock, e.g., write a lock file in a control folder or set a flag row in a control table (Lakehouse or SQL). If a lock exists → skip (return success/no‑op).
- Once locked:
- Wait briefly (e.g., 30–120 seconds) to allow the full file burst to land.
- List files in the inbound folder, build a batch, and process them together.
- Move/archive processed files (so subsequent queued runs see “nothing new” and exit).
- Release the lock.
Here's the control flow you can use in your Fabric pipeline:
Get/Set Lock
- If Condition: lock_exists() → true → Set variable “SkippedDueToLock” = true → End
- Else → Create lock (e.g., write /control/ingest.lock or INSERT INTO control_batch(status='running'))
Wait (optional)
- Wait activity: 60–120 seconds (tune to your source uploader behavior)
Enumerate batch
- Get Metadata/List → enumerate /raw/inbound/YYYY/MM/DD
- Filter to only new files (since last watermark or not in archive)
Process
- ForEach files → copy/transform (or load to staging and Spark job to batch merge)
Archive + Watermark
- Move files to /raw/archive/...
- Update control table last_processed_timestamp / last_processed_file
Release lock
- Delete /control/ingest.lock or set control row status='completed'
If this helps please mark this as a solution or a thumbs up!
6 Replies
Native grouping event based triggers is not supported today.
The event-based trigger with pipelines is based on a specific event.
Maybe you should use schedule one, to ingest data batchly.
Hello PAVAN_111
Native “group/aggregate events into one trigger” isn’t available today. Fabric’s event-based triggers (via Real‑Time hub/Activator) create a separate event—and therefore a separate pipeline run—for each file event that matches your filter.Best‑practice is to design for batching inside your pipeline (or via a small orchestrator pattern) so only one run does the work and any concurrently triggered runs quickly no‑op.You can use the following pattern:Allow event triggers to fire, but ensure only one pipeline run actually processes the batch. Others detect the lock and exit fast.How to implement:
- In Pipeline settings, set Concurrency to 1 so runs queue; only one run is active. (This setting is available in ADF/Synapse and behaves similarly in Fabric.)
- At the start of the pipeline:
- Try to acquire a lease/lock, e.g., write a lock file in a control folder or set a flag row in a control table (Lakehouse or SQL). If a lock exists → skip (return success/no‑op).
- Once locked:
- Wait briefly (e.g., 30–120 seconds) to allow the full file burst to land.
- List files in the inbound folder, build a batch, and process them together.
- Move/archive processed files (so subsequent queued runs see “nothing new” and exit).
- Release the lock.
Here's the control flow you can use in your Fabric pipeline:
Get/Set Lock
- If Condition: lock_exists() → true → Set variable “SkippedDueToLock” = true → End
- Else → Create lock (e.g., write /control/ingest.lock or INSERT INTO control_batch(status='running'))
Wait (optional)
- Wait activity: 60–120 seconds (tune to your source uploader behavior)
Enumerate batch
- Get Metadata/List → enumerate /raw/inbound/YYYY/MM/DD
- Filter to only new files (since last watermark or not in archive)
Process
- ForEach files → copy/transform (or load to staging and Spark job to batch merge)
Archive + Watermark
- Move files to /raw/archive/...
- Update control table last_processed_timestamp / last_processed_file
Release lock
- Delete /control/ingest.lock or set control row status='completed'
If this helps please mark this as a solution or a thumbs up!
Hi PAVAN_111
Thanks for reaching out to the Microsoft fabric community forum.I would also take a moment to thank deborshi_nag , tayloramy and tayloramy , for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.
I hope the above details help you fix the issue. If you still have any questions or need more help, feel free to reach out. We’re always here to support you .
Best Regards,
Community Support TeamHi PAVAN_111 ,
I hope the above details help you fix the issue. If you still have any questions or need more help, feel free to reach out. We’re always here to support you .
Best Regards,
Community Support Team
Hi PAVAN_111,
I can't think of a nice way to do this natively. What's the reason that you don't want each file ingested as its own event?
Generally speaking for event based ingestion, you want the data to be ingested as the event happens.
If you want to wait until all files have arrived, I agree with AntoineW, schedule a batch load at a time when you know all files have arrived, or have a pipeline that checks if all files are there, and ingests the data.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
Hi PAVAN_111 ,
There is one option you can try. While creating the event based trigger, in eventstream group the events for the files using group by option , add appropriate sliding window as per your usecase. This will reduce the number of events generation which are eventually passed to activator.
Now the key the setup would the groupby operation and the sliding window.
Let me know if you have any doubts !!
