Forum Discussion
Duplicate Event-Based Triggers for Same OneLake File Despite Filtering on data.api = FlushWithClose
Community SupportHi mahadev93 ,
Thank you for reaching out to fabric community.
This behavior is expected for Fabric OneLake events. Microsoft documents that Fabric events follow at-least-once delivery, meaning the same event may occasionally be delivered multiple times. The Microsoft.Fabric.OneLake.FileCreated event is also triggered when a file is either created or replaced.
Using data.api = FlushWithClose as a filter can help identify a completed write, but it cannot guarantee exactly-once delivery.
Continue filtering on FileCreated, subject/path, and data.api = FlushWithClose.
Deduplicate events using source + id, which Microsoft identifies as the event's unique identifier.
Ensure the downstream pipeline is idempotent, so reprocessing the same file or event does not produce duplicate results.
As a result, Eventstream filtering alone cannot guarantee exactly one pipeline execution per file. The recommended solution is to combine event deduplication with idempotent file processing.
Fabric OneLake Events in Fabric Real-Time Hub - Microsoft Fabric | Microsoft Learn
Events delivery guarantees - Microsoft Fabric | Microsoft Learn
Thanks!!
Thanks for the workaround. However, this is quite complex for a basic event-based ingestion scenario. Implementing claim, deduplication, status tracking, concurrency handling, and completion logic in every pipeline creates significant overhead.
The expected behavior should ideally be: One completed file → One event → One pipeline execution
Is there a simpler built-in option, or is a product-level fix planned for duplicate OneLake file events?
- v-sathmakuri
Hi mahadev93 ,
Event stream currently has no built-in option to guarantee exactly once delivery or limit processing to one pipeline execution per file. Microsoft documents Fabric/One Lake events as at-least-once delivery, which means duplicate events may occur.
Although data.api = FlushWithClose can help filter for completed writes, it does not provide exactly-once semantics. The supported approach is to deduplicate events using source + id and ensure processing is idempotent.
To request this feature for future consideration, please submit an idea here ( Fabric Ideas | Microsoft Fabric Community ).
Thanks!!
Thanks for confirming. I understand that OneLake events currently use at-least-once delivery and that pipeline-level deduplication is the supported approach.
However, implementing event claiming, deduplication, status tracking, concurrency handling, and idempotency in every pipeline is a complex workaround for a basic file-arrival scenario.
I have submitted a Fabric Idea requesting built-in support for one pipeline trigger per completed file. Provide Exactly-Once Triggering for OneLake File Events | Microsoft Fabric Community
- ShivekMaharaj
Memorable Member
Hi mahadev93,
The Community Support answers are correct about Fabric events using at-least-once delivery. I would add one diagnostic step because it determines what you should actually use as the deduplication key.
For the two FileCreated events that caused the duplicate pipeline runs, compare the CloudEvent source and id values.
Microsoft's Fabric event delivery documentation states that if the same event is redelivered because of a retry, it keeps the same source + id.
So:
- same source + same id -> duplicate delivery of the same event; deduplicate using source + id
- same file subject, but different event IDs -> Fabric considers them distinct events, so source + id deduplication will intentionally allow both
That second case is worth checking because Microsoft's OneLake event schema exposes id, subject, eTag, clientRequestId, requestId, api, and sequencer, and FileCreated can be raised when a file is created or updated/replaced.I would therefore capture those fields for the two events and compare them before choosing the idempotency key.
Your data.api = FlushWithClose filter is still useful for filtering to the completed write operation, but it does not change the at-least-once delivery guarantee.
If your timestamped filename is guaranteed to represent one immutable file arrival and will never legitimately be replaced, you could additionally use the OneLake subject/file path as the business-level idempotency key before starting the expensive part of the pipeline.
If the same path can legitimately be overwritten or reprocessed, I would not deduplicate on subject alone; use a stable business/file-arrival identifier instead.
I do not see a currently documented Eventstream option that provides exactly-once pipeline invocation for OneLake events. Microsoft's documented design is still to make the consumer idempotent and tolerate duplicate delivery.
AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.
