Forum Discussion
Duplicate Event-Based Triggers for Same OneLake File Despite Filtering on data.api = FlushWithClose
Thanks for the workaround. However, this is quite complex for a basic event-based ingestion scenario. Implementing claim, deduplication, status tracking, concurrency handling, and completion logic in every pipeline creates significant overhead.
The expected behavior should ideally be: One completed file → One event → One pipeline execution
Is there a simpler built-in option, or is a product-level fix planned for duplicate OneLake file events?
Community SupportHi mahadev93 ,
Event stream currently has no built-in option to guarantee exactly once delivery or limit processing to one pipeline execution per file. Microsoft documents Fabric/One Lake events as at-least-once delivery, which means duplicate events may occur.
Although data.api = FlushWithClose can help filter for completed writes, it does not provide exactly-once semantics. The supported approach is to deduplicate events using source + id and ensure processing is idempotent.
To request this feature for future consideration, please submit an idea here ( Fabric Ideas | Microsoft Fabric Community ).
Thanks!!
Thanks for confirming. I understand that OneLake events currently use at-least-once delivery and that pipeline-level deduplication is the supported approach.
However, implementing event claiming, deduplication, status tracking, concurrency handling, and idempotency in every pipeline is a complex workaround for a basic file-arrival scenario.
I have submitted a Fabric Idea requesting built-in support for one pipeline trigger per completed file. Provide Exactly-Once Triggering for OneLake File Events | Microsoft Fabric Community