Currently, a single completed OneLake file can generate duplicate events and trigger the downstream Fabric pipeline multiple times.
Filtering on data.api = FlushWithClose helps identify completed file writes, but it does not guarantee exactly-once delivery. The recommended workaround requires every pipeline to implement event claiming, deduplication, status tracking, concurrency handling, and idempotent processing.
This creates significant complexity for a basic event-driven ingestion scenario, particularly when many files or pipelines are involved.
Expected behavior: One completed file → One valid event → One pipeline execution
Please provide one of the following:
- A built-in deduplication option for OneLake events
- An exactly-once trigger mode
- A unique, stable deduplication key with native pipeline support
- A configurable option to trigger only once for each finalized file
This enhancement would simplify event-driven ingestion, reduce unnecessary pipeline executions, and improve scalability.
1 Comment
Fabric could provide an optional Exactly-once per file mode for OneLake event-based triggers.
When enabled, Fabric could:
- Continue monitoring Microsoft.Fabric.OneLake.FileCreated.
- Wait for the final file operation, such as data.api = FlushWithClose.
- Internally deduplicate repeated deliveries using the CloudEvent source + id.
- Optionally deduplicate file operations using the OneLake path together with the file version or ETag.
- Trigger the destination pipeline only once for that completed file version.
- Maintain the deduplication state for at least the event retry period.
The trigger configuration could offer two delivery modes:
- At-least-once, current behavior
- Exactly-once per completed file, managed by Fabric
Fabric could also expose the deduplication result in monitoring, for example:
- Event received
- Duplicate event suppressed
- Pipeline triggered
- Delivery or processing failed
This would reduce the need for every customer to build a separate control table, locking mechanism, and idempotency logic in each downstream pipeline. Duplicate Event-Based Triggers for Same OneLake File Despite Filtering on data.api = FlushWithClose | Microsoft Fabric Community
