Currently, a single completed OneLake file can generate duplicate events and trigger the downstream Fabric pipeline multiple times. Filtering on data.api = FlushWithClose helps identify completed fi...
Fabric could provide an optional Exactly-once per file mode for OneLake event-based triggers.
When enabled, Fabric could:
- Continue monitoring Microsoft.Fabric.OneLake.FileCreated.
- Wait for the final file operation, such as data.api = FlushWithClose.
- Internally deduplicate repeated deliveries using the CloudEvent source + id.
- Optionally deduplicate file operations using the OneLake path together with the file version or ETag.
- Trigger the destination pipeline only once for that completed file version.
- Maintain the deduplication state for at least the event retry period.
The trigger configuration could offer two delivery modes:
- At-least-once, current behavior
- Exactly-once per completed file, managed by Fabric
Fabric could also expose the deduplication result in monitoring, for example:
- Event received
- Duplicate event suppressed
- Pipeline triggered
- Delivery or processing failed
This would reduce the need for every customer to build a separate control table, locking mechanism, and idempotency logic in each downstream pipeline. Duplicate Event-Based Triggers for Same OneLake File Despite Filtering on data.api = FlushWithClose | Microsoft Fabric Community
