Forum Discussion
Event based trigger causing duplicate runs
I figured out a workaround for this. Instead of using File creation, I used file deletion event.
The deletion event doesnt cause duplicate runs! and I can still capture the information i need from the file name after deletion.
This also works as an automatic clean up activity as well haha 😁
Thank you svenchio , tayloramy , & v-kpoloju-msft for your valuable inputs! 🤝
Hi Karol_PL,
Thank you for reaching out to the Microsoft Fabric Community Forum.
You are observing this behaviour because a single logical file write in One Lake generates multiple FileCreated events, each representing a different internal operation. While the event type stays the same, the api field changes (for example, CreateFile when the write starts and FlushWithClose when the file is finalized). This is expected behavior in Fabric, and without additional filtering, it can result in duplicate pipeline triggers for the same file.
To prevent this, it does make sense to filter your trigger based on the api parameter. The recommended approach is to trigger the pipeline only when api = FlushWithClose, as this ensures the file is fully written and ready for processing. Adding this condition will allow the pipeline to run once per file and avoid duplicates, while still reliably capturing all completed file uploads.
Refer these links:
1. https://learn.microsoft.com/en-us/fabric/real-time-hub/tutorial-build-event-driven-data-pipelines 2. https://learn.microsoft.com/en-us/fabric/real-time-hub/explore-fabric-onelake-eventsThanks again for using the Microsoft Fabric Community Forum.
I encountered the same issue with duplicate pipeline triggers and it looks like even though the event type FileCreated remains consistent across events, the associated api field varies depending on the underlying file operation, does it make sense to simply filter action based on api parameter?
Hi Karol_PL,
Thank you for reaching out to the Microsoft Fabric Community Forum.
You are observing this behaviour because a single logical file write in One Lake generates multiple FileCreated events, each representing a different internal operation. While the event type stays the same, the api field changes (for example, CreateFile when the write starts and FlushWithClose when the file is finalized). This is expected behavior in Fabric, and without additional filtering, it can result in duplicate pipeline triggers for the same file.
To prevent this, it does make sense to filter your trigger based on the api parameter. The recommended approach is to trigger the pipeline only when api = FlushWithClose, as this ensures the file is fully written and ready for processing. Adding this condition will allow the pipeline to run once per file and avoid duplicates, while still reliably capturing all completed file uploads.
Refer these links:
1. https://learn.microsoft.com/en-us/fabric/real-time-hub/tutorial-build-event-driven-data-pipelines 2. https://learn.microsoft.com/en-us/fabric/real-time-hub/explore-fabric-onelake-events
Thanks again for using the Microsoft Fabric Community Forum.