Forum Discussion
Event based trigger causing duplicate runs
- 7 months ago
I figured out a workaround for this. Instead of using File creation, I used file deletion event.
The deletion event doesnt cause duplicate runs! and I can still capture the information i need from the file name after deletion.
This also works as an automatic clean up activity as well haha 😁
Thank you svenchio , tayloramy , & v-kpoloju-msft for your valuable inputs! 🤝
- 6 months ago
Hi Karol_PL,
Thank you for reaching out to the Microsoft Fabric Community Forum.
You are observing this behaviour because a single logical file write in One Lake generates multiple FileCreated events, each representing a different internal operation. While the event type stays the same, the api field changes (for example, CreateFile when the write starts and FlushWithClose when the file is finalized). This is expected behavior in Fabric, and without additional filtering, it can result in duplicate pipeline triggers for the same file.
To prevent this, it does make sense to filter your trigger based on the api parameter. The recommended approach is to trigger the pipeline only when api = FlushWithClose, as this ensures the file is fully written and ready for processing. Adding this condition will allow the pipeline to run once per file and avoid duplicates, while still reliably capturing all completed file uploads.
Refer these links:
1. https://learn.microsoft.com/en-us/fabric/real-time-hub/tutorial-build-event-driven-data-pipelines 2. https://learn.microsoft.com/en-us/fabric/real-time-hub/explore-fabric-onelake-eventsThanks again for using the Microsoft Fabric Community Forum.
Hi Hamza_Amir,
This is fun. I have to agree with svenchio that somehow the file must be getting created and then being modified, I wonder if notebookutils first creates an empty file and then writes to it immediately after?
What I would do here is use an EventStream to transform the file event data, then shove it into a KQL database, and have activator sit on top of that.
You can use a Session Window to help pluck out duplicate events and ensure that only unique events end up in the KQL Database.