Forum Discussion
Event based trigger causing duplicate runs
I figured out a workaround for this. Instead of using File creation, I used file deletion event.
The deletion event doesnt cause duplicate runs! and I can still capture the information i need from the file name after deletion.
This also works as an automatic clean up activity as well haha 😁
Thank you svenchio , tayloramy , & v-kpoloju-msft for your valuable inputs! 🤝
Hi Karol_PL,
Thank you for reaching out to the Microsoft Fabric Community Forum.
You are observing this behaviour because a single logical file write in One Lake generates multiple FileCreated events, each representing a different internal operation. While the event type stays the same, the api field changes (for example, CreateFile when the write starts and FlushWithClose when the file is finalized). This is expected behavior in Fabric, and without additional filtering, it can result in duplicate pipeline triggers for the same file.
To prevent this, it does make sense to filter your trigger based on the api parameter. The recommended approach is to trigger the pipeline only when api = FlushWithClose, as this ensures the file is fully written and ready for processing. Adding this condition will allow the pipeline to run once per file and avoid duplicates, while still reliably capturing all completed file uploads.
Refer these links:
1. https://learn.microsoft.com/en-us/fabric/real-time-hub/tutorial-build-event-driven-data-pipelines 2. https://learn.microsoft.com/en-us/fabric/real-time-hub/explore-fabric-onelake-eventsThanks again for using the Microsoft Fabric Community Forum.
Hi Hamza_Amir , thinkig about your description, I would like to offer an hyphotesis of what may be happending, since the Microsoft.Fabric.OneLake.FileCreated event is raised for both file creation and updates, I,m thinking that if your notebook writes the text file and then somehow modify it (e.g., renames, appends, or sets metadata), you’ll get two events quickly .
Ref. Explore OneLake events in Fabric Real-Time hub - Microsoft Fabric | Microsoft Learn
Before exploring some possible solutions, I think it would be worth to prove/discard this hyphotehsis, can you check if your notebook is only creating or, if it's creating and updating, then you know why two events are getting fired... let us know the outcome and move from there 😉 ... all the best!
If you find this info useful, a thumb's up would be nice, and if I'm right, mark this as a solution/explanaition.
Hi, Thank you for your reply! I tested your hypothesis and double checked the notebook, but all i am doing is a simple write command once, here is the actual code in the cell where i create the test file:
file_content = ( f"brand_id={brand_id}\n" f"runtime={runtime}\n" f"pipeline_run_id={pipeline_run_id}\n" ) notebookutils.fs.put(file_path, file_content, overwrite=False)Hi Hamza_Amir , ok, please try overwrite=True, even though you’re creating a new file, forcing overwrite=True can reduce any immediate follow‑up “update” and check if the double event get's trigger. Let us know the outcome.
Hey svenchio!, I made this change in the code:
file_content = ( f"brand_id={brand_id}\n" f"runtime={runtime}\n" f"pipeline_run_id={pipeline_run_id}\n" ) notebookutils.fs.put(file_path, file_content, overwrite=True)Still getting duplicate runs 😔