Eventstreams and Activator duplicate runs

[jajumonville]

I have been having odd duplicate runs of Pipelines in Fabric because the Activator is triggering the 2x for the same event.

Senario: ADF creates a small csv file in a ADLS Gen2 location at the successful completion of a pipeline. This creatation of this file is set to be monitored by a fabric eventstream ( blob creation) and on of serveral activators will see the name of the test.txt file and run a fabric pipeline, then delete the file as a cleanup operation.

What is happing is that the text.txt will arrive and fabric will then trigger the pipeline twice. when one of them finail has come contention with the other it will fail, the other will succeed.

when you crate the fabric event stream it seems to only be capaible of looking over the entire container as there is no way to filter down like you can when do the add trigger method in a pipline dialoge to get to the file name. And doing the latter first and adding activators to it seems to not allow things to work correctly either.

Anyone else having these odd happenings, is this a flaw in the preview version of this, is there a way around it.

[v-saisrao-msft]

Hi @jajumonville ,

Could you please confirm if you've submitted this as an idea in the Ideas Forum? If so, sharing the link here would be helpful for other community members who may have similar feedback.

As we haven’t heard back, we’re closing this thread. For any further discussions or questions, please raise a new thread in the Microsoft Fabric Community Forum — we’ll be happy to assist.

Thank you for being part of the Microsoft Fabric Community.

[v-saisrao-msft]

Hi @jajumonville @andjjumonville,

I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.

Thank you.

[andjjumonville]

unfortuanately i believe the problem lies with ADF not just writing the file at once but rather is generating unseen .tmp files both on creatation and deletion, with the event stream set to only pay attention to blob creation, it still triggers when then the blob is deleted. - This preview function does not appear to be suitable for production work yet. There needs to be control in the pipelines to prevent duplicate runs, this doesn't exist either

[v-saisrao-msft]

Hi @jajumonville  @andjjumonville,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

[jajumonville]

This preview function is still too primitive to use without extreme amounts of workarounds, we are abandoning this and implementing a timed watch on a folder to check for the file, this is not ideal as 1 it eats up the amount of notebooks that can be called in a pipeline and 2 it just signals some things are ready for prime time yet, meaning we will need to slow our adoption of fabric down.

[v-saisrao-msft · Accepted solution]

Hi @jajumonville @andjjumonville,

Thank you for your feedback. I completely understand the importance of having a seamless and efficient solution, and I acknowledge the difficulties posed by the current preview functionality.

Since Fabric is still evolving, your insights are incredibly valuable in shaping its future. If you haven’t already, I encourage you to submit your feedback.

Microsoft Fabric Ideas Fabric Ideas - Microsoft Fabric Community, as this helps the product team prioritize improvements.

If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.     

Thank you. 

[v-saisrao-msft]

Hi @jajumonville @andjjumonville,

Thank you for identifying the core issue ADF generates temporary .tmp files during both creation and deletion. Fabric Eventstream, although configured to monitor blob creation, is triggering on both events. This lack of fine-grained filtering and state validation in Eventstream is causing the problem. We appreciate you highlighting this, as it helps us understand the limitations of the preview.

• Use a Success Marker File Instead of triggering on the CSV file, modify ADF to create a "done.signal" file only after successful completion. Eventstream can be configured to monitor only this marker file, preventing premature triggers.
• Modify ADF's Write Behavior If possible, adjust ADF to write to a temporary directory first, then move the final file to the monitored location once it's complete. This can help avoid detecting .tmp files.

Since it is still in preview, some of these limitations are expected, and improvements are actively being worked on. The full release will likely bring better event filtering options and built-in mechanisms to prevent duplicate pipeline executions.

If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.   

Thank you. 

[v-saisrao-msft]

Hi @jajumonville,

Thank you for reaching out to the Fabric support community.

Regarding the query about duplicate runs of Pipelines in Microsoft Fabric, I understand you are facing an issue where the Activator triggers the pipeline twice for the same event. This is causing contention, resulting in one instance failing while the other succeeds. 

Please review the following steps: 

• Please ensure the Event stream is configured correctly to monitor the specific events of interest. Verify whether there are settings or options available to filter events more precisely. 
• Develop a system to assign unique identifiers to each event, preventing multiple activations of the same event. This can be achieved by incorporating metadata or tags into the events. 
• Introduce a debounce mechanism to avoid multiple triggers for the same event in a short period. This can be implemented by adding a delay or cooldown period between triggers. 
• Implement custom logic in the Activators to verify if the event has already been processed before initiating the pipeline. This may involve keeping a record or log of processed events. 

 Please refer Below link: 

Data pipelines storage event triggers in Data Factory (Preview) - Microsoft Fabric | Microsoft Learn 

If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.   

Thank you.