Forum Discussion
Multiple pipeline runs triggered from single file creation in OneLake using Event Trigger
Hi MuhammedLabeeb,
single file creation triggers multiple pipeline runs is a known behavior in Microsoft Fabric and Azure Data Factory (ADF) when using storage event triggers. This phenomenon is often due to how Azure Blob Storage and Event Grid handle file creation events.
When a file is uploaded to Azure Data Lake Storage Gen2, especially using tools like Azure Storage Explorer, AzCopy, or programmatic methods, the upload process may involve multiple
steps please refer this doc :https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger?tabs=data-factory
- Chunked Uploads: Large files are often uploaded in chunks. Each chunk can emit its own event.
- Metadata Updates: After the file is fully uploaded, additional events like BlobPropertiesUpdated or BlobMetadataUpdated may be triggered. These multiple events can cause the event trigger to fire multiple times for what appears to be a single file creation.
Thanks,
Prashanth Are
MS Fabric community support
Hi MuhammedLabeeb,
Specifying a “blob path” with just a file extension near the root may cause an unexpected spike in triggered pipelines. Ensure that your trigger definitions are precise and not too broad, as this can lead to multiple triggers being fired
Can you please refer to below doc related Event-Driven Architecture Through Storage Event Trigger and let me know if this helps resolve your issue?
microsofteur.sharepoint.com/teams/MissionCriticalReadiness/Shared Documents/Forms/AllItems.aspx?id=%2Fteams%2FMissionCriticalReadiness%2FShared Documents%2F3rd edition - FY24%2FSession 01-Azure Data Factory %26 Azure Synapse Analytics%2FEvent-Driven Architecture Through Storage Event Triggers%2Epdf&parent=%2Fteams%2FMissionCriticalReadiness%2FShared Documents%2F3rd edition - FY24%2FSession 01-Azure Data Factory %26 Azure Synapse AnalyticsThanks,
Prashanth Are
Community SupportHi MuhammedLabeeb,
single file creation triggers multiple pipeline runs is a known behavior in Microsoft Fabric and Azure Data Factory (ADF) when using storage event triggers. This phenomenon is often due to how Azure Blob Storage and Event Grid handle file creation events.
When a file is uploaded to Azure Data Lake Storage Gen2, especially using tools like Azure Storage Explorer, AzCopy, or programmatic methods, the upload process may involve multiple
steps please refer this doc :https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger?tabs=data-factory
- Chunked Uploads: Large files are often uploaded in chunks. Each chunk can emit its own event.
- Metadata Updates: After the file is fully uploaded, additional events like BlobPropertiesUpdated or BlobMetadataUpdated may be triggered. These multiple events can cause the event trigger to fire multiple times for what appears to be a single file creation.
Thanks,
Prashanth Are
MS Fabric community support
Thank you for the update.
Just to clarify, we have only one pipeline configured, but it appears to be getting triggered multiple times for a single file upload.Additionally, the event-based trigger is scoped specifically to folder A inside the Lakehouse. However, we've observed that the pipeline is also triggered when files are uploaded to other folders such as B, C, or even the root folder.
We would appreciate it if you could kindly confirm whether this is expected behavior. If not, could you please advise if there are any additional settings required to ensure the trigger responds only to events within folder A?
- v-prasare
Hi MuhammedLabeeb,
Specifying a “blob path” with just a file extension near the root may cause an unexpected spike in triggered pipelines. Ensure that your trigger definitions are precise and not too broad, as this can lead to multiple triggers being fired
Can you please refer to below doc related Event-Driven Architecture Through Storage Event Trigger and let me know if this helps resolve your issue?
microsofteur.sharepoint.com/teams/MissionCriticalReadiness/Shared Documents/Forms/AllItems.aspx?id=%2Fteams%2FMissionCriticalReadiness%2FShared Documents%2F3rd edition - FY24%2FSession 01-Azure Data Factory %26 Azure Synapse Analytics%2FEvent-Driven Architecture Through Storage Event Triggers%2Epdf&parent=%2Fteams%2FMissionCriticalReadiness%2FShared Documents%2F3rd edition - FY24%2FSession 01-Azure Data Factory %26 Azure Synapse AnalyticsThanks,
Prashanth Are
