Forum Discussion
How to Trigger pipeline using Event trigger
I have an ETL which I need to trigger when blob created or updated.
In my Azure storage, My blob path is
landing/General/test.csv
My storage account:Peausefilesystem
How do I get this created, using event triggers with filters.
When I added a trigger with the path filer by subject
Sbject begins with landing/General/test.csv, it triggers everytime when a blob created in any directory than what I have filtered.
Bit confisued with the set up.
Hi KGU_16031986
I know you are only having issues with a small part of this, but for other community users I will detail the full configuration. Firstly, you will need to register 'Microsoft.EventGrid' in your azure subscription. Go to Subscription/Resource providers and search 'Microsoft.EventGrid' and click 'register'.
In your data pipeline, select 'triggers' from the top toolbar, then 'add trigger'. You'll be presented with this menu on the right.
When you select 'select source monitor' you are presented with a list of option, choose 'Azure blob storage events', then you will see a menu where you can select your storage account.
Next you need to configure the alert, choose all events you want to be included in the trigger, for this example i've select 'blobcreated', 'blobdeleted' and 'blobrenamed'.
Now we need to add a filter that contains the full path. Use 'Subject', 'Begins with', then add the following path
/blobServices/default/containers/<your-storage-account-name>/blobs/landing/General/test.csvClick save, then create rule.
In my test pipeline, I have a set variable activity to show the successful test.
When I change the title of test.csv in my storage account the pipeline runs.
You can view the details of the trigger by opening the activator. Go to 'triggers' on the top tool bar of the pipeline, view triggers, the 'open in Activator'
I hope this helps, please give a thumbs up and mark as solved if it does, thanks!
3 Replies
Hi KGU_16031986
I know you are only having issues with a small part of this, but for other community users I will detail the full configuration. Firstly, you will need to register 'Microsoft.EventGrid' in your azure subscription. Go to Subscription/Resource providers and search 'Microsoft.EventGrid' and click 'register'.
In your data pipeline, select 'triggers' from the top toolbar, then 'add trigger'. You'll be presented with this menu on the right.
When you select 'select source monitor' you are presented with a list of option, choose 'Azure blob storage events', then you will see a menu where you can select your storage account.
Next you need to configure the alert, choose all events you want to be included in the trigger, for this example i've select 'blobcreated', 'blobdeleted' and 'blobrenamed'.
Now we need to add a filter that contains the full path. Use 'Subject', 'Begins with', then add the following path
/blobServices/default/containers/<your-storage-account-name>/blobs/landing/General/test.csvClick save, then create rule.
In my test pipeline, I have a set variable activity to show the successful test.
When I change the title of test.csv in my storage account the pipeline runs.
You can view the details of the trigger by opening the activator. Go to 'triggers' on the top tool bar of the pipeline, view triggers, the 'open in Activator'
I hope this helps, please give a thumbs up and mark as solved if it does, thanks!
Hi wardy912
Thanks for the great explanation. Glad you shared this to help anyone who struggles with this problem
I managed to handle the issue by adding a condition to the trigger. Now my ETL triggers only when it detected the file I expect to see.
I use Event strem , which is already filtered by the subject, and then activator with the condition filter by the url contains the file name. I will add the soution here when I get a time.
Trying to learn the concept of this. This is a bit different to Synapse or ADF though.Hi KGU_16031986 ,
Thanks for reaching out to the Microsoft fabric community forum.
wardy912 ,
Thank you for your prompt response it's truly appreciated.
Best Regards,
Lakshmi
