topic Can we trigger data pipelines when file is updated in Fabric Lakehouse via events in Data Engineering

Can we trigger data pipelines when file is updated in Fabric Lakehouse via events

Anonymous — Mon, 11 Nov 2024 11:52:22 GMT

Hello people,

I had a requirement my fabric lakehouse will get the data dump everyday from external sources from there on am using data pipelines to use that data for further transformations. Here i want to trigger the data pipelines automatically whenever lakehouse gets updated everyday through events (using event driven architecture just like in azure how we use blob storage events). I did lot of research and found presently there is no way to do that i want to know is this feature is expected in future releases in Fabric Roadmap,

If any one encountered this issue let us know

Re: Can we trigger data pipelines when file is updated in Fabric Lakehouse via events

Anonymous — Thu, 14 Nov 2024 01:57:31 GMT

HI @Anonymous,

You can tried to create 'workspace item events' to 'set alert' to trigger and run the data pipeline:

Set alerts based on Fabric events in Real-Time hub - Microsoft Fabric | Microsoft Learn

BTW, since your requirement is more related to real-time intelligence, you can also post ot that sub forum to get further support.

Eventstream - Microsoft Fabric Community

Regards,

Xiaoxin Sheng

Re: Can we trigger data pipelines when file is updated in Fabric Lakehouse via events

FabianSchut — Tue, 12 Nov 2024 06:49:55 GMT

Hi, it is indeed not possible to create an event based on a Lakehouse update. Here's an idea on Microsoft for it you can vote: https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=3a8a68e8-2c8a-ef11-9443-6045bdafcead.

This is another topic with the same request:
https://community.fabric.microsoft.com/t5/Data-Engineering/Event-Trigger-to-execute-pipeline-when-a-new-file-is-created/m-p/4266159

Someone posted a workaround, it may work for you: https://community.fabric.microsoft.com/t5/Data-Warehouse/Onelake-trigger-to-Ingest-CSV/m-p/4120366

Re: Can we trigger data pipelines when file is updated in Fabric Lakehouse via events

Anonymous — Thu, 14 Nov 2024 01:58:48 GMT

Hi @Anonymous,

Any update on this? Did these suggestion help for your scenario? If that is the case, you can consider Kudo or Accept the helpful suggestions to help others who faced similar requirements.

Regards,

Xiaoxin Sheng

Re: Can we trigger data pipelines when file is updated in Fabric Lakehouse via events

Anonymous — Thu, 14 Nov 2024 08:32:18 GMT

Hi @Anonymous

Thanks for your reply, am glad. we can create alerts on workspace items like pipeline, notebook, lakehouse, report etc whenever we create/delete them it triggers an event and sends this to event stream and we can trigger Data pipeline I tried this it will work for workspace items level. But this is not my requirement.

My requirement is we are receiving data daily to lakehouse in Files section in form of csv files, so when we receive new csv files everyday data pipeline should be triggered automatically maybe listening to Lakehouse events, I am not sure this feature is there or not in fabric Have you tried this approach am looking for this solution

Re: Can we trigger data pipelines when file is updated in Fabric Lakehouse via events

Anonymous — Fri, 15 Nov 2024 06:14:53 GMT

Hi @Anonymous ,

After I double check on these, I found currently it seem not include correspond type of condition to invoke pipeline.
For this scenario, I'd like to suggest you submit idea to add these types of conditions in aleart or add REST API support to run/cancel data pipeline.

Microsoft Fabric Ideas

Regards,

Xiaoxin Sheng