topic Re: Event Trigger to execute pipeline when a new file is created inside Lakehouse in Data Engineering

Event Trigger to execute pipeline when a new file is created inside Lakehouse

abhiram342 — Thu, 31 Oct 2024 22:35:52 GMT

Hi,

We want to create an Event based trigger to run Data Pipeline whenever a new file is created inside Lakehouse.

Please let us know how to achieve it. I tried with storage account trigger and it works fine but I don't want to create seperate storage account for this scenario, instead want to leverage existing lakehouse .

Thanks,

Abhiram

Re: Event Trigger to execute pipeline when a new file is created inside Lakehouse

Anonymous — Fri, 01 Nov 2024 06:59:11 GMT

Hi @abhiram342 ,

In the Fabric Data Factory Pipeline Design canvas, there is a Triggers button that allows you to create Reflex triggers for the pipeline, or you can create triggers directly from Data Activator.

It's still in preview, but I think it does what you want.

For more information, please refer to:

Data pipelines storage event triggers in Data Factory (Preview) - Microsoft Fabric | Microsoft Learn

Re: Event based trigger to run Data Pipeline when ... - Microsoft Fabric Community

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

Re: Event Trigger to execute pipeline when a new file is created inside Lakehouse

FabianSchut — Fri, 01 Nov 2024 16:40:29 GMT

Unfortunately, The data activator (advanced pipeline trigger) is not working for a lakehouse yet. Here is a workaround someone posted: https://community.fabric.microsoft.com/t5/Data-Warehouse/Onelake-trigger-to-Ingest-CSV/m-p/4120366

Re: Event Trigger to execute pipeline when a new file is created inside Lakehouse

Anonymous — Mon, 04 Nov 2024 07:07:20 GMT

Hi @abhiram342 ,

Thanks to FabianSchut for the solution.

Were the above suggestions helpful in your situation? If this is the case, you may want to consider Accept is helpful suggestions to help others facing similar requirements.

Thank you very much for your cooperation!

Best Regards,
Yang
Community Support Team

Re: Event Trigger to execute pipeline when a new file is created inside Lakehouse

Anonymous — Thu, 14 Nov 2024 02:10:45 GMT

Hi @abhiram342 ,

Is my follow-up just to ask if the problem has been solved?

If so, can you accept the correct answer as a solution or share your solution to help other members find it faster?

Thank you very much for your cooperation!

Best Regards,
Yang
Community Support Team