Event based trigger to run Data Pipeline when a new file is created inside Lakehouse

[supriyapandey19]

Hi,

We want to create an Event based trigger to run Data Pipeline whenever a new file is created inside Lakehouse.

Please assist us in achieving this scenario, any work arounds or reference articles would be appreciated.

Thanks.

[jwinchell40]

@supriyapandey19 - As far as I know this only works with external ADLS Gen2 environments in Azure.  The ability to trigger a Pipeline when a file is created in a Fabric Lakehouse does not exist yet.

The process @FabianSchut sent a link for works really well.

[supriyapandey19]

Does it work for files placed in Fabric Lakehouse also? As, we can only see the option of getting the files detected from Azure storage account. 

Please explain if with current availability in Fabric event based triggers, is it possible to implement it for Files placed in Fabric Lakehouse?

[FabianSchut]

Hi @supriyapandey19,

you can take a look at this documenten: https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-storage-event-triggers

although still in preview, it think it can achieve what you are looking for. 

[supriyapandey19]

Does it work for files placed in Fabric Lakehouse ? As, we can only see the option of getting the files detected from Azure storage account. 

Please explain if with current availability in Fabric event based triggers, is it possible to implement it for Files placed in Fabric Lakehouse? If yes, how?

[FabianSchut]

You are right, it does not work for a lakehouse (yet I suppose).

[jwinchell40]

@supriyapandey19 - It does appear not work with a Fabric Lakehouse yet.  You can tie into Fabric events for Item creation/updates but creating a file in a Fabric Lakehouse doesn't fall under those events.  I"ll mess with Data Activator just to make sure.

I am wanting to do the same thing, so if I find a solution I will let you know.