Data flow refresh when new file is added in Sharepoint folder

[manishkumar13]

Hi all,

I receive a new file every day in a SharePoint folder, and I have created a GenFlow 2 pipeline that takes the SharePoint file and appends the data to a lake house. However, I am stuck on how to trigger the data flow refresh when a new file is added to the SharePoint folder. I cannot use a scheduled refresh because the file can arrive at any time, and if the file is missed, it will append duplicate values if a new file is not added. I have tried online solutions and Copilot suggested writing a notebook, but it did not work in my case.

Is there anything I can do in the data pipeline? Please suggest the correct approach.

@nilendraFabric , @NandanHegde @FabianSchut 

Thank you!

[manishkumar13]

Thanks for the reply,

Can I use notebook code to automate rather than using Power Automate?

[NandanHegde · Accepted solution]

Unfortunately, notebook needs to be triggered and there is no event driven trigger for notebook.
Power automate has an inbulit mechanism to trigger on arrival of file in sharepoint

[Anonymous]

Hi @manishkumar13 ,

Based on your description, try using power automate+data factory. when new files are added to the SharePoint folder, the Azure Data Factory (ADF) data flow is automatically triggered to append the file data to Lakehouse.

For more details, you can refer to below document:

Use SharePoint and Power Automate to build workflows - Power Automate | Microsoft Learn

Parameters - Microsoft Fabric | Microsoft Learn

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[NandanHegde]

You can use Power automate to trigger something on file arrival within sharepoint :

https://manueltgomes.com/microsoft/sharepoint-when-a-file-is-created-in-a-folder-trigger/

And once triggered, trigger the Data pipeline and in turn dataflow via power automate via API