topic Data flow refresh when new file is added in Sharepoint folder in Dataflow

Data flow refresh when new file is added in Sharepoint folder

manishkumar13 — Wed, 26 Feb 2025 13:41:05 GMT

Hi all,

I receive a new file every day in a SharePoint folder, and I have created a GenFlow 2 pipeline that takes the SharePoint file and appends the data to a lake house. However, I am stuck on how to trigger the data flow refresh when a new file is added to the SharePoint folder. I cannot use a scheduled refresh because the file can arrive at any time, and if the file is missed, it will append duplicate values if a new file is not added. I have tried online solutions and Copilot suggested writing a notebook, but it did not work in my case.

Is there anything I can do in the data pipeline? Please suggest the correct approach.

nilendraFabric , NandanHegde FabianSchut

Thank you!

Re: Data flow refresh when new file is added in Sharepoint folder

NandanHegde — Thu, 27 Feb 2025 08:21:15 GMT

You can use Power automate to trigger something on file arrival within sharepoint :

https://manueltgomes.com/microsoft/sharepoint-when-a-file-is-created-in-a-folder-trigger/

And once triggered, trigger the Data pipeline and in turn dataflow via power automate via API

Re: Data flow refresh when new file is added in Sharepoint folder

Anonymous — Thu, 27 Feb 2025 09:05:11 GMT

Hi manishkumar13 ,

Based on your description, try using power automate+data factory. when new files are added to the SharePoint folder, the Azure Data Factory (ADF) data flow is automatically triggered to append the file data to Lakehouse.

For more details, you can refer to below document:

Use SharePoint and Power Automate to build workflows - Power Automate | Microsoft Learn

Parameters - Microsoft Fabric | Microsoft Learn

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Data flow refresh when new file is added in Sharepoint folder

manishkumar13 — Fri, 28 Feb 2025 09:27:34 GMT

Thanks for the reply,

Can I use notebook code to automate rather than using Power Automate?

Re: Data flow refresh when new file is added in Sharepoint folder

NandanHegde — Fri, 28 Feb 2025 09:52:33 GMT

Unfortunately, notebook needs to be triggered and there is no event driven trigger for notebook.
Power automate has an inbulit mechanism to trigger on arrival of file in sharepoint