Incremental uploading of files using pipeline

[munindra]

 i want to automate a data pipeline in Microsoft Fabric so that it runs every time a file is uploaded to a specific local folder.but rather than running every file i only need the new file to get added rather than adding all at a time like incremental copy in tables.so could any one help me with this

[govindarajan_d]

Hi @munindra ,

You can use event triggers with Data pipeline to do that. Every time a file comes into the storage, you can trigger pipeline which contains metadata about that file (filename, time, EventType). 

The below URL describes how to do that:

https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-storage-event-triggers

[munindra]

But the event trigger doesn't work on local folders right?

[govindarajan_d · Accepted solution]

Hi @munindra , 

For local folders -> I would recommend setting up a python program that monitors the folder and calls the Fabric REST API to run the pipeline with parameters. 

You can use watchdog python library to monitor your folder in local sytem. Please read the below blog:
https://www.kdnuggets.com/monitor-your-file-system-with-pythons-watchdog

[NandanHegde]

By Local folder, you mean on your local system or any server correct?
If that is the case, event trigger wont work as it is specific to blobs

[NandanHegde]

@govindarajan_d  Based on the treminology Specific local folder, I dont see that as a storage account but a server local folder. If that is the case, the above suggestion wont work.

I may be wrong here!!!! 

[govindarajan_d]

Hi @NandanHegde,

Ahh.. Didn't catch the "local" word.  That's true. It won't work on folders on local sytem!

[NandanHegde]

Unfortunately, there is no direct way or in built functionality in Fabric data pipelines to acheive this.

You would need to have some custom logics for you to trigger once the file is uploaded in the local folder