Forum Discussion
Incremental uploading of files using pipeline
i want to automate a data pipeline in Microsoft Fabric so that it runs every time a file is uploaded to a specific local folder.but rather than running every file i only need the new file to get added rather than adding all at a time like incremental copy in tables.so could any one help me with this
Hi munindra ,
For local folders -> I would recommend setting up a python program that monitors the folder and calls the Fabric REST API to run the pipeline with parameters.
You can use watchdog python library to monitor your folder in local sytem. Please read the below blog:
https://www.kdnuggets.com/monitor-your-file-system-with-pythons-watchdog
7 Replies
- NandanHegde
Super User
Unfortunately, there is no direct way or in built functionality in Fabric data pipelines to acheive this.
You would need to have some custom logics for you to trigger once the file is uploaded in the local folder
- govindarajan_d
Hi munindra ,
You can use event triggers with Data pipeline to do that. Every time a file comes into the storage, you can trigger pipeline which contains metadata about that file (filename, time, EventType).
The below URL describes how to do that:
https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-storage-event-triggers
- NandanHegde
govindarajan_d Based on the treminology Specific local folder, I dont see that as a storage account but a server local folder. If that is the case, the above suggestion wont work.
I may be wrong here!!!!
- govindarajan_d
Hi NandanHegde,
Ahh.. Didn't catch the "local" word. That's true. It won't work on folders on local sytem!
But the event trigger doesn't work on local folders right?
- govindarajan_d
Hi munindra ,
For local folders -> I would recommend setting up a python program that monitors the folder and calls the Fabric REST API to run the pipeline with parameters.
You can use watchdog python library to monitor your folder in local sytem. Please read the below blog:
https://www.kdnuggets.com/monitor-your-file-system-with-pythons-watchdog
