Power BI is turning 10! Tune in for a special live episode on July 24 with behind-the-scenes stories, product evolution highlights, and a sneak peek at what’s in store for the future.
Save the dateEnhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.
i want to automate a data pipeline in Microsoft Fabric so that it runs every time a file is uploaded to a specific local folder.but rather than running every file i only need the new file to get added rather than adding all at a time like incremental copy in tables.so could any one help me with this
Solved! Go to Solution.
Hi @munindra ,
For local folders -> I would recommend setting up a python program that monitors the folder and calls the Fabric REST API to run the pipeline with parameters.
You can use watchdog python library to monitor your folder in local sytem. Please read the below blog:
https://www.kdnuggets.com/monitor-your-file-system-with-pythons-watchdog
Hi @munindra ,
You can use event triggers with Data pipeline to do that. Every time a file comes into the storage, you can trigger pipeline which contains metadata about that file (filename, time, EventType).
The below URL describes how to do that:
https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-storage-event-triggers
But the event trigger doesn't work on local folders right?
Hi @munindra ,
For local folders -> I would recommend setting up a python program that monitors the folder and calls the Fabric REST API to run the pipeline with parameters.
You can use watchdog python library to monitor your folder in local sytem. Please read the below blog:
https://www.kdnuggets.com/monitor-your-file-system-with-pythons-watchdog
By Local folder, you mean on your local system or any server correct?
If that is the case, event trigger wont work as it is specific to blobs
@govindarajan_d Based on the treminology Specific local folder, I dont see that as a storage account but a server local folder. If that is the case, the above suggestion wont work.
I may be wrong here!!!!
Hi @NandanHegde,
Ahh.. Didn't catch the "local" word. That's true. It won't work on folders on local sytem!
Unfortunately, there is no direct way or in built functionality in Fabric data pipelines to acheive this.
You would need to have some custom logics for you to trigger once the file is uploaded in the local folder