Forum Discussion
Solved
DataFlow Gen2 process new file
Hey guys, I have couple questions: 1. How can I invoke my DataFlow to process the new file that landed in Lakehouse? See image at the bottom of the thread. 2. How can I tell the DataFlow to proc...
I had to move it to Power Query inside PBIX file instead of hopping around Fabric services as they still not fully developped to handle a fully automated incremental ETL.
v-cboorla-msft
Microsoft Employee
Microsoft EmployeeHi JohnAG
- This could be possible by event driven refresh, but there is only scheduled refresh available in Fabric right now. You can write your own solution using REST APIs, etc. to poll for changes to the file system.
- You would want to connect to the folder location of the storage and filter out the content either by Name / date or some other metadata factor and choose to only append those files to your table.
I hope this helps. Please do let us know if you have any further questions.
v-cboorla-msftMicrosoft Employee
Microsoft EmployeeHi JohnAG
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.
Thanks
I had to move it to Power Query inside PBIX file instead of hopping around Fabric services as they still not fully developped to handle a fully automated incremental ETL.