Forum Discussion

JohnAG's avatar
JohnAG
Frequent Visitor
2 years ago
Solved

DataFlow Gen2 process new file

Hey guys,  I have couple questions: 1. How can I invoke my DataFlow to process the new file that landed in Lakehouse? See image at the bottom of the thread.  2. How can I tell the DataFlow to proc...
data engineering
Data Integration
data science
data warehouse
  • JohnAG's avatar
    JohnAG
    2 years ago

    I had to move it to Power Query inside PBIX file instead of hopping around Fabric services as they still not fully developped to handle a fully automated incremental ETL.   

v-cboorla-msft's avatar
v-cboorla-msft
Icon for Microsoft Employee rankMicrosoft Employee
2 years ago

Hi JohnAG 

 

Thanks for using Microsoft Fabric Community.
At this time, we are reaching out to the internal team to get some help on this.
We will update you once we hear back from them.

Appreciate your patience.


Thanks

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Icon for Microsoft Employee rankMicrosoft Employee
    2 years ago

    Hi JohnAG 

     

    1. This could be possible by event driven refresh, but there is only scheduled refresh available in Fabric right now. You can write your own solution using REST APIs, etc. to poll for changes to the file system.
    2. You would want to connect to the folder location of the storage and filter out the content either by Name / date or some other metadata factor and choose to only append those files to your table.

    I hope this helps. Please do let us know if you have any further questions.

    • v-cboorla-msft's avatar
      v-cboorla-msft
      Icon for Microsoft Employee rankMicrosoft Employee
      2 years ago

      Hi JohnAG 

       

      We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
      In case if you have any resolution please do share that same with the community as it can be helpful to others.
      Otherwise, will respond back with the more details and we will try to help.


      Thanks

      • JohnAG's avatar
        JohnAG
        Frequent Visitor
        2 years ago

        I had to move it to Power Query inside PBIX file instead of hopping around Fabric services as they still not fully developped to handle a fully automated incremental ETL.