Forum Discussion

JohnAG's avatar
JohnAG
Frequent Visitor
2 years ago
Solved

DataFlow Gen2 process new file

Hey guys, 

I have couple questions:
1. How can I invoke my DataFlow to process the new file that landed in Lakehouse? See image at the bottom of the thread. 

2. How can I tell the DataFlow to process the new file with the new name when the name is hard coded in the M Language? see code below, line 8

 

//Comp worksheet
let
  Source = Lakehouse.Contents(null){[workspaceId = "xxxxxx-xxxx-xxxx-xxx-d59bxxxbddfc"]}[Data]{[lakehouseId = "xxxxxx-xxxx-xxxx-xxx-16f3xxxca0e5"]}[Data],
  #"Navigation 1" = Source{[Id = "Files", ItemKind = "Folder"]}[Data],
  #"Navigation 2" = #"Navigation 1"{[Name = "STR"]}[Content],
  #"Navigation 3" = #"Navigation 2"{[Name = "Excel"]}[Content],
  Navigation = #"Navigation 3"{[Name = "Monthly"]}[Content],
  #"Navigation 4" = Navigation{[Name = "Monthly STAR_9580-20230800-USD-E.xlsx"]}[Content],
  #"Imported Excel workbook" = Excel.Workbook(#"Navigation 4", null, true),
  #"Navigation 5" = #"Imported Excel workbook"{[Item = "Comp", Kind = "Sheet"]}[Data],
  #"Removed columns" = Table.RemoveColumns(#"Navigation 5", {"Column1"}),
  #"Trimmed text" = Table.TransformColumns(#"Removed columns", {{"Column2", each Text.Trim(Text.From(_)), type nullable text}}),
  #"Cleaned text" = Table.TransformColumns(#"Trimmed text", {{"Column2", each Text.Clean(_), type nullable text}}),
  #"Added custom" = Table.TransformColumnTypes(Table.AddColumn(#"Cleaned text", "STRID", each if Text.StartsWith([Column2],"Property ID") and [Column2] <> null then [Column2] else ""), {{"STRID", type text}})
in
  #"Added custom"

 

 

 

data engineering
Data Integration
data science
data warehouse
  • JohnAG's avatar
    JohnAG
    2 years ago

    I had to move it to Power Query inside PBIX file instead of hopping around Fabric services as they still not fully developped to handle a fully automated incremental ETL.   

4 Replies

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Icon for Microsoft Employee rankMicrosoft Employee
    2 years ago

    Hi JohnAG 

     

    Thanks for using Microsoft Fabric Community.
    At this time, we are reaching out to the internal team to get some help on this.
    We will update you once we hear back from them.

    Appreciate your patience.


    Thanks

    • v-cboorla-msft's avatar
      v-cboorla-msft
      Icon for Microsoft Employee rankMicrosoft Employee
      2 years ago

      Hi JohnAG 

       

      1. This could be possible by event driven refresh, but there is only scheduled refresh available in Fabric right now. You can write your own solution using REST APIs, etc. to poll for changes to the file system.
      2. You would want to connect to the folder location of the storage and filter out the content either by Name / date or some other metadata factor and choose to only append those files to your table.

      I hope this helps. Please do let us know if you have any further questions.

      • v-cboorla-msft's avatar
        v-cboorla-msft
        Icon for Microsoft Employee rankMicrosoft Employee
        2 years ago

        Hi JohnAG 

         

        We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
        In case if you have any resolution please do share that same with the community as it can be helpful to others.
        Otherwise, will respond back with the more details and we will try to help.


        Thanks