DataFlow Gen2 process new file

[JohnAG]

Hey guys, 

I have couple questions:
1. How can I invoke my DataFlow to process the new file that landed in Lakehouse? See image at the bottom of the thread. 

2. How can I tell the DataFlow to process the new file with the new name when the name is hard coded in the M Language? see code below, line 8

//Comp worksheet
let
  Source = Lakehouse.Contents(null){[workspaceId = "xxxxxx-xxxx-xxxx-xxx-d59bxxxbddfc"]}[Data]{[lakehouseId = "xxxxxx-xxxx-xxxx-xxx-16f3xxxca0e5"]}[Data],
  #"Navigation 1" = Source{[Id = "Files", ItemKind = "Folder"]}[Data],
  #"Navigation 2" = #"Navigation 1"{[Name = "STR"]}[Content],
  #"Navigation 3" = #"Navigation 2"{[Name = "Excel"]}[Content],
  Navigation = #"Navigation 3"{[Name = "Monthly"]}[Content],
  #"Navigation 4" = Navigation{[Name = "Monthly STAR_9580-20230800-USD-E.xlsx"]}[Content],
  #"Imported Excel workbook" = Excel.Workbook(#"Navigation 4", null, true),
  #"Navigation 5" = #"Imported Excel workbook"{[Item = "Comp", Kind = "Sheet"]}[Data],
  #"Removed columns" = Table.RemoveColumns(#"Navigation 5", {"Column1"}),
  #"Trimmed text" = Table.TransformColumns(#"Removed columns", {{"Column2", each Text.Trim(Text.From(_)), type nullable text}}),
  #"Cleaned text" = Table.TransformColumns(#"Trimmed text", {{"Column2", each Text.Clean(_), type nullable text}}),
  #"Added custom" = Table.TransformColumnTypes(Table.AddColumn(#"Cleaned text", "STRID", each if Text.StartsWith([Column2],"Property ID") and [Column2] <> null then [Column2] else ""), {{"STRID", type text}})
in
  #"Added custom"

[v-cboorla-msft]

Hi @JohnAG 

Thanks for using Microsoft Fabric Community.
At this time, we are reaching out to the internal team to get some help on this.
We will update you once we hear back from them.

Appreciate your patience.

Thanks

[v-cboorla-msft]

Hi @JohnAG 

1. This could be possible by event driven refresh, but there is only scheduled refresh available in Fabric right now. You can write your own solution using REST APIs, etc. to poll for changes to the file system.
2. You would want to connect to the folder location of the storage and filter out the content either by Name / date or some other metadata factor and choose to only append those files to your table.

I hope this helps. Please do let us know if you have any further questions.

[v-cboorla-msft]

Hi @JohnAG 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks

[JohnAG · Accepted solution]

I had to move it to Power Query inside PBIX file instead of hopping around Fabric services as they still not fully developped to handle a fully automated incremental ETL.