Forum Discussion
Advocate IIDownload and unzip multiple files from URL then query csv file/s within
If you open that URL as a web page in Power Query then drill down through the tables, you can get a list of ZIP files:
let
Source = Web.Page(Web.Contents("http://www.nemweb.com.au/REPORTS/CURRENT/Daily_Reports/")),
Data0 = Source{0}[Data],
Children = Data0{0}[Children],
Children1 = Children{1}[Children],
Children2 = Children1{2}[Children],
#"Removed Other Columns" = Table.SelectColumns(Children2,{"Children"}),
#"Expanded Children" = Table.ExpandTableColumn(#"Removed Other Columns", "Children", {"Text"}, {"Children.Text"}),
#"Filtered Rows" = Table.SelectRows(#"Expanded Children", each ([Children.Text] <> null and [Children.Text] <> "[To Parent Directory]"))
in
#"Filtered Rows"You could save that as a CSV each day with the R script you mention, then join with the result above the following day to find any new files, and pass those as parameters to a function to Unzip-process-save as CSV etc.
Is that the sort of thing you're after?
Advocate IIYes Anonymous - your suggested process would seem to fit, however I am concerned about the maintainability and portability given the number of steps and reliance on coming in and out of R scripts.
One of the original intents of the question was to draw an indication from the Power BI / Power Query development team if this something that is on the radar.
As public data sets are becoming increasing large and available on line as zip files, the ability to connect to a web source, download, unzip and then process without specialist coding in Power BI would be a distinct product advantage. There appear to be many pieces of the puzzle already, so joining the dots would be a fantastic improvement !
Maybe this could go to the ideas forum / group ??
I too would be intrigued to see what feedback you get from the Dev team on this. There are at least 3 entries on this already in the ideas forum - see https://ideas.powerbi.com/search?filter=ideas&query=zip%20file