Forum Discussion

RobGer's avatar
RobGer
Frequent Visitor
2 years ago
Solved

Transform CSV Files on a monthly basis

I have the following issue regarding transforming csv files that are sent to me monthly. The files are named after the YearMonth and the analyzed Subject. Example: 202404 – Web traffic.   The fil...
  • AlienSx's avatar
    2 years ago

    This query reads all files in specified folder and grabs all tables. Replace full_folder_path with your own and give it a try.  

    let
        file_processing = (b as binary) as table =>
            [s = Lines.FromBinary(b, QuoteStyle.None, false, 65001),
            rm = Table.FromColumns({List.RemoveItems(s, {"", "message", "ok"})}, {"table_name"}),
            group = Table.Group(
                rm, "table_name", 
                {"data", (x) => Table.PromoteHeaders(
                    Table.FromList(List.Skip(x[table_name]))
                )}, 
                GroupKind.Local, 
                (s, c) => Number.From(not Text.Contains(c, ","))
            )][group],
        files = Folder.Files("full_folder_path")[[Name], [Content]],
        tra = Table.TransformColumns(files, {"Content", file_processing}),
        xpand = Table.ExpandTableColumn(tra, "Content", {"table_name", "data"})
    in
        xpand

     

     

     

     

     

     

     

     

    Then you may create a reference to this query, filter by table name and expand data column to get what you want.