Forum Discussion
RobGer
2 years agoFrequent Visitor
Transform CSV Files on a monthly basis
I have the following issue regarding transforming csv files that are sent to me monthly. The files are named after the YearMonth and the analyzed Subject. Example: 202404 – Web traffic. The fil...
- 2 years ago
This query reads all files in specified folder and grabs all tables. Replace full_folder_path with your own and give it a try.
let file_processing = (b as binary) as table => [s = Lines.FromBinary(b, QuoteStyle.None, false, 65001), rm = Table.FromColumns({List.RemoveItems(s, {"", "message", "ok"})}, {"table_name"}), group = Table.Group( rm, "table_name", {"data", (x) => Table.PromoteHeaders( Table.FromList(List.Skip(x[table_name])) )}, GroupKind.Local, (s, c) => Number.From(not Text.Contains(c, ",")) )][group], files = Folder.Files("full_folder_path")[[Name], [Content]], tra = Table.TransformColumns(files, {"Content", file_processing}), xpand = Table.ExpandTableColumn(tra, "Content", {"table_name", "data"}) in xpandThen you may create a reference to this query, filter by table name and expand data column to get what you want.
AlienSx
2 years agoSuper User
This query reads all files in specified folder and grabs all tables. Replace full_folder_path with your own and give it a try.
let
file_processing = (b as binary) as table =>
[s = Lines.FromBinary(b, QuoteStyle.None, false, 65001),
rm = Table.FromColumns({List.RemoveItems(s, {"", "message", "ok"})}, {"table_name"}),
group = Table.Group(
rm, "table_name",
{"data", (x) => Table.PromoteHeaders(
Table.FromList(List.Skip(x[table_name]))
)},
GroupKind.Local,
(s, c) => Number.From(not Text.Contains(c, ","))
)][group],
files = Folder.Files("full_folder_path")[[Name], [Content]],
tra = Table.TransformColumns(files, {"Content", file_processing}),
xpand = Table.ExpandTableColumn(tra, "Content", {"table_name", "data"})
in
xpand
Then you may create a reference to this query, filter by table name and expand data column to get what you want.
RobGer
2 years agoFrequent Visitor
Thanks a lot! This has been eating at me for the past weeks, the solution worked perfectly!