Forum Discussion
Flattening multiple related rows in Power Query
- 9 years ago
Thanks for the replies! I actually managed to solve this on my own in the end :)
ImkeF - your solution is interesting, I'm guessing FillUp will find the bottom-most non-null value and fill any rows above with it?
My solution - wrote a function that will find and return the first non-null value in a list (or a default if all null), and use that as the aggregator. I then build a list from the original list of column names that will run the operation on each named column:
//FirstNotNull let Source = (sourceList as list) => let firstNotNull = List.First(List.RemoveNulls(sourceList), "Not Applicable") in firstNotNull in Source //DynamicTableGroupColumns let Source = (sourceTable as table, columns as list, aggregateFunction as function) => let result = List.Transform(columns, each // build lists with {columnName, aggregateFunction} let //save current _ (column name) to use in next each statement columnName = _, columnToFunctionList = {columnName, each //_ will be the grouping table as it's called by Table.Group aggregateFunction(Table.Column(_, columnName))} in columnToFunctionList) in result in Source //DynamicTableGroup let Source = (sourceTable as table, groupBy as list, columns as list, aggregateFunction as function) => let result = Table.Group(sourceTable , groupBy, DynamicTableGroupColumns(sourceTable, columns, aggregateFunction)) in result in SourceAny comments on one method being better than the other? Will your method of FillUp into a single column and then expanding the relevant fields be more performant that preparing a list of lists to feed to Table.Group?
EDIT: ImkeF just timed the 2 queries, and filling up into one column and then expanding seemed to take 2min20s, while my approach took 58s! Yesterday I had also timed doing an unpivot/pivot over all columns, and that was taking about 1min45s. I'm not sure how the unpivot/pivot scales with more columns and rows, but I'd assume our 2 methods would scale similarly.
Feel free to use the set of functions I put up in case you find use for them to speed up any queries! Or let me know if don't see similar results :)
Thats actually not trivial if you want to make it dynamic. Check out this code:
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("fZE7DsNQCATv4hopfBYelFGOYbnIGaLcP9hJOj9XNKNhYdd1kbyx35RlLLTcX8N7IIsYkCy4ifqy0Tno2qCwCQoCjikILgHtyIWtNU6Wo+zYa3ObJUClFTlOdI8n0EMtfF9pyellOQeZtcOVWWMXwkAVeVgFElcgyp1JNfz03B/Vj+1DyWome8d+bEpShrEN4Um8LyjKns2m5mlnP8zBQjFp4q/yiG4COoZdhhOpiEFVKkcV2wc=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type text) meta [Serialized.Text = true]) in type table [Column10 = _t, Column3 = _t, Column14 = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Column10", type date}, {"Column3", type text}, {"Column14", type number}}),
#"Added Index" = Table.AddIndexColumn(#"Changed Type", "Index", 0, 1),
#"Added to Column" = Table.TransformColumns(#"Added Index", {{"Index", each Number.Mod(_, List.Count(#"Added Index"[Column3])/List.Count(List.Distinct(#"Added Index"[Column3]))), type number}}),
#"Pivoted Column" = Table.Pivot(#"Added to Column", List.Distinct(#"Added to Column"[Column3]), "Column3", "Column14"),
#"Removed Columns" = Table.RemoveColumns(#"Pivoted Column",{"Index"})
in
#"Removed Columns"
If you don't know how to apply this code, check out this video: https://www.youtube.com/watch?v=S9xlq5KUZ60
There is a more elegant version for it, which should also perform faster:
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("fZExDsMwCEXv4hlFgAHDWPUYUYaeoer96ypul0Anhv/gf2DfG/mGujHSaNBuz6GziAegtANyWXnKVMqCQQJU93cShR71gO4iEP0K3B8is3A3TQ2+OiLPBInDAkwiQK3UJVQRuE5ArD0EBBPiZZ8lnBw8czhlYlSfhJeEChJYtuSaoGbQ636iMBsQ6wrxJwN7yawUaKwJs45xJhEe48qsn/7SMLfjeAM=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type text) meta [Serialized.Text = true]) in type table [Column10 = _t, Column3 = _t, Column14 = _t]),
Group = Table.Group(Source, {"Column10", "Column3"}, {{"All", each _[Column14], type table}}),
ToColumns = Table.Group(Group, {"Column10"}, {{"All", each Table.FromColumns(_[All], _[Column3]), type table}}),
Expand = Table.ExpandTableColumn(ToColumns, "All", List.Distinct(Group[Column3]))
in
Expand