Forum Discussion
Flattening multiple related rows in Power Query
- 9 years ago
Thanks for the replies! I actually managed to solve this on my own in the end :)
ImkeF - your solution is interesting, I'm guessing FillUp will find the bottom-most non-null value and fill any rows above with it?
My solution - wrote a function that will find and return the first non-null value in a list (or a default if all null), and use that as the aggregator. I then build a list from the original list of column names that will run the operation on each named column:
//FirstNotNull let Source = (sourceList as list) => let firstNotNull = List.First(List.RemoveNulls(sourceList), "Not Applicable") in firstNotNull in Source //DynamicTableGroupColumns let Source = (sourceTable as table, columns as list, aggregateFunction as function) => let result = List.Transform(columns, each // build lists with {columnName, aggregateFunction} let //save current _ (column name) to use in next each statement columnName = _, columnToFunctionList = {columnName, each //_ will be the grouping table as it's called by Table.Group aggregateFunction(Table.Column(_, columnName))} in columnToFunctionList) in result in Source //DynamicTableGroup let Source = (sourceTable as table, groupBy as list, columns as list, aggregateFunction as function) => let result = Table.Group(sourceTable , groupBy, DynamicTableGroupColumns(sourceTable, columns, aggregateFunction)) in result in SourceAny comments on one method being better than the other? Will your method of FillUp into a single column and then expanding the relevant fields be more performant that preparing a list of lists to feed to Table.Group?
EDIT: ImkeF just timed the 2 queries, and filling up into one column and then expanding seemed to take 2min20s, while my approach took 58s! Yesterday I had also timed doing an unpivot/pivot over all columns, and that was taking about 1min45s. I'm not sure how the unpivot/pivot scales with more columns and rows, but I'd assume our 2 methods would scale similarly.
Feel free to use the set of functions I put up in case you find use for them to speed up any queries! Or let me know if don't see similar results :)
Yea, I thought it was curious too :) But again, I'm not sure if that approach would scale better than pivoting, it might be a case of one approach being better in particular scenarios.
I do intend to play with Buffering at some point to see where it can help improve performance. Do you know of any general rules where using Buffer will help?
No, most often it is trial & error.
Only for List.Generate I will always use it for the input-tables or -lists to the function
- JasonG9 years agoRegular Visitor
Maybe it's too early in the morning that I tackled this, but I can't yet visualize how to flatten this data.
What I start with is as follows:
Column10 Column3 Column14 18/05/2017 As75 489.044189453125 18/05/2017 As75 529.010314941406 18/05/2017 As75 40914.140625 18/05/2017 As75 3145.38793945313 18/05/2017 As75 43844.9296875 18/05/2017 Ca44 2365.14038085938 18/05/2017 Ca44 20024.193359375 18/05/2017 Ca44 6499.56396484375 18/05/2017 Ca44 49550.2265625 18/05/2017 Ca44 125394.3984375 18/05/2017 Cu65 818.863037109375 18/05/2017 Cu65 120588.8828125 18/05/2017 Cu65 5401.640625 18/05/2017 Cu65 1566.38427734375 18/05/2017 Cu65 119667.9921875
What I would like to do is turn it into
Date As75 Ca44 Cu65 18/05/2017 489.044189453125 2365.14038085938 818.863037109375 18/05/2017 529.010314941406 20024.193359375 120588.8828125 18/05/2017 40914.140625 6499.56396484375 5401.640625 18/05/2017 3145.38793945313 49550.2265625 1566.38427734375 18/05/2017 43844.9296875 125394.3984375 119667.9921875 Any ideas?
- JasonG9 years agoRegular Visitor
Maybe it's too early in the morning that I tackled this, but I can't yet visualize how to flatten this data.
What I start with is as follows:
Column10 Column3 Column14 18/05/2017 As75 489.044189453125 18/05/2017 As75 529.010314941406 18/05/2017 As75 40914.140625 18/05/2017 As75 3145.38793945313 18/05/2017 As75 43844.9296875 18/05/2017 Ca44 2365.14038085938 18/05/2017 Ca44 20024.193359375 18/05/2017 Ca44 6499.56396484375 18/05/2017 Ca44 49550.2265625 18/05/2017 Ca44 125394.3984375 18/05/2017 Cu65 818.863037109375 18/05/2017 Cu65 120588.8828125 18/05/2017 Cu65 5401.640625 18/05/2017 Cu65 1566.38427734375 18/05/2017 Cu65 119667.9921875
What I would like to do is turn it into
Date As75 Ca44 Cu65 18/05/2017 489.044189453125 2365.14038085938 818.863037109375 18/05/2017 529.010314941406 20024.193359375 120588.8828125 18/05/2017 40914.140625 6499.56396484375 5401.640625 18/05/2017 3145.38793945313 49550.2265625 1566.38427734375 18/05/2017 43844.9296875 125394.3984375 119667.9921875 Any ideas?
- ImkeF9 years agoCommunity Champion
Thats actually not trivial if you want to make it dynamic. Check out this code:
let Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("fZE7DsNQCATv4hopfBYelFGOYbnIGaLcP9hJOj9XNKNhYdd1kbyx35RlLLTcX8N7IIsYkCy4ifqy0Tno2qCwCQoCjikILgHtyIWtNU6Wo+zYa3ObJUClFTlOdI8n0EMtfF9pyellOQeZtcOVWWMXwkAVeVgFElcgyp1JNfz03B/Vj+1DyWome8d+bEpShrEN4Um8LyjKns2m5mlnP8zBQjFp4q/yiG4COoZdhhOpiEFVKkcV2wc=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type text) meta [Serialized.Text = true]) in type table [Column10 = _t, Column3 = _t, Column14 = _t]), #"Changed Type" = Table.TransformColumnTypes(Source,{{"Column10", type date}, {"Column3", type text}, {"Column14", type number}}), #"Added Index" = Table.AddIndexColumn(#"Changed Type", "Index", 0, 1), #"Added to Column" = Table.TransformColumns(#"Added Index", {{"Index", each Number.Mod(_, List.Count(#"Added Index"[Column3])/List.Count(List.Distinct(#"Added Index"[Column3]))), type number}}), #"Pivoted Column" = Table.Pivot(#"Added to Column", List.Distinct(#"Added to Column"[Column3]), "Column3", "Column14"), #"Removed Columns" = Table.RemoveColumns(#"Pivoted Column",{"Index"}) in #"Removed Columns"If you don't know how to apply this code, check out this video: https://www.youtube.com/watch?v=S9xlq5KUZ60
- ImkeF9 years agoCommunity Champion
There is a more elegant version for it, which should also perform faster:
let Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("fZExDsMwCEXv4hlFgAHDWPUYUYaeoer96ypul0Anhv/gf2DfG/mGujHSaNBuz6GziAegtANyWXnKVMqCQQJU93cShR71gO4iEP0K3B8is3A3TQ2+OiLPBInDAkwiQK3UJVQRuE5ArD0EBBPiZZ8lnBw8czhlYlSfhJeEChJYtuSaoGbQ636iMBsQ6wrxJwN7yawUaKwJs45xJhEe48qsn/7SMLfjeAM=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type text) meta [Serialized.Text = true]) in type table [Column10 = _t, Column3 = _t, Column14 = _t]), Group = Table.Group(Source, {"Column10", "Column3"}, {{"All", each _[Column14], type table}}), ToColumns = Table.Group(Group, {"Column10"}, {{"All", each Table.FromColumns(_[All], _[Column3]), type table}}), Expand = Table.ExpandTableColumn(ToColumns, "All", List.Distinct(Group[Column3])) in Expand