Forum Discussion
Flattening multiple related rows in Power Query
- 9 years ago
Thanks for the replies! I actually managed to solve this on my own in the end :)
ImkeF - your solution is interesting, I'm guessing FillUp will find the bottom-most non-null value and fill any rows above with it?
My solution - wrote a function that will find and return the first non-null value in a list (or a default if all null), and use that as the aggregator. I then build a list from the original list of column names that will run the operation on each named column:
//FirstNotNull let Source = (sourceList as list) => let firstNotNull = List.First(List.RemoveNulls(sourceList), "Not Applicable") in firstNotNull in Source //DynamicTableGroupColumns let Source = (sourceTable as table, columns as list, aggregateFunction as function) => let result = List.Transform(columns, each // build lists with {columnName, aggregateFunction} let //save current _ (column name) to use in next each statement columnName = _, columnToFunctionList = {columnName, each //_ will be the grouping table as it's called by Table.Group aggregateFunction(Table.Column(_, columnName))} in columnToFunctionList) in result in Source //DynamicTableGroup let Source = (sourceTable as table, groupBy as list, columns as list, aggregateFunction as function) => let result = Table.Group(sourceTable , groupBy, DynamicTableGroupColumns(sourceTable, columns, aggregateFunction)) in result in SourceAny comments on one method being better than the other? Will your method of FillUp into a single column and then expanding the relevant fields be more performant that preparing a list of lists to feed to Table.Group?
EDIT: ImkeF just timed the 2 queries, and filling up into one column and then expanding seemed to take 2min20s, while my approach took 58s! Yesterday I had also timed doing an unpivot/pivot over all columns, and that was taking about 1min45s. I'm not sure how the unpivot/pivot scales with more columns and rows, but I'd assume our 2 methods would scale similarly.
Feel free to use the set of functions I put up in case you find use for them to speed up any queries! Or let me know if don't see similar results :)
Thanks for the replies! I actually managed to solve this on my own in the end :)
ImkeF - your solution is interesting, I'm guessing FillUp will find the bottom-most non-null value and fill any rows above with it?
My solution - wrote a function that will find and return the first non-null value in a list (or a default if all null), and use that as the aggregator. I then build a list from the original list of column names that will run the operation on each named column:
//FirstNotNull
let
Source = (sourceList as list) =>
let
firstNotNull = List.First(List.RemoveNulls(sourceList), "Not Applicable")
in
firstNotNull
in
Source
//DynamicTableGroupColumns
let
Source = (sourceTable as table, columns as list, aggregateFunction as function) =>
let
result = List.Transform(columns, each
// build lists with {columnName, aggregateFunction}
let
//save current _ (column name) to use in next each statement
columnName = _,
columnToFunctionList = {columnName, each
//_ will be the grouping table as it's called by Table.Group
aggregateFunction(Table.Column(_, columnName))}
in
columnToFunctionList)
in
result
in
Source
//DynamicTableGroup
let
Source = (sourceTable as table, groupBy as list, columns as list, aggregateFunction as function) =>
let
result = Table.Group(sourceTable , groupBy, DynamicTableGroupColumns(sourceTable, columns, aggregateFunction))
in
result
in
Source
Any comments on one method being better than the other? Will your method of FillUp into a single column and then expanding the relevant fields be more performant that preparing a list of lists to feed to Table.Group?
EDIT: ImkeF just timed the 2 queries, and filling up into one column and then expanding seemed to take 2min20s, while my approach took 58s! Yesterday I had also timed doing an unpivot/pivot over all columns, and that was taking about 1min45s. I'm not sure how the unpivot/pivot scales with more columns and rows, but I'd assume our 2 methods would scale similarly.
Feel free to use the set of functions I put up in case you find use for them to speed up any queries! Or let me know if don't see similar results :)
Hi jPinhao, that's pretty cool!
Wasn't aware that FillUp is even slower than pivoting :-)
You can further play around with List or Table.Buffer to see if this speeds it up even more.
- jPinhao9 years agoAdvocate II
Yea, I thought it was curious too :) But again, I'm not sure if that approach would scale better than pivoting, it might be a case of one approach being better in particular scenarios.
I do intend to play with Buffering at some point to see where it can help improve performance. Do you know of any general rules where using Buffer will help?
- ImkeF9 years agoCommunity Champion
No, most often it is trial & error.
Only for List.Generate I will always use it for the input-tables or -lists to the function
- JasonG9 years agoRegular Visitor
Maybe it's too early in the morning that I tackled this, but I can't yet visualize how to flatten this data.
What I start with is as follows:
Column10 Column3 Column14 18/05/2017 As75 489.044189453125 18/05/2017 As75 529.010314941406 18/05/2017 As75 40914.140625 18/05/2017 As75 3145.38793945313 18/05/2017 As75 43844.9296875 18/05/2017 Ca44 2365.14038085938 18/05/2017 Ca44 20024.193359375 18/05/2017 Ca44 6499.56396484375 18/05/2017 Ca44 49550.2265625 18/05/2017 Ca44 125394.3984375 18/05/2017 Cu65 818.863037109375 18/05/2017 Cu65 120588.8828125 18/05/2017 Cu65 5401.640625 18/05/2017 Cu65 1566.38427734375 18/05/2017 Cu65 119667.9921875
What I would like to do is turn it into
Date As75 Ca44 Cu65 18/05/2017 489.044189453125 2365.14038085938 818.863037109375 18/05/2017 529.010314941406 20024.193359375 120588.8828125 18/05/2017 40914.140625 6499.56396484375 5401.640625 18/05/2017 3145.38793945313 49550.2265625 1566.38427734375 18/05/2017 43844.9296875 125394.3984375 119667.9921875 Any ideas?