Forum Discussion
Flattening multiple related rows in Power Query
- 9 years ago
Thanks for the replies! I actually managed to solve this on my own in the end :)
ImkeF - your solution is interesting, I'm guessing FillUp will find the bottom-most non-null value and fill any rows above with it?
My solution - wrote a function that will find and return the first non-null value in a list (or a default if all null), and use that as the aggregator. I then build a list from the original list of column names that will run the operation on each named column:
//FirstNotNull let Source = (sourceList as list) => let firstNotNull = List.First(List.RemoveNulls(sourceList), "Not Applicable") in firstNotNull in Source //DynamicTableGroupColumns let Source = (sourceTable as table, columns as list, aggregateFunction as function) => let result = List.Transform(columns, each // build lists with {columnName, aggregateFunction} let //save current _ (column name) to use in next each statement columnName = _, columnToFunctionList = {columnName, each //_ will be the grouping table as it's called by Table.Group aggregateFunction(Table.Column(_, columnName))} in columnToFunctionList) in result in Source //DynamicTableGroup let Source = (sourceTable as table, groupBy as list, columns as list, aggregateFunction as function) => let result = Table.Group(sourceTable , groupBy, DynamicTableGroupColumns(sourceTable, columns, aggregateFunction)) in result in SourceAny comments on one method being better than the other? Will your method of FillUp into a single column and then expanding the relevant fields be more performant that preparing a list of lists to feed to Table.Group?
EDIT: ImkeF just timed the 2 queries, and filling up into one column and then expanding seemed to take 2min20s, while my approach took 58s! Yesterday I had also timed doing an unpivot/pivot over all columns, and that was taking about 1min45s. I'm not sure how the unpivot/pivot scales with more columns and rows, but I'd assume our 2 methods would scale similarly.
Feel free to use the set of functions I put up in case you find use for them to speed up any queries! Or let me know if don't see similar results :)
Yes, it does run faster. I'm using Power Query from Excel 365. The Excel workbook is a template that I will eventually distribute to my estimators. My estimators will use the template to prepare bid proposals for commercial construction projects. The original template was made over 20 years ago. It is not dynamic, and has been abused over the years by estimators trying to add functionality to it. So, I've tasked myself with upgrading to the 21st century. (I still believe I shouldn't be using Excel at all)
Speed is very much an issue. On bid day, we will run multiple scenerios, adjusting labor costs, crew size, playing with profit and overhead, etc. Pivoting, merging, and expanding does slow down refreshes. The fact table is 50 - 60 columns, and averages +/- 500 rows and 3 dimension tables. Not a big fact table, but I need refreshes to be as close to a 1/10th second as possible, or estimators will refuse to use it.
The nice thing about your solution is the larger the fact table, the more benifit I get from grouping first. On the tests I've ran so far, whether I have 700 rows or 50, I'm able to group down to around 25-30 rows before I have to start shaping.
The main structure of my new template is complete. Now, I'm performance tuning. Unfortunately, I'm new to PQ, so this has been a very slow process.
Thanks for confirming jfclark27!
Performance tuning can be a pain. I've collected some tipps for it here: https://www.thebiccountant.com/speedperformance-aspects/