Forum Discussion

jPinhao's avatar
jPinhao
Advocate II
9 years ago
Solved

Flattening multiple related rows in Power Query

We are loading some JSON data, which holds some arrays with varying property-value pairs. Due to the JSON structure, once you start expanding the fields you end up with something along these lines: ...
  • jPinhao's avatar
    jPinhao
    9 years ago

    Thanks for the replies! I actually managed to solve this on my own in the end :)

     

    ImkeF - your solution is interesting, I'm guessing FillUp will find the bottom-most non-null value and fill any rows above with it? 

     

    My solution - wrote a function that will find and return the first non-null value in a list (or a default if all null), and use that as the aggregator. I then build a list from the original list of column names that will run the operation on each named column:

     

    //FirstNotNull
    let
        Source = (sourceList as list) =>
            let
                firstNotNull = List.First(List.RemoveNulls(sourceList), "Not Applicable")
            in
                firstNotNull
    in
        Source
    
    //DynamicTableGroupColumns
    let
        Source = (sourceTable as table, columns as list, aggregateFunction as function) =>
            let
                result = List.Transform(columns, each 
                                                // build lists with {columnName, aggregateFunction}
                                                let
                                                    //save current _ (column name) to use in next each statement
                                                    columnName = _,
                                                    columnToFunctionList =  {columnName, each 
                                                                                //_ will be the grouping table as it's called by Table.Group
                                                                                aggregateFunction(Table.Column(_, columnName))}
                                                in
                                                    columnToFunctionList)
            in
                result
    in
        Source
    
    //DynamicTableGroup
    let
        Source = (sourceTable as table, groupBy as list, columns as list, aggregateFunction as function) =>
            let 
                result = Table.Group(sourceTable , groupBy, DynamicTableGroupColumns(sourceTable, columns, aggregateFunction))
            in
                result
    in
        Source

     

    Any comments on one method being better than the other? Will your method of FillUp into a single column and then expanding the relevant fields be more performant that preparing a list of lists to feed to Table.Group?

     

    EDIT: ImkeF just timed the 2 queries, and filling up into one column and then expanding seemed to take 2min20s, while my approach took 58s! Yesterday I had also timed doing an unpivot/pivot over all columns, and that was taking about 1min45s. I'm not sure how the unpivot/pivot scales with more columns and rows, but I'd assume our 2 methods would scale similarly. 

     

    Feel free to use the set of functions I put up in case you find use for them to speed up any queries! Or let me know if don't see similar results :)