Forum Discussion

jPinhao's avatar
jPinhao
Icon for Advocate II rankAdvocate II
9 years ago
Solved

Flattening multiple related rows in Power Query

We are loading some JSON data, which holds some arrays with varying property-value pairs. Due to the JSON structure, once you start expanding the fields you end up with something along these lines:

 

id | key1 | key2 | key3 | key4
0  | a    | null | null | null 
0  | null | b    | null | null
0  | null | null | c    | null
1  | a    | null | null | null
1  | null | b    | null | null
1  | null | null | c    | null
1  | null | null | null | d

We'd like to flatten these rows by ID to end up with something like:

id | key1 | key2 | key3 | key4
0  | a    | b    | c    | null 
1  | a    | b    | c    | d

The simplest way I found to solve this is by unpivotting the key columns, and then pivotting it back. However in some cases, we actually have quite a few more columns, and thousands of rows, in a quite sparse table. From previous trials I found that unpivot/pivot can be a big performance hit.

 

Is there a better way to solve this issue? Either in Power Query or with an R script?

 

 

EDIT: I beleive I found a way to do this with Table.Group:

Table.Group(sourceTable, {"id"}, {{"key1", each List.RemoveNulls([key1]){0},  type none}, {"key2", ...},  ...})

I'd like to do this across all columns (except ID), or from a List of column names - how would I go about making Table.Group work from a list of values dynamically?

  • Thanks for the replies! I actually managed to solve this on my own in the end :)

     

    ImkeF - your solution is interesting, I'm guessing FillUp will find the bottom-most non-null value and fill any rows above with it? 

     

    My solution - wrote a function that will find and return the first non-null value in a list (or a default if all null), and use that as the aggregator. I then build a list from the original list of column names that will run the operation on each named column:

     

    //FirstNotNull
    let
        Source = (sourceList as list) =>
            let
                firstNotNull = List.First(List.RemoveNulls(sourceList), "Not Applicable")
            in
                firstNotNull
    in
        Source
    
    //DynamicTableGroupColumns
    let
        Source = (sourceTable as table, columns as list, aggregateFunction as function) =>
            let
                result = List.Transform(columns, each 
                                                // build lists with {columnName, aggregateFunction}
                                                let
                                                    //save current _ (column name) to use in next each statement
                                                    columnName = _,
                                                    columnToFunctionList =  {columnName, each 
                                                                                //_ will be the grouping table as it's called by Table.Group
                                                                                aggregateFunction(Table.Column(_, columnName))}
                                                in
                                                    columnToFunctionList)
            in
                result
    in
        Source
    
    //DynamicTableGroup
    let
        Source = (sourceTable as table, groupBy as list, columns as list, aggregateFunction as function) =>
            let 
                result = Table.Group(sourceTable , groupBy, DynamicTableGroupColumns(sourceTable, columns, aggregateFunction))
            in
                result
    in
        Source

     

    Any comments on one method being better than the other? Will your method of FillUp into a single column and then expanding the relevant fields be more performant that preparing a list of lists to feed to Table.Group?

     

    EDIT: ImkeF just timed the 2 queries, and filling up into one column and then expanding seemed to take 2min20s, while my approach took 58s! Yesterday I had also timed doing an unpivot/pivot over all columns, and that was taking about 1min45s. I'm not sure how the unpivot/pivot scales with more columns and rows, but I'd assume our 2 methods would scale similarly. 

     

    Feel free to use the set of functions I put up in case you find use for them to speed up any queries! Or let me know if don't see similar results :)

16 Replies

  • ImkeF's avatar
    ImkeF
    Icon for Community Champion rankCommunity Champion

    A possible solution with dynamic column headers is this:

     

    let
        Source = YourTable,
        #"Grouped Rows" = Table.Group(Source, {"id"}, {{"FillUp", each Table.FirstN(Table.FillUp(_,Table.ColumnNames(_)), 1), type table}}),
        #"Expanded FillUp" = Table.ExpandTableColumn(#"Grouped Rows", "FillUp", List.Skip(Table.ColumnNames(Source),1), List.Skip(Table.ColumnNames(Source),1))
    in
        #"Expanded FillUp"

     

    It pushes all items to the first row and then just keeps that.

    • jPinhao's avatar
      jPinhao
      Icon for Advocate II rankAdvocate II

      Thanks for the replies! I actually managed to solve this on my own in the end :)

       

      ImkeF - your solution is interesting, I'm guessing FillUp will find the bottom-most non-null value and fill any rows above with it? 

       

      My solution - wrote a function that will find and return the first non-null value in a list (or a default if all null), and use that as the aggregator. I then build a list from the original list of column names that will run the operation on each named column:

       

      //FirstNotNull
      let
          Source = (sourceList as list) =>
              let
                  firstNotNull = List.First(List.RemoveNulls(sourceList), "Not Applicable")
              in
                  firstNotNull
      in
          Source
      
      //DynamicTableGroupColumns
      let
          Source = (sourceTable as table, columns as list, aggregateFunction as function) =>
              let
                  result = List.Transform(columns, each 
                                                  // build lists with {columnName, aggregateFunction}
                                                  let
                                                      //save current _ (column name) to use in next each statement
                                                      columnName = _,
                                                      columnToFunctionList =  {columnName, each 
                                                                                  //_ will be the grouping table as it's called by Table.Group
                                                                                  aggregateFunction(Table.Column(_, columnName))}
                                                  in
                                                      columnToFunctionList)
              in
                  result
      in
          Source
      
      //DynamicTableGroup
      let
          Source = (sourceTable as table, groupBy as list, columns as list, aggregateFunction as function) =>
              let 
                  result = Table.Group(sourceTable , groupBy, DynamicTableGroupColumns(sourceTable, columns, aggregateFunction))
              in
                  result
      in
          Source

       

      Any comments on one method being better than the other? Will your method of FillUp into a single column and then expanding the relevant fields be more performant that preparing a list of lists to feed to Table.Group?

       

      EDIT: ImkeF just timed the 2 queries, and filling up into one column and then expanding seemed to take 2min20s, while my approach took 58s! Yesterday I had also timed doing an unpivot/pivot over all columns, and that was taking about 1min45s. I'm not sure how the unpivot/pivot scales with more columns and rows, but I'd assume our 2 methods would scale similarly. 

       

      Feel free to use the set of functions I put up in case you find use for them to speed up any queries! Or let me know if don't see similar results :)

      • ImkeF's avatar
        ImkeF
        Icon for Community Champion rankCommunity Champion

        Hi jPinhao, that's pretty cool!

        Wasn't aware that FillUp is even slower than pivoting :-)

         

        You can further play around with List or Table.Buffer to see if this speeds it up even more.

         

    • jfclark27's avatar
      jfclark27
      Regular Visitor

      I'm late to the party here, but was wondering if you could help me. I'm trying to use the functions OP posted above. Unfortunately, I cannot figure out how to call them in my query.

       

      I have a list containing my "groupBy" columns

      I have a list containing my "columns" I'd like summed

       

      I just cant figure out the "aggregateFunction" argument.

       

      Do you know how to actually use these functions in a query to List.Sum the dynamic columns? If I can impliment these functions, it would save me a lot of steps and an expensive unpivot.

      • ImkeF's avatar
        ImkeF
        Icon for Community Champion rankCommunity Champion

        Sounds to me that what you jfclark27 are asking for is a bit different. Does this code do the job?:

         

        let
            GroupColumns = {"Group"},
            SumColumn = {"Col1", "Col2"},
            AggregationFunctions = List.Transform(SumColumn, each {_, (x)=> List.Sum(Table.Column(x,_)), type number}),
            Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WclTSUTIEYmOlWB0IzwiITcA8JyDLHIhN4TwLIDZTio0FAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type text) meta [Serialized.Text = true]) in type table [Group = _t, Col1 = _t, Col2 = _t]),
            #"Changed Type" = Table.TransformColumnTypes(Source,{{"Group", type text}, {"Col1", Int64.Type}, {"Col2", Int64.Type}}),
            DynamicAggregation = Table.Group(#"Changed Type", GroupColumns, AggregationFunctions)
        in
            DynamicAggregation

        The tricky part is how to create the AggregationFunction: 

         

        List.Transform(SumColumn, each {_, (x)=> List.Sum(Table.Column(x,_)), type number}) 

         

        There you have to work with different environments: The _ represents each element from your list with columns to be aggregated ("SumColumn") and will actually be "used" in the step "AggregationFunction", while the "x" represents the table that will be passed into the function once it is called in step "DynamicAggregation".

         

        If you are interested to learn more about Power Query's environment-concept, I recommend this article-series: http://ssbi-blog.de/technical-topics-english/the-environment-concept-in-m-for-power-query-and-power-bi-desktop-part-1/

  • v-haibl-msft's avatar
    v-haibl-msft
    Icon for Microsoft Employee rankMicrosoft Employee

    jPinhao

     

    Please try with following Power Query in Advanced Editor.

     

    let
        Source = Excel.Workbook(File.Contents("C:\11032016\Flattening multiple related rows in Power Query.xlsx"), null, true),
        Table1_Table = Source{[Item="Table1",Kind="Table"]}[Data],
        #"Changed Type" = Table.TransformColumnTypes(Table1_Table,{{"id", Int64.Type}, {"key1", type text}, {"key2", type text}, {"key3", type text}, {"key4", type text}}),
        #"Combine" = Combiner.CombineTextByDelimiter(""),
        #"GroupRows" = Table.Group( #"Changed Type", {"id"}, {{"key1", each Combine([key1]), type text}, {"key2", each Combine([key2]), type text}, {"key3", each Combine([key3]), type text}, {"key4", each Combine([key4]), type text}} )
    in
        #"GroupRows"

     

    Best Regards,

    Herbert