Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
5 years ago
Solved

Faster Division/Multiplication using Merge same table Query - Power Query

I have a table of Columns containing Attribute counts. I have added a column that does row-wise summation of the Attribute column (denoted by Numbers) values. I want to Divide the Attribute columns by the "Count" column, in-place (replace values) i.e. do a `Percentage of Row-Total`. I don't want to add new columns, then do the calculation, then delete the original column.

 

"ID"  "CD"     "6"    "5"     "4"    "3"   "1"   "Count"

1RJ.14512453755302
2RJ.2436636579211
3RJ.32910441768258
4RJ.42646233411140
5RJ.53451412213161
6RJ.63363265813193
7RJ.7225640418167
8RJ.83493405411232
9RJ.93469345011198
10RJ.10274633428156
11RJ.113270264215185
12RJ.12225735455164
13RJ.13258339439199
14RJ.14316926508184
15RJ.152462393510170
16RJ.16117543576192
17RJ.17214521238118
18RJ.182051303212145
19RJ.19286034379168
20RJ.20263829414138
21RJ.212164234211161
22RJ.223361353811178
23RJ.23195136368150
24RJ.242265323611166
25RJ.252885354511204
26RJ.262656282711148
27RJ.27294341337153
28RJ.282257154311148
29RJ.29143022359110
30RJ.30193525278114
31RJ.311830282612114
32RJ.321354364410157
33RJ.33205919409147
34RJ.34243438235124
35RJ.35266737398177
36RJ.36274631347145
37RJ.372426273613126
38RJ.38142424211093
39RJ.393158342810161
40RJ.401845253710135

 

In one of my searches, someone had suggested that the values for multiple columns could be calculated in a faster way (avoid row level context evaluation) by merging the table with itself and with some M-code. But not shared the code example.

Is it possible?

  • Hello Anonymous 

     

    here some dynamic solution. Uses Unpivoting, Grouping and twice transforming of columns. The only parameter you have to change is the list in the Unpivot other (in case you have some additional columns). As you can see you can also remove the column "count" from your table.

    let
        Origine = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("VVRLdiQxCLtLr/PmGTD+XGGWs+3X979GCgmYZNFWUiVTIMl+v1/y+nr9+/snYPqziM5ndXuWHf/Hz4a+Pl/vl5IbMIOwVryMxfez3OenIqAaqQEaz2VE2Rmf2bHhxAs/4E5yAzTezVg0tlo8k9gkc4Dr5Hq9dKmyGm1J7JLFHha5aBLtWn3BT3OvgbvJ3VXI0cio4uCvDeoh9VQL14rq3a4aFbvk3uKu/stHj3Ypg4y0IlB3CYHWp1YTT2Mgl2+BFm/3aP2gBMw8TnY6B+R8Ud+8XAd5TZLTO6DGmwMvbtkOP+8lOc0DmtSIVHlU0ycrp3tA5GxpVUYvGF02rZb0D4g5EUi0gPbx/FJoSQOB2mlWqSihD0ml00KgjoqRjdISOslM8dJEoMaWNcpFq9zLYmlNE7XdsFNnAElCRCzJaSK7hHiz+qWJ8iPOWsdPO9BSwtlp9s7a6aIy5T1jH1mmiVJrughE/eWlha3/jTB7mjYCIcjxX2niERg0XdNGIHOxah9Snuc7204fgdqRg3SYGTuch1bTR9bqWEvH5Hfp9JH30SzDKaeXj0JFLH0ESgeUI+/OE0e09BGIULHyqYmZp2Knj9YXFu4N6DxnnwLnbWN1k1pn9VZLuHQ4TZLTRyDMZEpPpapueJDTRg4WX1+7Io0jiRF3Vk4XgT8uJ6lv7B8nxtJEoPa9jm2ME8xRxsnSRWa4O+cipUfe05Ym8s6It94XMfSmeHlkZroIhDGzXURzZD8KfD7f", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID = _t, CD = _t, #"6" = _t, #"5" = _t, #"4" = _t, #"3" = _t, #"1" = _t, Count = _t]),
        ChangeType = Table.TransformColumnTypes(Origine,{{"6", Int64.Type}, {"5", Int64.Type}, {"4", Int64.Type}, {"3", Int64.Type}, {"1", Int64.Type}}),
        #"Removed Columns" = Table.RemoveColumns(ChangeType,{"Count"}),
        #"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Removed Columns", {"ID", "CD"}, "Attribute", "Value"),
        #"Grouped Rows" = Table.Group(#"Unpivoted Other Columns", {"CD"}, {{"AllRows", each _, type table [ID=text, CD=text, Attribute=text, Value=number]}}),
        TransformAllRows = Table.TransformColumns
        (
            #"Grouped Rows",
            {
                {
                    "AllRows",
                    (tbl)=> 
                    let
                        SumValue = List.Sum(tbl[Value]),
                        TransformTbl = Table.TransformColumns
                        (
                            tbl,
                            {
                                {
                                    "Value",
                                    each _ / SumValue
                                }
                            }
                        )
                    in 
                        TransformTbl
    
                }
            }
        ),
        #"Expanded AllRows" = Table.ExpandTableColumn(TransformAllRows, "AllRows", {"ID", "Attribute", "Value"}, {"ID", "Attribute", "Value"}),
        #"Pivoted Column" = Table.Pivot(#"Expanded AllRows", List.Distinct(#"Expanded AllRows"[Attribute]), "Attribute", "Value", List.Sum),
        #"Changed Type" = Table.TransformColumnTypes(#"Pivoted Column",{{"ID", Int64.Type}}),
        #"Sorted Rows" = Table.Sort(#"Changed Type",{{"ID", Order.Ascending}})
    in
        #"Sorted Rows"

     

    Copy paste this code to the advanced editor in a new blank query to see how the solution works.

    If this post helps or solves your problem, please mark it as solution (to help other users find useful content and to acknowledge the work of users that helped you)
    Kudoes are nice too

    Have fun

    Jimmy

     

24 Replies

  • Jimmy801's avatar
    Jimmy801
    Community Champion

    Hello Anonymous 

     

    here some dynamic solution. Uses Unpivoting, Grouping and twice transforming of columns. The only parameter you have to change is the list in the Unpivot other (in case you have some additional columns). As you can see you can also remove the column "count" from your table.

    let
        Origine = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("VVRLdiQxCLtLr/PmGTD+XGGWs+3X979GCgmYZNFWUiVTIMl+v1/y+nr9+/snYPqziM5ndXuWHf/Hz4a+Pl/vl5IbMIOwVryMxfez3OenIqAaqQEaz2VE2Rmf2bHhxAs/4E5yAzTezVg0tlo8k9gkc4Dr5Hq9dKmyGm1J7JLFHha5aBLtWn3BT3OvgbvJ3VXI0cio4uCvDeoh9VQL14rq3a4aFbvk3uKu/stHj3Ypg4y0IlB3CYHWp1YTT2Mgl2+BFm/3aP2gBMw8TnY6B+R8Ud+8XAd5TZLTO6DGmwMvbtkOP+8lOc0DmtSIVHlU0ycrp3tA5GxpVUYvGF02rZb0D4g5EUi0gPbx/FJoSQOB2mlWqSihD0ml00KgjoqRjdISOslM8dJEoMaWNcpFq9zLYmlNE7XdsFNnAElCRCzJaSK7hHiz+qWJ8iPOWsdPO9BSwtlp9s7a6aIy5T1jH1mmiVJrughE/eWlha3/jTB7mjYCIcjxX2niERg0XdNGIHOxah9Snuc7204fgdqRg3SYGTuch1bTR9bqWEvH5Hfp9JH30SzDKaeXj0JFLH0ESgeUI+/OE0e09BGIULHyqYmZp2Knj9YXFu4N6DxnnwLnbWN1k1pn9VZLuHQ4TZLTRyDMZEpPpapueJDTRg4WX1+7Io0jiRF3Vk4XgT8uJ6lv7B8nxtJEoPa9jm2ME8xRxsnSRWa4O+cipUfe05Ym8s6It94XMfSmeHlkZroIhDGzXURzZD8KfD7f", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID = _t, CD = _t, #"6" = _t, #"5" = _t, #"4" = _t, #"3" = _t, #"1" = _t, Count = _t]),
        ChangeType = Table.TransformColumnTypes(Origine,{{"6", Int64.Type}, {"5", Int64.Type}, {"4", Int64.Type}, {"3", Int64.Type}, {"1", Int64.Type}}),
        #"Removed Columns" = Table.RemoveColumns(ChangeType,{"Count"}),
        #"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Removed Columns", {"ID", "CD"}, "Attribute", "Value"),
        #"Grouped Rows" = Table.Group(#"Unpivoted Other Columns", {"CD"}, {{"AllRows", each _, type table [ID=text, CD=text, Attribute=text, Value=number]}}),
        TransformAllRows = Table.TransformColumns
        (
            #"Grouped Rows",
            {
                {
                    "AllRows",
                    (tbl)=> 
                    let
                        SumValue = List.Sum(tbl[Value]),
                        TransformTbl = Table.TransformColumns
                        (
                            tbl,
                            {
                                {
                                    "Value",
                                    each _ / SumValue
                                }
                            }
                        )
                    in 
                        TransformTbl
    
                }
            }
        ),
        #"Expanded AllRows" = Table.ExpandTableColumn(TransformAllRows, "AllRows", {"ID", "Attribute", "Value"}, {"ID", "Attribute", "Value"}),
        #"Pivoted Column" = Table.Pivot(#"Expanded AllRows", List.Distinct(#"Expanded AllRows"[Attribute]), "Attribute", "Value", List.Sum),
        #"Changed Type" = Table.TransformColumnTypes(#"Pivoted Column",{{"ID", Int64.Type}}),
        #"Sorted Rows" = Table.Sort(#"Changed Type",{{"ID", Order.Ascending}})
    in
        #"Sorted Rows"

     

    Copy paste this code to the advanced editor in a new blank query to see how the solution works.

    If this post helps or solves your problem, please mark it as solution (to help other users find useful content and to acknowledge the work of users that helped you)
    Kudoes are nice too

    Have fun

    Jimmy

     

    • Anonymous's avatar
      Anonymous
      Not applicable

      Hi Jimmy801 ,

       

      I found this solution last night which is pretty fast and consice:

       

      let
          Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("VVRLdiQxCLtLr/PmGTD+XGGWs+3X979GCgmYZNFWUiVTIMl+v1/y+nr9+/snYPqziM5ndXuWHf/Hz4a+Pl/vl5IbMIOwVryMxfez3OenIqAaqQEaz2VE2Rmf2bHhxAs/4E5yAzTezVg0tlo8k9gkc4Dr5Hq9dKmyGm1J7JLFHha5aBLtWn3BT3OvgbvJ3VXI0cio4uCvDeoh9VQL14rq3a4aFbvk3uKu/stHj3Ypg4y0IlB3CYHWp1YTT2Mgl2+BFm/3aP2gBMw8TnY6B+R8Ud+8XAd5TZLTO6DGmwMvbtkOP+8lOc0DmtSIVHlU0ycrp3tA5GxpVUYvGF02rZb0D4g5EUi0gPbx/FJoSQOB2mlWqSihD0ml00KgjoqRjdISOslM8dJEoMaWNcpFq9zLYmlNE7XdsFNnAElCRCzJaSK7hHiz+qWJ8iPOWsdPO9BSwtlp9s7a6aIy5T1jH1mmiVJrughE/eWlha3/jTB7mjYCIcjxX2niERg0XdNGIHOxah9Snuc7204fgdqRg3SYGTuch1bTR9bqWEvH5Hfp9JH30SzDKaeXj0JFLH0ESgeUI+/OE0e09BGIULHyqYmZp2Knj9YXFu4N6DxnnwLnbWN1k1pn9VZLuHQ4TZLTRyDMZEpPpapueJDTRg4WX1+7Io0jiRF3Vk4XgT8uJ6lv7B8nxtJEoPa9jm2ME8xRxsnSRWa4O+cipUfe05Ym8s6It94XMfSmeHlkZroIhDGzXURzZD8KfD7f", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID = _t, CD = _t, #"6" = _t, #"5" = _t, #"4" = _t, #"3" = _t, #"1" = _t, Counts = _t]),
          #"Changed Type" = Table.TransformColumnTypes(Source,{{"6", Int64.Type}, {"5", Int64.Type}, {"4", Int64.Type}, {"3", Int64.Type}, {"1", Int64.Type}, {"Counts", Int64.Type}}),
          
          PercentOfRow = Table.FromRecords(
          Table.TransformRows(
              #"Changed Type",
              (r) => 
              Record.TransformFields(
                  r,
                  {
                      {"6", each r[#"6"] / r[Counts] },
                      {"5", each r[#"5"] / r[Counts] },
                      {"4", each r[#"4"] / r[Counts] },
                      {"3", each r[#"3"] / r[Counts] },
                      {"1", each r[#"1"] / r[Counts] }
                  }
              )
          ),        
          type table [
                     CD= Text.Type,
                     ID = Int64.Type,
                     #"6" = Percentage.Type, 
                     #"5" = Percentage.Type,
                     #"4" = Percentage.Type,
                     #"3" = Percentage.Type,
                     #"1" = Percentage.Type
          ]
      )
      in
          PercentOfRow

       

      taken from here:

      https://stackoverflow.com/questions/31885049/power-query-transform-a-column-by-multiplying-by-another-column?answertab=votes#tab-top 

      and here:

      https://stackoverflow.com/questions/31548135/power-query-transform-a-column-based-on-another-column?answertab=votes#tab-top 

       

      • Jimmy801's avatar
        Jimmy801
        Community Champion

        hello Anonymous 

         

        i wanted propose you this solution, but it's not dynamically and you need the counts-column to make it nice. therefore I opted for the TransformColumns-solution

         

        BR

         

        Jimmy

  • Anonymous's avatar
    Anonymous
    Not applicable

    It's not clear why you don't want to do row-level transformations.
    Nothing comes to mind about how to use the Table.Join functions to do these transformations (considering that these functions do not expose parameters which are functions that act on the rows / columns of the tables). You should provide the exact references of the post where you found these tips.
    In the meantime, I can offer you this "strange" solution that makes use of the pivot and unpivot functions.

     

     

     

    let
        Origine = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("VVRLdiQxCLtLr/PmGTD+XGGWs+3X979GCgmYZNFWUiVTIMl+v1/y+nr9+/snYPqziM5ndXuWHf/Hz4a+Pl/vl5IbMIOwVryMxfez3OenIqAaqQEaz2VE2Rmf2bHhxAs/4E5yAzTezVg0tlo8k9gkc4Dr5Hq9dKmyGm1J7JLFHha5aBLtWn3BT3OvgbvJ3VXI0cio4uCvDeoh9VQL14rq3a4aFbvk3uKu/stHj3Ypg4y0IlB3CYHWp1YTT2Mgl2+BFm/3aP2gBMw8TnY6B+R8Ud+8XAd5TZLTO6DGmwMvbtkOP+8lOc0DmtSIVHlU0ycrp3tA5GxpVUYvGF02rZb0D4g5EUi0gPbx/FJoSQOB2mlWqSihD0ml00KgjoqRjdISOslM8dJEoMaWNcpFq9zLYmlNE7XdsFNnAElCRCzJaSK7hHiz+qWJ8iPOWsdPO9BSwtlp9s7a6aIy5T1jH1mmiVJrughE/eWlha3/jTB7mjYCIcjxX2niERg0XdNGIHOxah9Snuc7204fgdqRg3SYGTuch1bTR9bqWEvH5Hfp9JH30SzDKaeXj0JFLH0ESgeUI+/OE0e09BGIULHyqYmZp2Knj9YXFu4N6DxnnwLnbWN1k1pn9VZLuHQ4TZLTRyDMZEpPpapueJDTRg4WX1+7Io0jiRF3Vk4XgT8uJ6lv7B8nxtJEoPa9jm2ME8xRxsnSRWa4O+cipUfe05Ym8s6It94XMfSmeHlkZroIhDGzXURzZD8KfD7f", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID = _t, CD = _t, #"6" = _t, #"5" = _t, #"4" = _t, #"3" = _t, #"1" = _t, Count = _t]),
        #"Rimosse colonne" = Table.RemoveColumns(Origine,{"ID","Count"}),
        #"Modificato tipo" = Table.TransformColumnTypes(#"Rimosse colonne",{{"6", Int64.Type}, {"5", Int64.Type}, {"4", Int64.Type}, {"3", Int64.Type}, {"1", Int64.Type}}),
        #"Trasformate altre colonne tramite UnPivot" = Table.UnpivotOtherColumns(#"Modificato tipo", {"CD"}, "Attributo", "Valore"),
        #"Rimosse colonne1" = Table.RemoveColumns(#"Trasformate altre colonne tramite UnPivot",{"Attributo"}),
        #"Colonna trasformata tramite Pivot" = Table.Pivot(#"Rimosse colonne1", List.Distinct(#"Rimosse colonne1"[CD]), "CD", "Valore", (v)=>List.Transform(v, each _/List.Sum(v))),
        #"Trasposta colonna" = Table.Transpose(#"Colonna trasformata tramite Pivot"),
        #"Valori estratti" = Table.TransformColumns(#"Trasposta colonna", {"Column1", each Text.Combine(List.Transform(_, Text.From), ":"), type text}),
        #"Suddividi colonna in base al delimitatore" = Table.SplitColumn(#"Valori estratti", "Column1", Splitter.SplitTextByDelimiter(":", QuoteStyle.Csv), {"Column1.1", "Column1.2", "Column1.3", "Column1.4", "Column1.5"}),
        #"Modificato tipo1" = Table.TransformColumnTypes(#"Suddividi colonna in base al delimitatore",{{"Column1.1", type number}, {"Column1.2", type number}, {"Column1.3", type number}, {"Column1.4", type number}, {"Column1.5", type number}})
    in
        #"Modificato tipo1"

     

     

  • edhans's avatar
    edhans
    Community Champion

    Hi Anonymous - oddly enough I do not think you can reference another column within a Table.TransformColumn. I've not see a way to do it.

    However, you can use the Table.ReplaceValue function. See below - this is just for column 6. You'd need to repeat for each column.

     

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("VVRLdiQxCLtLr/PmGTD+XGGWs+3X979GCgmYZNFWUiVTIMl+v1/y+nr9+/snYPqziM5ndXuWHf/Hz4a+Pl/vl5IbMIOwVryMxfez3OenIqAaqQEaz2VE2Rmf2bHhxAs/4E5yAzTezVg0tlo8k9gkc4Dr5Hq9dKmyGm1J7JLFHha5aBLtWn3BT3OvgbvJ3VXI0cio4uCvDeoh9VQL14rq3a4aFbvk3uKu/stHj3Ypg4y0IlB3CYHWp1YTT2Mgl2+BFm/3aP2gBMw8TnY6B+R8Ud+8XAd5TZLTO6DGmwMvbtkOP+8lOc0DmtSIVHlU0ycrp3tA5GxpVUYvGF02rZb0D4g5EUi0gPbx/FJoSQOB2mlWqSihD0ml00KgjoqRjdISOslM8dJEoMaWNcpFq9zLYmlNE7XdsFNnAElCRCzJaSK7hHiz+qWJ8iPOWsdPO9BSwtlp9s7a6aIy5T1jH1mmiVJrughE/eWlha3/jTB7mjYCIcjxX2niERg0XdNGIHOxah9Snuc7204fgdqRg3SYGTuch1bTR9bqWEvH5Hfp9JH30SzDKaeXj0JFLH0ESgeUI+/OE0e09BGIULHyqYmZp2Knj9YXFu4N6DxnnwLnbWN1k1pn9VZLuHQ4TZLTRyDMZEpPpapueJDTRg4WX1+7Io0jiRF3Vk4XgT8uJ6lv7B8nxtJEoPa9jm2ME8xRxsnSRWa4O+cipUfe05Ym8s6It94XMfSmeHlkZroIhDGzXURzZD8KfD7f", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [DI = _t, CD = _t, #"6" = _t, #"5" = _t, #"4" = _t, #"3" = _t, #"1" = _t, Count = _t]),
        #"Changed Type" = Table.TransformColumnTypes(Source,{{"6", Int64.Type}, {"5", Int64.Type}, {"4", Int64.Type}, {"3", Int64.Type}, {"1", Int64.Type}, {"Count", Int64.Type}}),
        #"Replaced Value" = 
        Table.ReplaceValue(
            #"Changed Type",
            each
            let
                varCurrentValue = [6]
            in
                varCurrentValue,
    
                each [6] / [Count],
                Replacer.ReplaceValue,{"6"})
    in
        #"Replaced Value"

     

    Here is what it does:

    • It assigns the current value of column 6 to the varCurrentValue variable. (current value is whatever it is for each row)
    • It searches for the current value.
    • Then it replaces it with [6] / [Count]

    I did test this with 2 records that had the same value (45) but different values in the Count column for that row, and it worked.

     

     

    The first value was 45, and now divided by 302 gives the .149 value.

     

    How to use M code provided in a blank query:
    1) In Power Query, select New Source, then Blank Query
    2) On the Home ribbon, select "Advanced Editor" button
    3) Remove everything you see, then paste the M code I've given you in that box.
    4) Press Done
    5) See this article if you need help using this M code in your model.

  • Anonymous's avatar
    Anonymous
    Not applicable

    Have you tried just transforming the columns?

     

    Table.TransformColumns(PriorStep, {{"6",each _/[Count]}, {"5", each _/[Count]}})

     

    No reason that shouldn't work for ya. 

    • Anonymous's avatar
      Anonymous
      Not applicable

      Hi Anonymous,

       

      Thanks for your reply. Could you look at the sample files that i had shared earlier and see if you could recreate the Output sheet table from the given sample file?