Forum Discussion

ngct1112's avatar
ngct1112
Post Patron
4 years ago
Solved

PowerQuery - Remove Duplicated lines based on criteria

Hi all, I would like to remove lines with duplicated "ID" and keep the lowest "price" line. May I know if there any quick way(not using groupby) I could do for this case?

Original Table

IDPrice
14
1346
327
4458
5734
56
745
785
77

 

Desired outcome:

IDPrice
14
327
4458
56
77

 

Appreciated if any help

Need Help
  • smpa01's avatar
    smpa01
    4 years ago

    ngct1112  try this

    CT = Table.TransformColumnTypes(#"Promoted Headers", {{"ID", Int64.Type}, {"Price", Int64.Type}}), 
  #"Filtered Rows" = Table.SelectRows(
    CT, 
    each ([Price] = List.Min(Table.SelectRows(CT, (q) => q[ID] = [ID])[Price]))
  )
  • freginier's avatar
    freginier
    4 years ago

    Easier way, add sorted rows, buffer and remove duplicate step 

    let
    Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WclTSUTI0UIrVgTDNwSwnIMsSzjI0UoqNBQA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t, Column2 = _t]),
    #"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", type text}, {"Column2", Int64.Type}}),
    #"Sorted Rows" = Table.Sort(#"Changed Type",{{"Column2", Order.Ascending}}),
    #"Buffered" = Table.Buffer(#"Sorted Rows"),
    #"Removed Duplicates" = Table.Distinct(Buffered, {"Column1"})
in
    #"Removed Duplicates"

     

7 Replies

  • freginier's avatar
    freginier
    Solution Sage
    4 years ago

    Easier way, add sorted rows, buffer and remove duplicate step 

    let
    Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WclTSUTI0UIrVgTDNwSwnIMsSzjI0UoqNBQA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t, Column2 = _t]),
    #"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", type text}, {"Column2", Int64.Type}}),
    #"Sorted Rows" = Table.Sort(#"Changed Type",{{"Column2", Order.Ascending}}),
    #"Buffered" = Table.Buffer(#"Sorted Rows"),
    #"Removed Duplicates" = Table.Distinct(Buffered, {"Column1"})
in
    #"Removed Duplicates"

     

  • smpa01's avatar
    smpa01
    Community Champion
    4 years ago

    ngct1112  try this

    CT = Table.TransformColumnTypes(#"Promoted Headers", {{"ID", Int64.Type}, {"Price", Int64.Type}}), 
  #"Filtered Rows" = Table.SelectRows(
    CT, 
    each ([Price] = List.Min(Table.SelectRows(CT, (q) => q[ID] = [ID])[Price]))
  )
    • AlexisOlson's avatar
      AlexisOlson
      Super User
      4 years ago

      This feels like a DAX sort of approach. In M, you want to be very careful about doing table filtering like this since the O(n^2) complexity will make it impractical for larger datasets.

       

      In this particular case, we can do this with a simple Group By:

      = Table.Group(#"Changed Type", {"ID"}, {{"Price", each List.Min([Price]), type nullable number}})

      • smpa01's avatar
        smpa01
        Community Champion
        4 years ago

        AlexisOlson  This feels like a DAX sort of approach -  yes it is  ğŸ˜€ I was simply testing out how can I avoid  groupby (OP wants to avoid groupby)and still get there and translate DAX to M