Forum Discussion

Cado_one's avatar
Cado_one
Resolver III
4 years ago
Solved

Conditionnaly remove duplicates

Hello !   I face an issue with one table, I will try to reproduce this issue as a fictive exercise below. Here is a table : ID NAME SOURCE 1 PLANT A MYSQL 2 PLANT B1 MYSQL 2 PL...
  • AlexisOlson's avatar
    4 years ago

    Split the table into WEB API and not WEB API rows. In the former, remove rows with IDs that exist in the latter and then append them together.

     

    Try pasting this into the Advanced Editor of a new blank query to see the steps:

     

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMlTSUQrwcfQLUXAEsnwjgwN9lGJ1opWM4OJOhrgkjFAkjOESzijiJnBxFyAr3NVJwTHAE90oYzQpU7iUK7JMLAA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID = _t, NAME = _t, SOURCE = _t]),
        #"Changed Type" = Table.TransformColumnTypes(Source,{{"ID", Int64.Type}, {"NAME", type text}, {"SOURCE", type text}}),
        #"Filtered NonWEB" = Table.SelectRows(#"Changed Type", each ([SOURCE] <> "WEB API")),
        #"Filtered WEBAPI" = Table.SelectRows(#"Changed Type", each ([SOURCE] = "WEB API")),
        #"Removed Duplicates" = Table.SelectRows(#"Filtered WEBAPI", each not List.Contains(#"Filtered NonWEB"[ID], [ID])),
        #"Appended Query" = Table.Combine({#"Filtered NonWEB", #"Removed Duplicates"})
    in
        #"Appended Query"

     

    For large datasets, it's probably more efficient to use an anti-join rather than List.Contains but the idea is the same.