Forum Discussion

v_mark's avatar
v_mark
Helper V
5 years ago
Solved

Identify Duplicates based on multiple conditions/columns

I have an entire dataset which has multiple entries that has duplicates. 

 

So far this is how it looks like. The 2nd row has a duplicate 8 Characters on the top
I need to identify in any event these behaviors occurs as considered duplicates.  TIA

Need Help
Show and Tell
Tips and Tricks
Tutorial Requests
  • v-alq-msft's avatar
    v-alq-msft
    5 years ago

    Hi, v_mark 

     

    Based on your description, I created data to reproduce your scenario. The pbix file is attached in the end.

    Table:

     

    You may create a custom column with the following m codes.

     

    let 
device=[Device],
status=[Status],
state=[State],
sn=[SN],
tab = Table.SelectRows(#"Renamed Columns",each 
[Device]=device and 
[Status]=status and 
[State]=state and (
Text.EndsWith([SN],sn) or Text.EndsWith(sn,[SN])) ),
re = Table.RowCount(tab)
in 
if re>=2 
then 
"duplicate"
else null

     

     

    Result:

     

    Best Regards

    Allan

     

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

     

3 Replies

  • v-alq-msft's avatar
    v-alq-msft
    Community Support
    5 years ago

    Hi, v_mark 

     

    Based on your description, I created data to reproduce your scenario. The pbix file is attached in the end.

    Table:

     

    You may create a custom column with the following m codes.

     

    let 
device=[Device],
status=[Status],
state=[State],
sn=[SN],
tab = Table.SelectRows(#"Renamed Columns",each 
[Device]=device and 
[Status]=status and 
[State]=state and (
Text.EndsWith([SN],sn) or Text.EndsWith(sn,[SN])) ),
re = Table.RowCount(tab)
in 
if re>=2 
then 
"duplicate"
else null

     

     

    Result:

     

    Best Regards

    Allan

     

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

     

  • edhans's avatar
    edhans
    Community Champion
    5 years ago

    Is it alwaysthe last 8 chars, or is it random lengths, random positions? Also, we cannot work with images. Images are great for expected outcome, but not for supplying data.

     

    How to get good help fast. Help us help you.
    How to Get Your Question Answered Quickly - Give us a good and concise explanation
    How to provide sample data in the Power BI Forum - Provide data in a table format per the link. Provide expected output using a screenshot of Excel or other image. Do not provide a screenshot of the source data. I cannot paste an image into Power BI tables.

  • Jimmy801's avatar
    Jimmy801
    Community Champion
    5 years ago

    Hello v_mark 

     

    For this kinda request Table.Distinct isn't working because it doesn't accept a custom comparer. However there is a workaround using Table.Group where you can use the 5th parameter to create your own custom comparer but you have to implement it for every column in your table (as well as in the second parameter). Here an example

    code that does the comparing of your SN. It checks wheter value a is an ending of value b or the other way round (dont't forget you have to implement all your columns that you need the distincted value as well)

    Text.EndsWith(o[C], n[C]) or Text.EndsWith(n[C], o[C])

     

    let
    Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WclTSUXICYkMj48SkZKVYHYQQmB8LAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [A = _t, B = _t, C = _t]),
    #"Grouped Rows" = Table.Group(Source, {"A", "B", "C"}, {{"AllRows", each _, type table [A=text, B=text, C=text]}}, GroupKind.Global, (o,n)=> if o[A]=n[A] and o[B]=n[B] and (Text.EndsWith(o[C], n[C]) or Text.EndsWith(n[C], o[C])) then 0 else -1)
in
    #"Grouped Rows"

    Outcome

    Copy paste this code to the advanced editor in a new blank query to see how the solution works.

    If this post helps or solves your problem, please mark it as solution (to help other users find useful content and to acknowledge the work of users that helped you)
    Kudoes are nice too

    Have fun

    Jimmy