A complex remove duplicates. Please help.

Hi Anonymous 

without more specifics hard to say, but take a look at this. It turns this:

into this:

I used this code:

let
    Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45W8srPyIt3yU9V0lEy1DfUNzIwMlCK1UERN8UhnleakwMRS8xLRRGLBQA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"New Hire Name" = _t, #"Start Date" = _t]),
    #"Changed Type" = Table.TransformColumnTypes(Source,{{"Start Date", type date}}),
    #"Grouped Rows" = 
        Table.Group(
            #"Changed Type", 
            {"New Hire Name"}, 
            {
                {"AllRows", 
                each if Table.RowCount(_) = 1 then _ else
                    Table.Distinct(
                        Table.SelectRows(_, each [Start Date] <> null)
                    ), 
                type table [New Hire Name=nullable text, Start Date=nullable date]
                }
            }
        ),
    #"Expanded AllRows" = Table.ExpandTableColumn(#"Grouped Rows", "AllRows", {"Start Date"}, {"Start Date"})
in
    #"Expanded AllRows"

The logic I used was this:

• Group all data by the New Hire Name
• Count the rows in the nested tables. If there is just one record keep it.
• If there are 2+ records, remove any rows with null, then do a Table.Distinct() over that result to remove any duplicate records. So if John Doe had two rows that were 5/1/2020, my logic would keep only one of those.
• Expanded the Start Date column again.

If that isn't what you want, please be more specific on how the logic should work.

How to use M code provided in a blank query:
1) In Power Query, select New Source, then Blank Query
2) On the Home ribbon, select "Advanced Editor" button
3) Remove everything you see, then paste the M code I've given you in that box.
4) Press Done
5) See this article if you need help using this M code in your model.

Anonymous ,

It doesn't like you need to remove the duplicated, actually you don't have any.

Does it work if you remove when there is more than 1 row for the New Hire column and delete the null row ?

Anonymous 

What is your logic to remove the record?

Do want to remove the lines for new hire names that have at least one record with start date?

________________________

If my answer was helpful, please consider Accept it as the solution to help the other members find it

Click on the Thumbs-Up icon if you like this reply 🙂

YouTube  LinkedIn

You could just use the Fill Down function in the GUI on the Date column, and then your next step would be Table.Distinct(PriorStep)