Forum Discussion
A complex remove duplicates. Please help.
- 5 years ago
Hi, Anonymous
Based on your description, I created data to reproduce your scenario. The pbix file is attached in the end.
Table:
You may add a new step like below.
= Table.SelectRows(#"Changed Type",each let x = [New Hire Name], y=Table.RowCount( Table.SelectRows(#"Changed Type",each [New Hire Name]=x) ) in (y>1 and [Start Date]<>null) or y=1 )Result:
Best Regards
Allan
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Hi Anonymous
without more specifics hard to say, but take a look at this. It turns this:
into this:
I used this code:
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45W8srPyIt3yU9V0lEy1DfUNzIwMlCK1UERN8UhnleakwMRS8xLRRGLBQA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"New Hire Name" = _t, #"Start Date" = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Start Date", type date}}),
#"Grouped Rows" =
Table.Group(
#"Changed Type",
{"New Hire Name"},
{
{"AllRows",
each if Table.RowCount(_) = 1 then _ else
Table.Distinct(
Table.SelectRows(_, each [Start Date] <> null)
),
type table [New Hire Name=nullable text, Start Date=nullable date]
}
}
),
#"Expanded AllRows" = Table.ExpandTableColumn(#"Grouped Rows", "AllRows", {"Start Date"}, {"Start Date"})
in
#"Expanded AllRows"
The logic I used was this:
- Group all data by the New Hire Name
- Count the rows in the nested tables. If there is just one record keep it.
- If there are 2+ records, remove any rows with null, then do a Table.Distinct() over that result to remove any duplicate records. So if John Doe had two rows that were 5/1/2020, my logic would keep only one of those.
- Expanded the Start Date column again.
If that isn't what you want, please be more specific on how the logic should work.
How to use M code provided in a blank query:
1) In Power Query, select New Source, then Blank Query
2) On the Home ribbon, select "Advanced Editor" button
3) Remove everything you see, then paste the M code I've given you in that box.
4) Press Done
5) See this article if you need help using this M code in your model.