Forum Discussion
Remove Duplicate by comparing dates
Dear Community,
I have a some data and I'm updating every month and comparing those data. But I have a problem because of Duplicate.
I have created an example table myself below. My goal is to compare this previous month and the month before that. (February and March for today)
If I use the "Remove Duplicate" option in the query, I get an incorrect result. Because if there is a person named A in February or before, it automatically deletes that person from March. In comparison, person A in Germany does not appear at all in March. That's why my comparison is wrong.
What should I do for this situation? If I'm going to do the "Remove duplicate" that I need, I need to make it do it separately for each month. I divided it into different signs, but this time I am having trouble making comparisons as a comparison diagram.
| 01.03.2022 | A | Germany |
| 01.03.2022 | B | England |
| 03.03.2022 | B | England |
| 04.03.2022 | C | Spain |
| 05.03.2022 | D | Spain |
| 01.02.2022 | A | Germany |
| 01.02.2022 | A | Germany |
| 02.02.2022 | B | England |
| 04.02.2022 | C | Germany |
| 05.02.2022 | D | Spain |
Thank you for your support
Best Regards
See the working here - Open a blank query - Home - Advanced Editor - Remove everything from there and paste the below code to test (later on when you use the query on your dataset, you will have to change the source appropriately)
let Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMtY31DcyMDJS0lFyBGL31KLcxLxKpVgdFCknIHbNS89JzEuBShnjljKBSTljGGgKk3IB4uCCxMw8sIQRbkfglTLC5Qgj3I4wwuUIQxw2xQIA", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Date = _t, Group = _t, Country = _t]), #"Changed Type" = Table.TransformColumnTypes(Source,{{"Date", type date}, {"Group", type text}, {"Country", type text}}), #"Added Custom" = Table.AddColumn(#"Changed Type", "CurrentPreviousMonth", each [Date]>=Date.AddMonths(Date.StartOfMonth(Date.From(DateTime.FixedLocalNow())),-2) and [Date]<=Date.AddMonths(Date.EndOfMonth(Date.From(DateTime.FixedLocalNow())),-1)), #"Filtered Rows" = Table.SelectRows(#"Added Custom", each ([CurrentPreviousMonth] = true)), #"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"CurrentPreviousMonth"}), #"Added Custom1" = Table.AddColumn(#"Removed Columns", "Month", each Date.Month([Date])), #"Grouped Rows" = Table.Group(#"Added Custom1", {"Month"}, {{"Temp", each _, type table [Date=nullable date, Group=nullable text, Country=nullable text, Month=number]}}), #"Added Custom2" = Table.AddColumn(#"Grouped Rows", "Custom", each Table.Distinct([Temp], {"Group", "Country"})), #"Expanded Custom" = Table.ExpandTableColumn(#"Added Custom2", "Custom", {"Date", "Group", "Country"}, {"Date", "Group", "Country"}), #"Removed Columns1" = Table.RemoveColumns(#"Expanded Custom",{"Month", "Temp"}) in #"Removed Columns1"
7 Replies
For the given data, can you also post expected result?
Good Morning,
I'm sorry. I forgot to write expected result :For February User-A connected to the system 2 times and its duplicate for me, I'm removing it and
for March User-B connected to the system 2 times and I need to see it only one time.
February March Germany 2 1 England 1 1 Spain 1 2 But if I say from query "Remove duplicates" I cant see any result at March !!
Thanks.Let me rephrase it.
1. User A appears one time in March and 2 times in Feb.
Do you need to keep March entry and delete both entries of Feb or you want to keep one entry from Feb as well?
2. User B has 2 entries for Mar and one for Feb.
Thank you for your help.
I need to work on this code. It is a bit complicated.
