Forum Discussion
Eliminate duplicate records while keeping the most recent record.
- 1 year ago
Assuming the latest is the last entry for any set of duplicates, you can
- Group by Invoice Number
- Return only the last row each subtable
let //Read in your data Source = Csv.Document(File.Contents("C:\Users\ron\Desktop\Dataset.csv"),[Delimiter=";", Columns=2, Encoding=1252, QuoteStyle=QuoteStyle.None]), #"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]), #"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"N_Invoice", type text}, {"Index", Int64.Type}}), //Group by N_invoice and retain the last row #"Grouped Rows" = Table.Group(#"Changed Type", {"N_Invoice"}, { {"Latest", each Table.Last(_), type [N_Invoice=nullable text, Index=nullable number]}}), //remove column and re-expand #"Removed Columns" = Table.RemoveColumns(#"Grouped Rows",{"N_Invoice"}), #"Expanded Latest" = Table.ExpandRecordColumn(#"Removed Columns", "Latest", {"N_Invoice", "Index"}, {"N_Invoice", "Index"}) in #"Expanded Latest"This will also retain the original ordering of the entries. You can see in the screenshot below, FV-084482 is in the second row of both your original data, as well as the dedup'd data, even though the "latest" invoice was found near the bottom of the table as indicated by the associated Index number.
Note that Table.Group is not guaranteed to retain your original row order, which would be important in the above algorithm. However, others have reported that the presence of an Index column ameliorates this issue. So ensure that your actual data has an index column (or add one if it does not).
If there are issues of that nature, then merely change the custom aggregation in the Table.Group function to:
{"Latest", (t)=> Table.SelectRows(t, each [Index]=List.Max(t[Index])), type table [N_Invoice=nullable text, Index=nullable number]}})If you want a different order, please specify.
Hi GFire ,
Thank you for reaching out to the Fabric community.
You are correct by default, Power Query keeps the first (oldest) record when removing duplicates. However, if you prefer to keep the latest (newest) record, your method of finding the max index per duplicate is accurate.
I tested your scenario with your provided dataset and implemented a more optimized approach.
- Added an Index column.
- Sorted the table in descending order based on the Index (to bring the newest rows to the top).
- Used Table.Distinct on the N_Invoice column to keep only the first occurrence of each which now is the newest record.
This solution avoids the row by row logic and performs efficiently on larger datasets.
FYI:
Thanks for your response ronrsnfld & Nasif_Azam .
Hope this helps. If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.