Forum Discussion
Eliminate duplicate records while keeping the most recent record.
- 1 year ago
Assuming the latest is the last entry for any set of duplicates, you can
- Group by Invoice Number
- Return only the last row each subtable
let //Read in your data Source = Csv.Document(File.Contents("C:\Users\ron\Desktop\Dataset.csv"),[Delimiter=";", Columns=2, Encoding=1252, QuoteStyle=QuoteStyle.None]), #"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]), #"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"N_Invoice", type text}, {"Index", Int64.Type}}), //Group by N_invoice and retain the last row #"Grouped Rows" = Table.Group(#"Changed Type", {"N_Invoice"}, { {"Latest", each Table.Last(_), type [N_Invoice=nullable text, Index=nullable number]}}), //remove column and re-expand #"Removed Columns" = Table.RemoveColumns(#"Grouped Rows",{"N_Invoice"}), #"Expanded Latest" = Table.ExpandRecordColumn(#"Removed Columns", "Latest", {"N_Invoice", "Index"}, {"N_Invoice", "Index"}) in #"Expanded Latest"This will also retain the original ordering of the entries. You can see in the screenshot below, FV-084482 is in the second row of both your original data, as well as the dedup'd data, even though the "latest" invoice was found near the bottom of the table as indicated by the associated Index number.
Note that Table.Group is not guaranteed to retain your original row order, which would be important in the above algorithm. However, others have reported that the presence of an Index column ameliorates this issue. So ensure that your actual data has an index column (or add one if it does not).
If there are issues of that nature, then merely change the custom aggregation in the Table.Group function to:
{"Latest", (t)=> Table.SelectRows(t, each [Index]=List.Max(t[Index])), type table [N_Invoice=nullable text, Index=nullable number]}})If you want a different order, please specify.
Assuming the latest is the last entry for any set of duplicates, you can
- Group by Invoice Number
- Return only the last row each subtable
let
//Read in your data
Source = Csv.Document(File.Contents("C:\Users\ron\Desktop\Dataset.csv"),[Delimiter=";", Columns=2, Encoding=1252, QuoteStyle=QuoteStyle.None]),
#"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
#"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"N_Invoice", type text}, {"Index", Int64.Type}}),
//Group by N_invoice and retain the last row
#"Grouped Rows" = Table.Group(#"Changed Type", {"N_Invoice"}, {
{"Latest", each Table.Last(_), type [N_Invoice=nullable text, Index=nullable number]}}),
//remove column and re-expand
#"Removed Columns" = Table.RemoveColumns(#"Grouped Rows",{"N_Invoice"}),
#"Expanded Latest" = Table.ExpandRecordColumn(#"Removed Columns", "Latest", {"N_Invoice", "Index"}, {"N_Invoice", "Index"})
in
#"Expanded Latest"
This will also retain the original ordering of the entries. You can see in the screenshot below, FV-084482 is in the second row of both your original data, as well as the dedup'd data, even though the "latest" invoice was found near the bottom of the table as indicated by the associated Index number.
Note that Table.Group is not guaranteed to retain your original row order, which would be important in the above algorithm. However, others have reported that the presence of an Index column ameliorates this issue. So ensure that your actual data has an index column (or add one if it does not).
If there are issues of that nature, then merely change the custom aggregation in the Table.Group function to:
{"Latest", (t)=> Table.SelectRows(t, each [Index]=List.Max(t[Index])), type table [N_Invoice=nullable text, Index=nullable number]}})
If you want a different order, please specify.