Forum Discussion
Efficiently iterate over and modify a list
My data consists of rows by company, reference, date and group. Some of the rows are missing the group, and I'm trying to populate this by finding earlier rows (by date) with a similar reference within the same company.
To do this I first do a nested self join on company, then I filter each nested table to remove dates after the current row's date, and rows missing the group. Then, for each row missing the group I calculate the Levenshtein distance between its reference and the reference of each row of the associated nested table. Finally I decide which row has the lowest distance, and keep that row (taking the first in case of ties).
I can't do this using a fuzzy merge because of the condition of matching by company and removing later dates. I also found clustering didn't help because the similarity algorithm that it uses doesn't seem to be granular enough (same would go for merging I guess).
I am blocked from uploading data to file sharing sites, but the following script covers the entire process and includes sample data. LevenshteinDist is the custom function from the original post.
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("jVNNc4IwEP0rGc/WfKAix7anTnvRHnpwPGTiFlMlMBC09td3IYRGcZwe2EmWfW/f2yTr9ehxNB4tl3E04aSAssoNsWAqnRvMC8ZnlEU04s2mC5uxA63ehZhEJKsrrUgmjUyh7DFTGjHc8NgFj3l75YxPYqLN50FabEJUnmXaWgAHjSmbUyFwM41d8NArjTrdWQdJKIudwkt9PSAvDnBhSnDKuBN4z5SWXbmgTFCx+JcfCzJzqKixwjpdnZEnXOqsKPMjZGAssflpMGceueDrV1DU1pEX8tzAglG1JpL28/UvAb/S9txX/00pYA+qK5QTIBa963t6+jOIenYn5bnrywV5ICe9TcHiLdn7OxJTLhxiNnMhAC3YvAdVuKqLrbSwJUd5qKFX17m/hW+bVuhM7bGtKutqByWmDJwIZre1sp7Gy4iFCwOaD6e9OVjcSZPbhkvtpEnhmmSgJLmtBHOY+gLVuNIGHwH4Qd7RkwxmGcymxFhdkwxnIxjjgaJmut/nH9n9a9/F7e7Ie2kl95jAfIPY/AI=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Company = _t, Reference = _t, Date = _t, Group = _t, #"Correct group" = _t]),
#"Replaced Value" = Table.ReplaceValue(Source,"",null,Replacer.ReplaceValue,{"Group"}),
#"Changed Type" = Table.TransformColumnTypes(#"Replaced Value",{{"Company", type text}, {"Reference", type text}, {"Date", type date}, {"Group", type nullable text}, {"Correct group", type nullable text}}),
#"Filter for rows missing group" = Table.SelectRows(#"Changed Type", each ([Group] = null)),
#"Self join" = Table.NestedJoin(#"Filter for rows missing group", "Company", #"Changed Type", "Company", "AllData"),
#"Filter for usable rows" = Table.AddColumn(#"Self join", "AllData filtered", each Table.SelectRows(_[AllData], (x) => (x[Date] < _[Date]) and (x[Group] <> null))),
#"Push reference to nested table" = Table.AddColumn(#"Filter for usable rows", "AllData with ref", each Table.AddColumn(_[AllData filtered], "Outer ref", (x) => _[Reference])),
#"Get Levenshtein Distance" = Table.TransformColumns(#"Push reference to nested table", {{"AllData with ref", each Table.AddColumn(_, "LD", (x) => LevenshteinDist(x[Reference], x[Outer ref]))}}),
#"Get best group" = Table.TransformColumns(#"Get Levenshtein Distance", {{"AllData with ref", each
let
m = List.Min(_[LD]),
filter = Table.SelectRows(_, (x) => x[LD] = m),
v = filter{0}[Group]
in
v}})
in
#"Get best group"