Forum Discussion

swalford's avatar
swalford
New Member
1 year ago
Solved

Removing Duplicate Strings from cells in a column

Hi,   Is there a way to remove duplicate strings from all cells in a column? I have a table with a column something like below.   Resources UserEmail SPN AAD UserEmail SPN AAD UserEmail ...
  • ronrsnfld's avatar
    1 year ago

    You can use the Table.TransformColumns function.

    See the code below for an example:

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WCi1OLXLNTczMUQgO8FNwdHRRwBBRitVBUofKI1bXYFQXCwA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Resources = _t]),
        #"Changed Type" = Table.TransformColumnTypes(Source,{{"Resources", type text}}),
        
        #"Remove Duplicate Phrases" = Table.TransformColumns(#"Changed Type",{"Resources", each 
            Text.Combine(
                List.Distinct(
                    Text.Split(_," ")
            ), " ")})
    in
        #"Remove Duplicate Phrases"

     

    or, since you wrote you have multiple tables, you can write it as a separate function, where you call it by specifying the Table and Column to process:

     

    Add as blank query and rename eg: fnDeDupPhrases 

    (tbl as table, col as text)=>
    
    Table.TransformColumns(tbl,{col, each 
            Text.Combine(
                List.Distinct(
                    Text.Split(_," ")
            ), " "), type text})

     

    Then use like this

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WCi1OLXLNTczMUQgO8FNwdHRRwBBRitVBUofKI1bXYFQXCwA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Resources = _t]),
        #"Changed Type" = Table.TransformColumnTypes(Source,{{"Resources", type text}}),
        
        #"Remove Duplicate Phrases" = fnDeDupPhrases(#"Changed Type","Resources")
    
    in
        #"Remove Duplicate Phrases"