Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
6 years ago

How to do full-text search and return coding for a new column?

Hi Sir/Madam,

 

I got a coding table like this:

NameCode
Bob1
Ann2
Cat3
Peter4
Mary5
Tina6

 

Then, I want to do full-text search in the first column of the following table and ask Query editor to fill-in the code for me in the 2nd column of the following table.

Full-text searchCode (This is the column I need Query Editor to fill-in for me)
David is hereN/A (no match)
Tina is leaving6
I am Peter4
Peter, Tina and Cat are here3; 4; 6 (more than one match)
Where is Bob?1
Ann and Bob are my friends1; 2 (more than one match)
Please catch-up with Ann

2 (did not confuse between Cat and Catup)

 

How to do it with Query Editor?

 

Many thanks!

 

Best regards,

Catherine

11 Replies

  • edhans's avatar
    edhans
    Community Champion

    This code will do it. (NOTE: The code using List.Generate is cleaner which is a but further down in this thread.)

     

     

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("RY07DgIxDESvMkq9ewbEp6GjQKKIUpiNIZZYg5LsIm5PYgq68fP42Xt3oFUipCBxZhcG786i1MGD20rvxo6gGSeunG20NMCapBF7qqDMf8elpy7ZPa8bI1tVqzZg1fmDWxbWWH7G9q0wJqpTGpcX3lIT2o0L4Qs=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Full-text search" = _t]),
        #"Simple Replace" =
            Table.AddColumn(
                Source,
                "New Col",
                each
                try
                    Text.RemoveRange(
                            List.Accumulate(
                                {0..Table.RowCount(Text)-1},
                                [#"Full-text search"],
                                (state, current) => 
                                    let
                                        varCurrentName = Text[Name]{current},
                                        varCurrentNumber = ", " & Text[Code]{current} 
                                    in
                                    if Text.Contains(state, varCurrentName) = true
                                        then Text.Combine({state & varCurrentNumber})
                                        else state
                            ),
                            0,
                            Text.Length([#"Full-text search"]) + 2
                    )
                otherwise ""
            )
    in
        #"Simple Replace"

     

     

    For it to work, you need a query called "Text" that is the following code:

     

     

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WcspPUtJRMlSK1YlWcszLA7KNwGznxBIg2xjMDkgtSS0C8kzAPN/EokogxxTMCcnMSwRyzJRiYwE=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Name = _t, Code = _t])
    in
        Source

     

     

    The Text query looks like this:

    THe main query - the big block of code at the top, starts with just the first column, and ends with the 2nd column being added:

    See below to put that code in your Query editor to play with.

    1) In Power Query, select New Source, then Blank Query
    2) On the Home ribbon, select "Advanced Editor" button
    3) Remove everything you see, then paste the M code I've given you in that box.
    4) Press Done
    5) See this article if you need help using this M code in your model.

     

    • edhans's avatar
      edhans
      Community Champion

      HotChilli - you might be able to clean this up, or ImkeF if you could assist. My solution works, but it goes the long way.
      This code:

       

                          List.Accumulate(
                              {0..Table.RowCount(Text)-1},
                              [#"Full-text search"],
                              (state, current) => 
                                  let
                                      varCurrentName = Text[Name]{current},
                                      varCurrentNumber = "~" & Text[Code]{current} 
                                  in
                                  if Text.Contains(state, varCurrentName) = true
                                      then Text.Combine({state & varCurrentNumber})
                                      else state
                          )

       

      What it does is takes the source text column, so "Tina is Leaving" for the second record (where there is a match), and using Text.Combine() appends the relevant code number with a comma for easy parsing later. "Tina is Leaving, 6" For more matches it just keeps adding a comma and the code. Then I remove the original text with the Text.RemoveRange function.

       

      What I wanted to do was start with nothing, and just accumulate the codes, but if Text.Combine() didn't have the state in it, nothing would accumulate. It returned either 6 or null.


      So, how would I accumulate just the codes without having to have the initial text (state) as I've defined it?

      • HotChilli's avatar
        HotChilli
        Community Champion

        Hello edhans , good stuff (as always, you're a great contributor to the forum).

        Personally, I will always try to persuade people to leave data in a 'relational' form when they want to put lists within columns.  They can always use CONCATENATEX to display the list in a visual.  

         

        Here's the M (copied from an old Imke post) for anyone who is interested

        Table.AddColumn(#"Changed Type", "NewCol",  (Other) => Table.SelectRows(TableQ, each Text.Contains(Other[#"Full-text search"],[Name])))

         

        Anonymous Do these posts help with your question?

         

         

  • HotChilli's avatar
    HotChilli
    Community Champion

    I was able to do it with a 'Merge as New' on the 2 tables.

    Select Name and Full Text Search column and use Fuzzy Match to perform the join.

    Options (Threshold 0.5 and Ignore case)-> Others may work, these were just the second ones i tried

     

    I think there's probably a solution without the merge and maybe using Text.Contains function.  If I get time tomorrow, I might try it. 

    • Anonymous's avatar
      Anonymous
      Not applicable

      now I could get my hands on a pc.
      here is the code I was talking about:

       

       

       

       

      let
          Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("RY07DgIxDESvMkq9ewbEp6GjQKKIUpiNIZZYg5LsIm5PYgq68fP42Xt3oFUipCBxZhcG786i1MGD20rvxo6gGSeunG20NMCapBF7qqDMf8elpy7ZPa8bI1tVqzZg1fmDWxbWWH7G9q0wJqpTGpcX3lIT2o0L4Qs=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Full-text search" = _t]),
      
      
          #"Added Custom" = Table.AddColumn(Source, "CodeName", each Text.Combine(List.Transform(Text.SplitAny([#"Full-text search"], Text.Combine({" ".."@"})), each Record.FieldOrDefault(Dict, _)),";"))
      
      in
          #"Added Custom"

       

       

       

       

       

      • edhans's avatar
        edhans
        Community Champion

        Anonymous - the Text.SplitAny is nice. I can redo this without the more complex List.Generate using this.

        let
            Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("RY07DgIxDESvMkq9ewbEp6GjQKKIUpiNIZZYg5LsIm5PYgq68fP42Xt3oFUipCBxZhcG786i1MGD20rvxo6gGSeunG20NMCapBF7qqDMf8elpy7ZPa8bI1tVqzZg1fmDWxbWWH7G9q0wJqpTGpcX3lIT2o0L4Qs=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Full-text search" = _t]),
            #"Added Custom" = 
                Table.AddColumn(
                    Source, 
                    "CodeName", 
                    
                    each
                    let 
                        varCurrentText = [#"Full-text search"]
                    in
                    Text.Combine(
                        List.Transform(
                            Table.SelectRows(
                                Codes,
                                each List.Contains(
                                    List.Intersect({
                                        Text.SplitAny(varCurrentText, Text.Combine({" ".."@"})),
                                        Codes[Name]}
                                        ),
                                        [Name]
                                    )
                            )[Code],
                        Text.From
                        ), 
                    ", "
                    )
                )
        in
            #"Added Custom"

         However, I cannot get your Record.FieldOrDefault to work. Your code (using my Codes table name) fails:

        let
            Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("RY07DgIxDESvMkq9ewbEp6GjQKKIUpiNIZZYg5LsIm5PYgq68fP42Xt3oFUipCBxZhcG786i1MGD20rvxo6gGSeunG20NMCapBF7qqDMf8elpy7ZPa8bI1tVqzZg1fmDWxbWWH7G9q0wJqpTGpcX3lIT2o0L4Qs=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Full-text search" = _t]),
            #"Added Custom" = 
                Table.AddColumn(
                    Source, 
                    "CodeName", 
                    each 
                        Text.Combine(
                            List.Transform(
                                Text.SplitAny(
                                    [#"Full-text search"], 
                                    Text.Combine({" ".."@"})
                                    ), 
                                    each Record.FieldOrDefault(Codes, _)
                                ),
                            ";"
                            )
                    )
        
        in
            #"Added Custom"

        Which makes sense as Codes is a table, not a record.

        How are you getting it to either select the records you want, or cycle through them?  I can select using Table.SelectRows, or cycle through them with List.Generate. 

        Record.FieldOrDefault only works with a single record it appears.

         

         

  • edhans's avatar
    edhans
    Community Champion

    Anonymous - Got it in a cleaner step.

     

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("RY07DgIxDESvMkq9ewbEp6GjQKKIUpiNIZZYg5LsIm5PYgq68fP42Xt3oFUipCBxZhcG786i1MGD20rvxo6gGSeunG20NMCapBF7qqDMf8elpy7ZPa8bI1tVqzZg1fmDWxbWWH7G9q0wJqpTGpcX3lIT2o0L4Qs=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Full-text search" = _t]),
        #"List Generate" = 
            Table.AddColumn(
                Source,
                "Generate",
                each 
                    let
                        varTotalRecords = Table.RowCount(Text),
                        varCurrentText = [#"Full-text search"]
                    in
                Text.Combine(
                    List.Transform(
                        List.Generate(
                            () => [x = -1, y = {}],
                            each [x] < varTotalRecords,
                            each [
                                    y = 
                                        if Text.Contains(varCurrentText, Text[Name]{x})
                                        then [y] & {Text[Code]{x}}
                                        else [y],
                                    x = [x] + 1
                                ]
                            ){varTotalRecords}[y],
                        Text.From),
                        ","
                    )
                )
    in
        #"List Generate"

     

     

    It uses List.Generate to cycle through each record and build a list of the hit codes. The table returned is:

    My full PBIX file is here with both List Accumulate and List Generate.

  • edhans's avatar
    edhans
    Community Champion

    Anonymous - can you advise if any of the above solutions work for your needs and mark it (them) as a solution? 
    If none do, please reply back with clarifications so we can assist.