Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago
Solved

Replicating regex in Power BI

Hello all,   I'm having trouble replicating some regex functionality using Power Query M.   I have a table with a single column of notes from an ODBC data source. The issue is that due to the wei...
  • Anonymous's avatar
    Anonymous
    2 years ago

    I decided to use Power Query M's `Text.Replace` function to solve this. This is not a very efficient solution and significantly increases the time for filtering the rtf text compared to my initial Python solution.

     

    In the code below, I create a list of the common rtf tags I found, and use M's `List.Accumulate` together with `Text.Replace` to replace all the items in the list from each row.

     

    #"Remove RTF tags" = Table.AddColumn(
            Notes, 
            "Edited_Note", 
            each 
            let
                note = [Note], 
                bList = List.Transform(List.Numbers(0, 24), each Text.Combine({"\b", Text.From(_)})),  //create a list of \b0 - \b24
                fsList = List.Transform(List.Numbers(0, 24), each Text.Combine({"\fs", Text.From(_)})),  //create a list of \fs0 - \fs24
                redList = List.Transform(List.Numbers(0, 255), each Text.Combine({"\red", Text.From(_)})),  //create a list of \red0 - \red255
                blueList = List.Transform(List.Numbers(0, 255), each Text.Combine({"\blue", Text.From(_)})),  //create a list of \blue0 - \blue255
                greenList = List.Transform(
                List.Numbers(0, 255), 
                each Text.Combine({"\green", Text.From(_)})
                ),  //create a list of \green0 - \green255
                items = {
                    "{\rtf1", 
                    "\ansicpg1252", 
                    "\ansi", 
                    "\deff0", 
                    "\deflang1033", 
                    "{\fonttbl", 
                    "{\f0", 
                    "\fnil", 
                    "\fcharset0", 
                    "\viewkind4", 
                    "\uc1", 
                    "\pard", 
                    "\f0", 
                    "\par}", 
                    "\par", 
                    "\nouicompat", 
                    "\froman", 
                    "\fprq2", 
                    " Arial;}}", 
                    "Cambria;}", 
                    "{\f1", 
                    "\f1", 
                    "\widctlpar", 
                    "{\*\generator Riched20 10.0.19041}", 
                    "\qj", 
                    "\nowidctlpar8", 
                    "\lang", 
                    "{\*\generator Riched20", 
                    "#(lf)}", 
                    "\tab", 
                    "{\colortbl ;", 
                    "\red0", 
                    "\green0", 
                    "\blue0",
                    "\cf1", 
                    "\cf0", 
                    "\fbidis", 
                    "\fswiss", 
                    "Calibri;}", 
                    "\ltrpar", 
                    "Arial;}",  
                    ";}", 
                    "\i0", 
                    "\i ",  
                    "\b ",
                    "\fcharset1 Segoe UI Symbol}"
                }, 
                combinedList = List.Combine(
                {
                    items, 
                    List.Reverse(fsList), 
                    List.Reverse(redList), 
                    List.Reverse(blueList), 
                    List.Reverse(greenList), 
                    List.Reverse(bList),
                    {"\b", "\i"}
                }
                )
            in
                List.Accumulate(combinedList, Note, (state, current) => Text.Replace(state, current, ""))
        ),
    
        #"Change RDbl Quotes" = Table.TransformColumns(
            #"Remove RTF tags", 
            {
                {"Edited_Note", each List.Accumulate({"\rdblquote"},_, (state, current) => Text.Replace(state, current,""""))} //replace "\rdblquote with a "
            }
        ),
    
        #"Change LSngl Quotes" = Table.TransformColumns(
            #"Change RDbl Quotes", 
            {
                {"Edited_Note", each List.Accumulate({"\rquote"},_, (state, current) => Text.Replace(state, current,"'"))} //replace "\rquote with a '
            }
        ),
        #"Trimmed Text1" = Table.TransformColumns(#"Remove RTF tags",{{"Note", Text.Trim, type text}}) //removes whitespaces created at the beginning and end of the text.