Forum Discussion
Replicating regex in Power BI
- Anonymous2 years ago
I decided to use Power Query M's `Text.Replace` function to solve this. This is not a very efficient solution and significantly increases the time for filtering the rtf text compared to my initial Python solution.
In the code below, I create a list of the common rtf tags I found, and use M's `List.Accumulate` together with `Text.Replace` to replace all the items in the list from each row.
#"Remove RTF tags" = Table.AddColumn( Notes, "Edited_Note", each let note = [Note], bList = List.Transform(List.Numbers(0, 24), each Text.Combine({"\b", Text.From(_)})), //create a list of \b0 - \b24 fsList = List.Transform(List.Numbers(0, 24), each Text.Combine({"\fs", Text.From(_)})), //create a list of \fs0 - \fs24 redList = List.Transform(List.Numbers(0, 255), each Text.Combine({"\red", Text.From(_)})), //create a list of \red0 - \red255 blueList = List.Transform(List.Numbers(0, 255), each Text.Combine({"\blue", Text.From(_)})), //create a list of \blue0 - \blue255 greenList = List.Transform( List.Numbers(0, 255), each Text.Combine({"\green", Text.From(_)}) ), //create a list of \green0 - \green255 items = { "{\rtf1", "\ansicpg1252", "\ansi", "\deff0", "\deflang1033", "{\fonttbl", "{\f0", "\fnil", "\fcharset0", "\viewkind4", "\uc1", "\pard", "\f0", "\par}", "\par", "\nouicompat", "\froman", "\fprq2", " Arial;}}", "Cambria;}", "{\f1", "\f1", "\widctlpar", "{\*\generator Riched20 10.0.19041}", "\qj", "\nowidctlpar8", "\lang", "{\*\generator Riched20", "#(lf)}", "\tab", "{\colortbl ;", "\red0", "\green0", "\blue0", "\cf1", "\cf0", "\fbidis", "\fswiss", "Calibri;}", "\ltrpar", "Arial;}", ";}", "\i0", "\i ", "\b ", "\fcharset1 Segoe UI Symbol}" }, combinedList = List.Combine( { items, List.Reverse(fsList), List.Reverse(redList), List.Reverse(blueList), List.Reverse(greenList), List.Reverse(bList), {"\b", "\i"} } ) in List.Accumulate(combinedList, Note, (state, current) => Text.Replace(state, current, "")) ), #"Change RDbl Quotes" = Table.TransformColumns( #"Remove RTF tags", { {"Edited_Note", each List.Accumulate({"\rdblquote"},_, (state, current) => Text.Replace(state, current,""""))} //replace "\rdblquote with a " } ), #"Change LSngl Quotes" = Table.TransformColumns( #"Change RDbl Quotes", { {"Edited_Note", each List.Accumulate({"\rquote"},_, (state, current) => Text.Replace(state, current,"'"))} //replace "\rquote with a ' } ), #"Trimmed Text1" = Table.TransformColumns(#"Remove RTF tags",{{"Note", Text.Trim, type text}}) //removes whitespaces created at the beginning and end of the text.
I decided to use Power Query M's `Text.Replace` function to solve this. This is not a very efficient solution and significantly increases the time for filtering the rtf text compared to my initial Python solution.
In the code below, I create a list of the common rtf tags I found, and use M's `List.Accumulate` together with `Text.Replace` to replace all the items in the list from each row.
#"Remove RTF tags" = Table.AddColumn(
Notes,
"Edited_Note",
each
let
note = [Note],
bList = List.Transform(List.Numbers(0, 24), each Text.Combine({"\b", Text.From(_)})), //create a list of \b0 - \b24
fsList = List.Transform(List.Numbers(0, 24), each Text.Combine({"\fs", Text.From(_)})), //create a list of \fs0 - \fs24
redList = List.Transform(List.Numbers(0, 255), each Text.Combine({"\red", Text.From(_)})), //create a list of \red0 - \red255
blueList = List.Transform(List.Numbers(0, 255), each Text.Combine({"\blue", Text.From(_)})), //create a list of \blue0 - \blue255
greenList = List.Transform(
List.Numbers(0, 255),
each Text.Combine({"\green", Text.From(_)})
), //create a list of \green0 - \green255
items = {
"{\rtf1",
"\ansicpg1252",
"\ansi",
"\deff0",
"\deflang1033",
"{\fonttbl",
"{\f0",
"\fnil",
"\fcharset0",
"\viewkind4",
"\uc1",
"\pard",
"\f0",
"\par}",
"\par",
"\nouicompat",
"\froman",
"\fprq2",
" Arial;}}",
"Cambria;}",
"{\f1",
"\f1",
"\widctlpar",
"{\*\generator Riched20 10.0.19041}",
"\qj",
"\nowidctlpar8",
"\lang",
"{\*\generator Riched20",
"#(lf)}",
"\tab",
"{\colortbl ;",
"\red0",
"\green0",
"\blue0",
"\cf1",
"\cf0",
"\fbidis",
"\fswiss",
"Calibri;}",
"\ltrpar",
"Arial;}",
";}",
"\i0",
"\i ",
"\b ",
"\fcharset1 Segoe UI Symbol}"
},
combinedList = List.Combine(
{
items,
List.Reverse(fsList),
List.Reverse(redList),
List.Reverse(blueList),
List.Reverse(greenList),
List.Reverse(bList),
{"\b", "\i"}
}
)
in
List.Accumulate(combinedList, Note, (state, current) => Text.Replace(state, current, ""))
),
#"Change RDbl Quotes" = Table.TransformColumns(
#"Remove RTF tags",
{
{"Edited_Note", each List.Accumulate({"\rdblquote"},_, (state, current) => Text.Replace(state, current,""""))} //replace "\rdblquote with a "
}
),
#"Change LSngl Quotes" = Table.TransformColumns(
#"Change RDbl Quotes",
{
{"Edited_Note", each List.Accumulate({"\rquote"},_, (state, current) => Text.Replace(state, current,"'"))} //replace "\rquote with a '
}
),
#"Trimmed Text1" = Table.TransformColumns(#"Remove RTF tags",{{"Note", Text.Trim, type text}}) //removes whitespaces created at the beginning and end of the text.
As for me, you should use re.gsub(), not re.sub(). re.sub() only deletes first occurrence, and re.gsub() deletes "globally" all
You may use https://regex101.com/ for testing you regex