Forum Discussion
Replicating regex in Power BI
- Anonymous2 years ago
I decided to use Power Query M's `Text.Replace` function to solve this. This is not a very efficient solution and significantly increases the time for filtering the rtf text compared to my initial Python solution.
In the code below, I create a list of the common rtf tags I found, and use M's `List.Accumulate` together with `Text.Replace` to replace all the items in the list from each row.
#"Remove RTF tags" = Table.AddColumn( Notes, "Edited_Note", each let note = [Note], bList = List.Transform(List.Numbers(0, 24), each Text.Combine({"\b", Text.From(_)})), //create a list of \b0 - \b24 fsList = List.Transform(List.Numbers(0, 24), each Text.Combine({"\fs", Text.From(_)})), //create a list of \fs0 - \fs24 redList = List.Transform(List.Numbers(0, 255), each Text.Combine({"\red", Text.From(_)})), //create a list of \red0 - \red255 blueList = List.Transform(List.Numbers(0, 255), each Text.Combine({"\blue", Text.From(_)})), //create a list of \blue0 - \blue255 greenList = List.Transform( List.Numbers(0, 255), each Text.Combine({"\green", Text.From(_)}) ), //create a list of \green0 - \green255 items = { "{\rtf1", "\ansicpg1252", "\ansi", "\deff0", "\deflang1033", "{\fonttbl", "{\f0", "\fnil", "\fcharset0", "\viewkind4", "\uc1", "\pard", "\f0", "\par}", "\par", "\nouicompat", "\froman", "\fprq2", " Arial;}}", "Cambria;}", "{\f1", "\f1", "\widctlpar", "{\*\generator Riched20 10.0.19041}", "\qj", "\nowidctlpar8", "\lang", "{\*\generator Riched20", "#(lf)}", "\tab", "{\colortbl ;", "\red0", "\green0", "\blue0", "\cf1", "\cf0", "\fbidis", "\fswiss", "Calibri;}", "\ltrpar", "Arial;}", ";}", "\i0", "\i ", "\b ", "\fcharset1 Segoe UI Symbol}" }, combinedList = List.Combine( { items, List.Reverse(fsList), List.Reverse(redList), List.Reverse(blueList), List.Reverse(greenList), List.Reverse(bList), {"\b", "\i"} } ) in List.Accumulate(combinedList, Note, (state, current) => Text.Replace(state, current, "")) ), #"Change RDbl Quotes" = Table.TransformColumns( #"Remove RTF tags", { {"Edited_Note", each List.Accumulate({"\rdblquote"},_, (state, current) => Text.Replace(state, current,""""))} //replace "\rdblquote with a " } ), #"Change LSngl Quotes" = Table.TransformColumns( #"Change RDbl Quotes", { {"Edited_Note", each List.Accumulate({"\rquote"},_, (state, current) => Text.Replace(state, current,"'"))} //replace "\rquote with a ' } ), #"Trimmed Text1" = Table.TransformColumns(#"Remove RTF tags",{{"Note", Text.Trim, type text}}) //removes whitespaces created at the beginning and end of the text.
I decided to use Power Query M's `Text.Replace` function to solve this. This is not a very efficient solution and significantly increases the time for filtering the rtf text compared to my initial Python solution.
In the code below, I create a list of the common rtf tags I found, and use M's `List.Accumulate` together with `Text.Replace` to replace all the items in the list from each row.
#"Remove RTF tags" = Table.AddColumn(
Notes,
"Edited_Note",
each
let
note = [Note],
bList = List.Transform(List.Numbers(0, 24), each Text.Combine({"\b", Text.From(_)})), //create a list of \b0 - \b24
fsList = List.Transform(List.Numbers(0, 24), each Text.Combine({"\fs", Text.From(_)})), //create a list of \fs0 - \fs24
redList = List.Transform(List.Numbers(0, 255), each Text.Combine({"\red", Text.From(_)})), //create a list of \red0 - \red255
blueList = List.Transform(List.Numbers(0, 255), each Text.Combine({"\blue", Text.From(_)})), //create a list of \blue0 - \blue255
greenList = List.Transform(
List.Numbers(0, 255),
each Text.Combine({"\green", Text.From(_)})
), //create a list of \green0 - \green255
items = {
"{\rtf1",
"\ansicpg1252",
"\ansi",
"\deff0",
"\deflang1033",
"{\fonttbl",
"{\f0",
"\fnil",
"\fcharset0",
"\viewkind4",
"\uc1",
"\pard",
"\f0",
"\par}",
"\par",
"\nouicompat",
"\froman",
"\fprq2",
" Arial;}}",
"Cambria;}",
"{\f1",
"\f1",
"\widctlpar",
"{\*\generator Riched20 10.0.19041}",
"\qj",
"\nowidctlpar8",
"\lang",
"{\*\generator Riched20",
"#(lf)}",
"\tab",
"{\colortbl ;",
"\red0",
"\green0",
"\blue0",
"\cf1",
"\cf0",
"\fbidis",
"\fswiss",
"Calibri;}",
"\ltrpar",
"Arial;}",
";}",
"\i0",
"\i ",
"\b ",
"\fcharset1 Segoe UI Symbol}"
},
combinedList = List.Combine(
{
items,
List.Reverse(fsList),
List.Reverse(redList),
List.Reverse(blueList),
List.Reverse(greenList),
List.Reverse(bList),
{"\b", "\i"}
}
)
in
List.Accumulate(combinedList, Note, (state, current) => Text.Replace(state, current, ""))
),
#"Change RDbl Quotes" = Table.TransformColumns(
#"Remove RTF tags",
{
{"Edited_Note", each List.Accumulate({"\rdblquote"},_, (state, current) => Text.Replace(state, current,""""))} //replace "\rdblquote with a "
}
),
#"Change LSngl Quotes" = Table.TransformColumns(
#"Change RDbl Quotes",
{
{"Edited_Note", each List.Accumulate({"\rquote"},_, (state, current) => Text.Replace(state, current,"'"))} //replace "\rquote with a '
}
),
#"Trimmed Text1" = Table.TransformColumns(#"Remove RTF tags",{{"Note", Text.Trim, type text}}) //removes whitespaces created at the beginning and end of the text.
- archery2 years agoRegular Visitor
As for me, you should use re.gsub(), not re.sub(). re.sub() only deletes first occurrence, and re.gsub() deletes "globally" all
You may use https://regex101.com/ for testing you regex