Forum Discussion

Neg's avatar
Neg
New Member
2 years ago
Solved

Cleaning Data with inconsistent decimal separators

Hi there everyone, I am trying to process a set of data that contains the gross weight of shipments, and its driving me nuts, due to how bad and inconsistent the data is. I get values such as below:...
  • ronrsnfld's avatar
    2 years ago

    This may work, if your differentiation is that a token followed by two (or one) digits is the decimal token:

     

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMtQxMjbRMzVQyE5XitWJVgIThnpAQR1Tb2QhI718YwVkkcSCgqL8Cj0FQx1TFM0GOgqGRpjGQU2BMYBCUIYhQo2CtgLQWjOl2FgA", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t]),
        #"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", type text}}),
        #"Removed Blank Rows" = Table.SelectRows(#"Changed Type", 
            each not List.IsEmpty(List.RemoveMatchingItems(Record.FieldValues(_), {"", null}))),
    
    //Clean the data
        #"Remove Leading/Trailing text" = Table.TransformColumns(#"Removed Blank Rows",
            {"Column1", each Text.Trim(_,{" ",".","a".."z","A".."Z"}), type text}),
        #"'o' to '0'"=Table.ReplaceValue(#"Remove Leading/Trailing text","o","0",Replacer.ReplaceText,{"Column1"}),
        #"Remove Spaces" =Table.ReplaceValue(#"'o' to '0'"," ","",Replacer.ReplaceText,{"Column1"}),
    
    //Convert the number based on algorithm that
    // if there are one or two digits after the `dot` or `comma` then that token = decimal token
        #"Add Converted" = Table.AddColumn(#"Remove Spaces", "Converted Number", each 
          let 
            #"Split Plus" = Text.Split([Column1],"+"),
            #"Split Dot" = List.Combine(List.Transform(#"Split Plus", each Text.Split(_,"."))),
            #"Split Comma" = List.Combine(List.Transform(#"Split Plus", each Text.Split(_,","))),
    
            #"Decimal" = if (Text.Length(#"Split Dot"{1}?) <= 2)??false then "."
                else if ((Text.Length(#"Split Comma"{1}?)??0) <= 2)??false then ","
                else null,
           
            #"Convert" = if #"Decimal" = "."
              then List.Transform(#"Split Plus", each Number.From(_, "en-US"))
              else List.Transform(#"Split Plus", each Number.FromText(_,"en-DE"))
          in 
            List.Sum(#"Convert"), type number)
    in
        #"Add Converted"

     

    See if the transform is what you expect

     

    Original

    Transformed