Forum Discussion

vineshparekh's avatar
vineshparekh
Helper I
2 years ago

Help required to separate the text

Hi Experts,

 

I have the following messy data and unorganized column extracted from PDF which requires separating the data in [ ] to another column.

For example, 

AF-20KC - Airframe 20,000 FC Structural Check - [SN: 556] -

Result: AF-20KC - Airframe 20,000 FC Structural Check in column 1 and 556 in another column.

Also, I have these rows in which I don't want -Correction *** words. I need only upto SN: 556 and then separating into another column.

AF-192M - Airframe 192 Month Structural Check - [SN: 556] -Correction from period 01-Dec-2023 to 31-Dec-2023

 

There are some rows which don't have space for the SN like below. I would like to separate SN from that row too.

 

EN-PR - EN-PR - Engine Full Performance Restoration
P783272]

 

The complete data is below:

Description
AF-96M - Airframe 96 Month Structural Check - [SN: 555]
AF-BC - Airframe Basic Check - [SN: 555]
AF-20KC - Airframe 20,000 FC Structural Check - [SN: 555]
LG-OH - LG-OH - Landing Gear Leg Overhaul
AF-192M - Airframe 192 Month Structural Check - [SN: 555]
AP-OH - AP-OH - APU Overhaul
EN-PR - EN-PR - Engine Full Performance Restoration
P783243]
EN-LP - EN-LP - Engine LLP Replacement
EN-PR - EN-PR - Engine Full Performance Restoration
P783245]
LG-OH - LG-OH - Landing Gear Leg Overhaul - [SN: 556] -Correction from period 01-Dec-2023 to 31-Dec-2023
AF-192M - Airframe 192 Month Structural Check - [SN: 556] -Correction from period 01-Dec-2023 to 31-Dec-2023
AF-BC - Airframe Basic Check - [SN: 556] - Correction from
period 01-Dec-2023 to 31-Dec-2023
AF-96M - Airframe 96 Month Structural Check - [SN: 556] -Correction from period 01-Dec-2023 to 31-Dec-2023
AF-20KC - Airframe 20,000 FC Structural Check - [SN: 556] -
Correction from period 01-Dec-2023 to 31-Dec-2023
AP-OH - AP-OH - APU Overhaul - [SN: PWC-ZD0103] - Correction
from period 01-Dec-2023 to 31-Dec-2023
- [SN: EN-LP - EN-LP - Engine LLP Replacement
P783246]Correction from period 01-Dec-2023 to 31-Dec-2023
- [SN: EN-PR - EN-PR - Engine Full Performance Restoration
P783246] - Correction from period 01-Dec-2023 to 31-Dec-2023
- [SN: P783255] EN-LP - EN-LP - Engine LLP Replacement
Correction from period 01-Dec-2023 to 31-Dec-2023
- [SN: EN-PR - EN-PR - Engine Full Performance Restoration
P783255] - Correction from period 01-Dec-2023 to 31-Dec-2023
AF-96M - Airframe 96 Month Structural Check - [SN: 557]
AF-192M - Airframe 192 Month Structural Check - [SN: 557]
AF-20KC - Airframe 20,000 FC Structural Check - [SN: 557]
AF-BC - Airframe Basic Check - [SN: 557]
EN-PR - EN-PR - Engine Full Performance Restoration
P783258]
EN-PR - EN-PR - Engine Full Performance Restoration
P783259]
AF-BC - Airframe Basic Check - [SN: 558]
AF-96M - Airframe 96 Month Structural Check - [SN: 558]
AF-192M - Airframe 192 Month Structural Check - [SN: 558]
AF-20KC - Airframe 20,000 FC Structural Check - [SN: 558]
EN-PR - EN-PR - Engine Full Performance Restoration
P783257]
EN-PR - EN-PR - Engine Full Performance Restoration
P783260]
AF-96M - Airframe 96 Month Structural Check - [SN: 561]
AF-192M - Airframe 192 Month Structural Check - [SN: 561]
AF-20KC - Airframe 20,000 FC Structural Check - [SN: 561]
AF-BC - Airframe Basic Check - [SN: 561]
EN-PR - EN-PR - Engine Full Performance Restoration
P783274]
EN-PR - EN-PR - Engine Full Performance Restoration
P783277]
AF-192M - Airframe 192 Month Structural Check - [SN: 559]
AF-96M - Airframe 96 Month Structural Check - [SN: 559]
AF-192M - Airframe 192 Month Structural Check - [SN: 556]
AF-BC - Airframe Basic Check - [SN: 556]
AF-96M - Airframe 96 Month Structural Check - [SN: 556]
AF-20KC - Airframe 20,000 FC Structural Check - [SN: 556]
EN-PR - EN-PR - Engine Full Performance Restoration
P783246]
EN-PR - EN-PR - Engine Full Performance Restoration
P783255]
AF-96M - Airframe 96 Month Structural Check - [SN: 560]
AF-192M - Airframe 192 Month Structural Check - [SN: 560]
AF-20KC - Airframe 20,000 FC Structural Check - [SN: 560]
AF-BC - Airframe Basic Check - [SN: 560]
EN-PR - EN-PR - Engine Full Performance Restoration
P783247]
EN-PR - EN-PR - Engine Full Performance Restoration
P783271]
AF-20KC - Airframe 20,000 FC Structural Check - [SN: 559]
AF-BC - Airframe Basic Check - [SN: 559]
EN-PR - EN-PR - Engine Full Performance Restoration
P783272]
EN-PR - EN-PR - Engine Full Performance Restoration
P783273] @

10 Replies

  • dufoq3's avatar
    dufoq3
    Community Champion

    Hi vineshparekh,

    check whether this is what you need.

     

    Result (top 20 rows)

     

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("xVdbb4IwFP4rJz6PpIIU2NMUp0vmhWiWJTN9aLAqGYKpuN8/hCm4S7ScJnuhqfR837l9p7hYtLoDw6NjMKAbyZXkWwEehXGaZBuYZ/IQZgfJY/A3InzPDy3mk3uwbZu12F1h2/Prpj2+j8K/D5vk+eK4Se4IITDwr1ONhsb0Kf/1vPJkGSVrGAouYSTWMP0QcsMP8Ymr7ZkXUeX7W8PixfNxYgSz/N15TdZRImBwiGMIhFylcsuTUMBM7LNU8ixKk8IucFzL7FjsBDIKSpByLUFG+WYmdjEPxVYkmQY+1TRVQVMGhp9KKcIjJKxkuoWdkFG6BNI2+iLMy2ZakKVgVVtMllGEN/TbER++ERTmN5OoCwIVUxNZHAkL+4asQdkf1fryozWCV99465M2sS4TWgAocH3BKUihbGnKmsVW8SH09FsTqfEXOPlIUQn8/wI+Oto04EaKcRhmhjioO8VRub4chh/PtqsDxFNx22XNa+OiauOiaqMlUzpqRknzDNI2JoM16wYZrFlfb5PTYVSmnI4OENxA8BDt7qGYqYoqKcJNitIV1VCjjg6Q2v8BdV0RlK4ISldERVdER7p1jDEHNUzU7hxPh7+mDpD8k/Whxdgn", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Description = _t]),
        AddedIndex = Table.AddIndexColumn(Source, "Index", 0, 1, Int64.Type),
        Ad_GroupHelper = Table.AddColumn(AddedIndex, "GroupHelper", each if (try Text.Range([Description], 2, 1) otherwise null) = "-" then [Index] else null, type text),
        FilledDown = Table.FillDown(Ad_GroupHelper,{"GroupHelper"}),
        GroupedRows = Table.Group(FilledDown, {"GroupHelper"}, {{"Description", each Text.Combine([Description], " "), type text}}),
        ExtractedTextBeforeDelimiter = Table.TransformColumns(GroupedRows, {{"Description", each
            [ a = Text.PositionOf(_, "]", Occurrence.First) +1,
              b = Text.Range(_, 0, if a > 0 then a else null )
            ][b], type text}}),
        TrimmedText = Table.TransformColumns(ExtractedTextBeforeDelimiter,{{"Description", Text.Trim, type text}}),
        Ad_SN = Table.AddColumn(TrimmedText, "SN", each 
            [ a = Text.PositionOf([Description], "[SN") +1,
              b = Text.PositionOf([Description], " ", Occurrence.Last),
              c = Text.Range([Description], if a > 0 then a else if Text.Contains([Description], "]") then b else Text.Length([Description]))
            ][c], type text),
        CleanDescription = Table.ReplaceValue(Ad_SN,
            each [SN],
            null,
            (x,y,z)=> if y = "" then x else Text.TrimEnd(Text.Replace(x, y, ""), {"[", " ", "-"}),
            {"Description"}),
        CleanSN = Table.ReplaceValue(CleanDescription,
            null,
            null,
            (x,y,z)=> if x = "" then null else Text.Replace(Text.Trim(x, {" ", "]"}), "SN: ", ""),
            {"SN"}),
        RemovedGroupHelper = Table.RemoveColumns(CleanSN,{"GroupHelper"})
    in
        RemovedGroupHelper

     

    • vineshparekh's avatar
      vineshparekh
      Helper I

      Hi dufoq3 

      Thank you for the response! 

      I am getting this error when I use the M code you provided. This is coming from Ad_GroupHelper step.

      Expression.Error: The 'offset' argument is out of range.

  • dufoq3 Thanks! It works now. Just last thing!

    "Restoration" word for the Engine Full Performance is going into the second column as well. Whereas, I would like it to remain in the first column.

    EN-PR - EN-PR - Engine Full PerformanceRestoration
    P783243
    • dufoq3's avatar
      dufoq3
      Community Champion

      There is nothing like that in sample data. You have to provide new sample data with this issue.

      • vineshparekh's avatar
        vineshparekh
        Helper I

        dufoq3 

        Sample data does have those rows. For example, if you search this text in Bold in the dataset, you will find the related row. I am also providing the sample data again.

        EN-PR - EN-PR - Engine Full Performance Restoration
        P783277]

         

        One additional question.

        The sample data I had provided had only one column but in my actual data set, I have multiple columns which I want to populate in the final step. Is it the group by where the column names have to be added to show them as well? Please review my sample data.

        DescriptionUtilisationColumn4RateAmount
        AF-96M - Airframe 96 Month Structural Check - [SN: 555]1.00MTH      702.00         702.00
        AF-BC - Airframe Basic Check - [SN: 555]241.20FH         37.00     8,924.40
        AF-20KC - Airframe 20,000 FC Structural Check - [SN: 555]95.00FC           4.40         418.00
        LG-OH - LG-OH - Landing Gear Leg Overhaul1.00MTH   2,652.00     2,652.00
        AF-192M - Airframe 192 Month Structural Check - [SN: 555]1.00MTH      610.00         610.00
        AP-OH - AP-OH - APU Overhaul95.10APUH         25.00     2,377.50
        EN-PR - EN-PR - Engine Full Performance Restoration
        P783243]
        241.20FH      135.00   32,562.00
        EN-LP - EN-LP - Engine LLP Replacement95.00FC      162.08   15,397.60
        EN-PR - EN-PR - Engine Full Performance Restoration
        P783245]
        241.20FH      135.00   32,562.00
        LG-OH - LG-OH - Landing Gear Leg Overhaul - [SN: 556] -Correction from period 01-Dec-2023 to 31-Dec-20231.00    2,652.00     2,652.00
        AF-192M - Airframe 192 Month Structural Check - [SN: 556] -Correction from period 01-Dec-2023 to 31-Dec-20231.00       610.00         610.00
        AF-BC - Airframe Basic Check - [SN: 556] - Correction from
        period 01-Dec-2023 to 31-Dec-2023
        260.23          37.00     9,628.51
        AF-96M - Airframe 96 Month Structural Check - [SN: 556] -Correction from period 01-Dec-2023 to 31-Dec-20231.00       702.00         702.00
        AF-20KC - Airframe 20,000 FC Structural Check - [SN: 556] -99.00            4.40         435.60
        Correction from period 01-Dec-2023 to 31-Dec-2023    
        AP-OH - AP-OH - APU Overhaul - [SN: PWC-ZD0103] - Correction
        from period 01-Dec-2023 to 31-Dec-2023
        90.30          25.00     2,257.50
        - [SN: EN-LP - EN-LP - Engine LLP Replacement
        P783246]Correction from period 01-Dec-2023 to 31-Dec-2023
        99.00       162.08   16,045.92
        - [SN: EN-PR - EN-PR - Engine Full Performance Restoration
        P783246] - Correction from period 01-Dec-2023 to 31-Dec-2023
        260.23       135.00   35,131.05
        - [SN: P783255] EN-LP - EN-LP - Engine LLP Replacement
        Correction from period 01-Dec-2023 to 31-Dec-2023
        99.00       162.08   16,045.92
        - [SN: EN-PR - EN-PR - Engine Full Performance Restoration
        P783255] - Correction from period 01-Dec-2023 to 31-Dec-2023
        260.23       135.00   35,131.05
        AF-96M - Airframe 96 Month Structural Check - [SN: 557]0.90MTH      702.00         631.80
        AF-192M - Airframe 192 Month Structural Check - [SN: 557]0.90MTH      610.00         549.00
        AF-20KC - Airframe 20,000 FC Structural Check - [SN: 557]2.00FC           4.40             8.80
        AF-BC - Airframe Basic Check - [SN: 557]11.08FH         37.00         409.96
        EN-PR - EN-PR - Engine Full Performance Restoration
        P783258]
        6.90APUH         25.00         172.50
        EN-PR - EN-PR - Engine Full Performance Restoration
        P783259]
        11.08FH      135.00     1,495.80
        AF-BC - Airframe Basic Check - [SN: 558]2.00FC                -                    -  
        AF-96M - Airframe 96 Month Structural Check - [SN: 558]11.08FH      135.00     1,495.80
        AF-192M - Airframe 192 Month Structural Check - [SN: 558]2.00FC                -                    -  
        AF-20KC - Airframe 20,000 FC Structural Check - [SN: 558]0.90MTH   2,652.00     2,386.80
        EN-PR - EN-PR - Engine Full Performance Restoration
        P783257]
        110.67FH         37.00     4,094.79
        EN-PR - EN-PR - Engine Full Performance Restoration
        P783260]
        0.67MTH      702.00         470.34
        AF-96M - Airframe 96 Month Structural Check - [SN: 561]0.67MTH      610.00         408.70
        AF-192M - Airframe 192 Month Structural Check - [SN: 561]40.00FC           4.40         176.00
        AF-20KC - Airframe 20,000 FC Structural Check - [SN: 561]49.00APUH         25.00     1,225.00
        AF-BC - Airframe Basic Check - [SN: 561]110.67FH      135.00   14,940.45
        EN-PR - EN-PR - Engine Full Performance Restoration
        P783274]
        40.00FC                -                    -  
        EN-PR - EN-PR - Engine Full Performance Restoration
        P783277]
        110.67FH      135.00   14,940.45
        AF-192M - Airframe 192 Month Structural Check - [SN: 559]40.00FC                -                    -  
        AF-96M - Airframe 96 Month Structural Check - [SN: 559]0.67MTH   2,652.00     1,776.84
        AF-192M - Airframe 192 Month Structural Check - [SN: 556]0.43MTH      702.00         301.86
        AF-BC - Airframe Basic Check - [SN: 556]0.43MTH      610.00         262.30
        AF-96M - Airframe 96 Month Structural Check - [SN: 556]17.00FC           4.40           74.80
        AF-20KC - Airframe 20,000 FC Structural Check - [SN: 556]55.22FH         37.00     2,043.14
        EN-PR - EN-PR - Engine Full Performance Restoration
        P783246]
        29.20APUH         25.00         730.00
        EN-PR - EN-PR - Engine Full Performance Restoration
        P783255]
        55.22FH      135.00     7,454.70
        AF-96M - Airframe 96 Month Structural Check - [SN: 560]17.00FC                -                    -  
        AF-192M - Airframe 192 Month Structural Check - [SN: 560]55.22FH      135.00     7,454.70
        AF-20KC - Airframe 20,000 FC Structural Check - [SN: 560]17.00FC                -                    -  
        AF-BC - Airframe Basic Check - [SN: 560]0.43MTH   2,652.00     1,140.36
        EN-PR - EN-PR - Engine Full Performance Restoration
        P783247]
        0.10       610.00           61.00
        EN-PR - EN-PR - Engine Full Performance Restoration
        P783271]
        0.10       702.00           70.20
        AF-20KC - Airframe 20,000 FC Structural Check - [SN: 559]0.10    2,652.00         265.20
        AF-BC - Airframe Basic Check - [SN: 559]1.00MTH      723.06         723.06
        EN-PR - EN-PR - Engine Full Performance Restoration
        P783272]
        224.48FH         38.11     8,554.93
        EN-PR - EN-PR - Engine Full Performance Restoration
        P783273]
        85.00FC           4.53         385.05