Forum Discussion

Data_Stylist's avatar
Data_Stylist
New Member
1 year ago
Solved

Dynamically skip or delete rows between specific values in power query

I am working with a bunch of merged tables containing redunandant null data, I still need to fill down some of the null data so I cannot outrighly delete all nulls. The best logic I came up with is to delete all rows starting from "Total:" and ending "Product Cost $/litre" in column PRODUCT_COST($/L) , reason being that there are several instances in the table and I need to be able to delete anwhere those two shows up and anything in between.

Please find the current state, and what I'm trying to achieve (future state) below.

Current State

Source.NameNamePRODUCT_COST($/L)
A.xlsxJan2.42
A.xlsxJan5.87
A.xlsxJan1.44
A.xlsxJan1.44
A.xlsxJan1.44
A.xlsxJannull
A.xlsxJan1.44
A.xlsxJannull
A.xlsxJanTotal:
A.xlsxJannull
A.xlsxJannull
A.xlsxJannull
A.xlsxJannull
A.xlsxJannull
A.xlsxJannull
A.xlsxJannull
A.xlsxJannull
A.xlsxFebProduct Cost $/litre
A.xlsxFebnull
A.xlsxFeb1.5
A.xlsxFebnull
A.xlsxFeb2.44
A.xlsxFeb5.26
A.xlsxFeb1.5
A.xlsxFebnull
A.xlsxFeb1.5
A.xlsxFebnull
A.xlsxFeb1.5
A.xlsxFebnull
A.xlsxFebTotal:
A.xlsxFebnull
A.xlsxFebnull
A.xlsxFebProduct Cost $/litre
A.xlsxFebnull
A.xlsxFeb5.12
A.xlsxFeb5.87
A.xlsxFeb4.47
A.xlsxFeb7.92


Transformed State

Source.NameNamePRODUCT_COST($/L)
A.xlsxJan2.42
A.xlsxJan5.87
A.xlsxJan1.44
A.xlsxJan1.44
A.xlsxJan1.44
A.xlsxJannull
A.xlsxJan1.44
A.xlsxJannull
A.xlsxFebnull
A.xlsxFeb1.5
A.xlsxFebnull
A.xlsxFeb2.44
A.xlsxFeb5.26
A.xlsxFeb1.5
A.xlsxFebnull
A.xlsxFeb1.5
A.xlsxFebnull
A.xlsxFeb1.5
A.xlsxFebnull
A.xlsxFebnull
A.xlsxFeb5.12
A.xlsxFeb5.87
A.xlsxFeb4.47
A.xlsxFeb7.92
General Comment
Need Help
Tips and Tricks
  • mromain's avatar
    mromain
    1 year ago

    Hello Data_Stylist,

    Here is a possible solution:

    let
    Source = #table(
        type table [Source.Name = text, Name = text, #"PRODUCT_COST($/L)" = text],
        {
            {"A.xlsx", "Jan", "2.42"},
            {"A.xlsx", "Jan", "5.87"},
            {"A.xlsx", "Jan", "1.44"},
            {"A.xlsx", "Jan", "1.44"},
            {"A.xlsx", "Jan", "1.44"},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", "1.44"},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", "Total:"},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Feb", "Product Cost $/litre"},
            {"A.xlsx", "Feb", null},
            {"A.xlsx", "Feb", "1.5"},
            {"A.xlsx", "Feb", null},
            {"A.xlsx", "Feb", "2.44"},
            {"A.xlsx", "Feb", "5.26"},
            {"A.xlsx", "Feb", "1.5"},
            {"A.xlsx", "Feb", null},
            {"A.xlsx", "Feb", "1.5"},
            {"A.xlsx", "Feb", null},
            {"A.xlsx", "Feb", "1.5"},
            {"A.xlsx", "Feb", null},
            {"A.xlsx", "Feb", "Total:"},
            {"A.xlsx", "Feb", null},
            {"A.xlsx", "Feb", null},
            {"A.xlsx", "Feb", "Product Cost $/litre"},
            {"A.xlsx", "Feb", null},
            {"A.xlsx", "Feb", "5.12"},
            {"A.xlsx", "Feb", "5.87"},
            {"A.xlsx", "Feb", "4.47"},
            {"A.xlsx", "Feb", "7.92"}
        }),
    ListMaches = let findMatches = (t) => List.PositionOf(Source[#"PRODUCT_COST($/L)"], t, Occurrence.All) in List.Zip({findMatches("Total:"), findMatches("Product Cost $/litre")}),
    RemoveRows = List.Accumulate(List.Reverse(ListMaches), Source, (s, c)=> Table.RemoveRows(s, c{0}, c{1} - c{0} + 1))
in
    RemoveRows

9 Replies

  • KNP's avatar
    KNP
    Super User
    1 year ago

    I don't know what other columns you may have but could you filter out 'Total:' and 'Product Cost $/litre' and then group? See code below.

     

    let
    Source = 
        Table.FromRows(
            Json.Document(
                Binary.Decompress(
                    Binary.FromText(
                        "i45WctSryCmuUNJR8krMA5JGeiZGSrE6GOKmehbm2MQN9UxMqCGeV5qTQw31IfkliTlWpOgY/OJuqUlAMqAoP6U0uUTBOb+4REFFPyezpCgVmzpc+g31TElRboQR8BBxUz0jMyoYT1vl2FIBfh20CnVTPUMj7OLo+QkibqJnglXcXM8SaE4sAA=="
                        , BinaryEncoding.Base64
                    )
                    , Compression.Deflate
                )
            )
            , let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [SourceName = _t, Name = _t, PRODUCT_COST_per_L = _t]
        )
    , ChangeType = 
        Table.TransformColumnTypes(
            Source
            , {
                { "SourceName", type text }
                , { "Name", type text }
                , { "PRODUCT_COST_per_L", type text }
            }
        )
    , FilterRows = 
        Table.SelectRows(
            ChangeType
            , each (
                [PRODUCT_COST_per_L] <> "Product Cost $/litre"
                and [PRODUCT_COST_per_L] <> "Total:"
            )
        )
    , ChangeType1 = 
        Table.TransformColumnTypes(
            FilterRows
            , {
                { "PRODUCT_COST_per_L", type number }
            }
        )
    , GroupRows = 
        Table.Group(
            ChangeType1
            , { "SourceName", "Name" }
            , {
                { "Count", each List.Sum([PRODUCT_COST_per_L]), type nullable text }
            }
        )
in
    GroupRows

     

    • Data_Stylist's avatar
      Data_Stylist
      New Member
      1 year ago

      Thanks KNP Unfortunately the solution took out the data I need as well (see screenshot below). I only need to remove rows from "Total:" and "Product Cost $/litre" while keeping everything else.

      Your solution


      Desired Solution

       

      • KNP's avatar
        KNP
        Super User
        1 year ago

        Yeah, I wasn't sure if this was the complete data or just what you're able to share.

        I thought you may have other columns to group on to allow you to keep the individual rows.

         

        At any point in the process, do you have more/other columns available that could be used for filtering?

         

  • mromain's avatar
    mromain
    Regular Visitor
    1 year ago

    Hello Data_Stylist,

    Here is a possible solution:

    let
    Source = #table(
        type table [Source.Name = text, Name = text, #"PRODUCT_COST($/L)" = text],
        {
            {"A.xlsx", "Jan", "2.42"},
            {"A.xlsx", "Jan", "5.87"},
            {"A.xlsx", "Jan", "1.44"},
            {"A.xlsx", "Jan", "1.44"},
            {"A.xlsx", "Jan", "1.44"},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", "1.44"},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", "Total:"},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Jan", null},
            {"A.xlsx", "Feb", "Product Cost $/litre"},
            {"A.xlsx", "Feb", null},
            {"A.xlsx", "Feb", "1.5"},
            {"A.xlsx", "Feb", null},
            {"A.xlsx", "Feb", "2.44"},
            {"A.xlsx", "Feb", "5.26"},
            {"A.xlsx", "Feb", "1.5"},
            {"A.xlsx", "Feb", null},
            {"A.xlsx", "Feb", "1.5"},
            {"A.xlsx", "Feb", null},
            {"A.xlsx", "Feb", "1.5"},
            {"A.xlsx", "Feb", null},
            {"A.xlsx", "Feb", "Total:"},
            {"A.xlsx", "Feb", null},
            {"A.xlsx", "Feb", null},
            {"A.xlsx", "Feb", "Product Cost $/litre"},
            {"A.xlsx", "Feb", null},
            {"A.xlsx", "Feb", "5.12"},
            {"A.xlsx", "Feb", "5.87"},
            {"A.xlsx", "Feb", "4.47"},
            {"A.xlsx", "Feb", "7.92"}
        }),
    ListMaches = let findMatches = (t) => List.PositionOf(Source[#"PRODUCT_COST($/L)"], t, Occurrence.All) in List.Zip({findMatches("Total:"), findMatches("Product Cost $/litre")}),
    RemoveRows = List.Accumulate(List.Reverse(ListMaches), Source, (s, c)=> Table.RemoveRows(s, c{0}, c{1} - c{0} + 1))
in
    RemoveRows
    • Data_Stylist's avatar
      Data_Stylist
      New Member
      1 year ago

      Thanks mromain The solution worked for the sample data I provided, but when applied to the actual data which contains thousands of row, I got the range error below.

       

      the first step (ListMaches) of extracting list that matched the appropriate conditions worked well, the extraction step is where there seems to be an issue. Any fix?

      Expression.Error: The 'count' argument is out of range.
      Details:
      -5