Forum Discussion

RichArt's avatar
RichArt
Frequent Visitor
2 years ago
Solved

Find days between different change types

Hi,

 

This is my first post on this forum and I am just learning Power Query, now i have an issue that is really put my mind out of it.

For a project change type i need to find the number of days between an Error change and the latest date before that Error

If there is no Error change within the same project i do not need this project in the outcome.

To be sure: The dates are in European style (e.g.  Day-month-year)  I do not mind if you change it to another style 😉

 

Available data

Project IDChange typeDate
P01Basis01/01/2024
P01Error04/01/2024
P02Basis01/03/2024
P02Agreement02/03/2024
P02Error03/03/2024
P02Error06/03/2024
P02Agreement05/03/2024
P03Basis01/04/2024
P03Agreement02/04/2024
P04Basis01/05/2024
P04Error03/05/2024

 

The result table should look like (third column is not needed in table but I added it for more clarity):

Project IDdays between error and earlier date in project

(explanation, not needed in actual table)

 

P013(difference between 04-01-2024 and 01-01-2024)
P021(difference between 03-03-2024 and 02-03-2024)
P023(difference between 06-03-2024 and 03-03-2024)
P042(difference between 03-05-2024 and 01-05-2024)

 

 

I dearly hope somebody can help me.

 

Regards

RichArt

  • Hi RichArt,

     

    Solution 1 (if you don't want date sort)

     

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WCjAwVNJRckosziwG0gaG+kBkZGBkohSrA5N0LSrKLwJJmqBJGqHpNEaXdEwvSk3NTc0rASkwwqIAbrQxPkkzQkaboikwRnOYCbokusNQFJig6TZFl0R2NUwyFgA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Project ID" = _t, #"Change type" = _t, Date = _t]),
        ChangedType = Table.TransformColumnTypes(Source,{{"Date", type date}}, "sk-SK"),
        GroupedRows = Table.Group(ChangedType, {"Project ID"}, {{"All", each Table.Sort(_, {{"Date", Order.Ascending}}) , type table}}),
        Ad_ErrorPositions = Table.AddColumn(GroupedRows, "ErrorPositions", each List.PositionOf([All][Change type], "Error", Occurrence.All), type list),
        Ad_Days = Table.AddColumn(Ad_ErrorPositions, "Days", each
         [ a = { Duration.TotalDays([All][Date]{[ErrorPositions]{0}} - [All][Date]{[ErrorPositions]{0}-1}) },
           b = if List.Count([ErrorPositions]) > 1 then a & List.Transform(List.Skip(List.Zip({ [ErrorPositions], {null} & List.RemoveLastN([ErrorPositions], 1) })), (x)=> Duration.TotalDays([All][Date]{x{0}} - [All][Date]{x{1}}))
               else if List.Count([ErrorPositions]) = 1 then a else null
         ][b], type list),
        FiteredRows = Table.SelectRows(Ad_Days, each [Days] <> null),
        RemovedColumns = Table.RemoveColumns(FiteredRows,{"All", "ErrorPositions"}),
        ExpandedDays = Table.ExpandListColumn(RemovedColumns, "Days")
    in
        ExpandedDays

     

     

    Solution 2 (with date sort)

     

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WCjAwVNJRckosziwG0gaG+kBkZGBkohSrA5N0LSrKLwJJmqBJGqHpNEaXdEwvSk3NTc0rASkwwqIAbrQxPkkzQkaboikwRnOYCbokusNQFJig6TZFl0R2NUwyFgA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Project ID" = _t, #"Change type" = _t, Date = _t]),
        ChangedType = Table.TransformColumnTypes(Source,{{"Date", type date}}, "sk-SK"),
        GroupedRows = Table.Group(ChangedType, {"Project ID"}, {{"All", each Table.Sort(_, {{"Date", Order.Ascending}}), type table}}),
        Ad_ErrorPositions = Table.AddColumn(GroupedRows, "ErrorPositions", each List.PositionOf([All][Change type], "Error", Occurrence.All), type list),
        Ad_Days = Table.AddColumn(Ad_ErrorPositions, "Days", each
           List.Transform([ErrorPositions], (x)=> Duration.TotalDays([All][Date]{x} - [All][Date]{x-1}))
           
      , type list),
        FiteredRows = Table.SelectRows(Ad_Days, each List.Count([Days]) > 0),
        RemovedColumns = Table.RemoveColumns(FiteredRows,{"All", "ErrorPositions"}),
        ExpandedDays = Table.ExpandListColumn(RemovedColumns, "Days")
    in
        ExpandedDays

     

  • Here is my corrected code. dufoq3  solution 2 will run quicker than this on larger datasets.

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WCjAwVNJRckosziwG0gaG+kBkZGBkohSrA5N0LSrKLwJJmqBJGqHpNEaXdEwvSk3NTc0rASkwwqIAbrQxPkkzQkaboikwRnOYCbokusNQFJig6TZFl0R2NUwyFgA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Project ID" = _t, #"Change type" = _t, Date = _t]),
        Custom2 = Table.TransformColumns(Source, {{"Date", each Date.From(_, "de-DE"), type date}}),
        #"Changed Type" = Table.TransformColumnTypes(Custom2,{{"Project ID", type text}, {"Change type", type text}, {"Date", type date}}),
        Custom1 = Table.AddColumn(#"Changed Type", "lastNonErrorDate", each List.Max(Table.SelectRows(#"Changed Type", (x)=> x[Project ID] = [Project ID] /*and x[Change type] <> "Error"*/ and x[Date] < [Date])[Date]), type date),
        #"Added Custom" = Table.AddColumn(Custom1, "Days", each if [Change type] = "Error" then Number.From([Date]) - Number.From([lastNonErrorDate]) else null, Int64.Type),
        #"Filtered Rows" = Table.SelectRows(#"Added Custom", each ([Days] <> null)),
        #"Removed Other Columns" = Table.SelectColumns(#"Filtered Rows",{"Project ID", "Days"})
    in
        #"Removed Other Columns"

     

13 Replies

  • There are likely a few ways to do this. Here is an example of one way...
    start

    end

    code

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WCjAwVNJRckosziwG0gaG+kBkZGBkohSrA5N0LSrKLwJJmqBJGqHpNEaXdEwvSk3NTc0rASkwwqIAbrQxPkkzQkaboikwRnOYCbokusNQFJig6TZFl0R2NUwyFgA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Project ID" = _t, #"Change type" = _t, Date = _t]),
        Custom2 = Table.TransformColumns(Source, {{"Date", each Date.From(_, "de-DE"), type date}}),
        #"Changed Type" = Table.TransformColumnTypes(Custom2,{{"Project ID", type text}, {"Change type", type text}, {"Date", type date}}),
        Custom1 = Table.AddColumn(#"Changed Type", "lastNonErrorDate", each List.Max(Table.SelectRows(#"Changed Type", (x)=> x[Project ID] = [Project ID] and x[Change type] <> "Error" and x[Date] < [Date])[Date]), type date),
        #"Added Custom" = Table.AddColumn(Custom1, "Days", each if [Change type] = "Error" then Number.From([Date]) - Number.From([lastNonErrorDate]) else null, Int64.Type)
    in
        #"Added Custom"
    • RichArt's avatar
      RichArt
      Frequent Visitor

      Oh Wauw,  thats quick.  thanks

       

      Looks like yout hit it but unfortunatly in your END table  row 6 , I need the days between the first error and the second error  date entry, so this row should end up with  3 days  (difference between row 5 and row 6)

       

      • jgeddes's avatar
        jgeddes
        Super User

        To clarify, if there is one error in a project then you require the days between the error date and the last non-error date. If there is more than one error in the project you require the days between the error dates regardless if there are non-error dates inbetween?

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi RichArt ,

    Please try this way:
    First, duplicate the original table:

    Put all of this M functions into the Advanced Editor in the duplicated table which will add an Index column for all rows where [Change type] = "Error":

     

    let
        Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WCjAwVNJRckosziwG0ob6hvpGBkYmSrE6MCnXoqL8IrCUCbKUEZIuY1RdICnH9KLU1NzUvBKwtBG6NMxQY31j3FJm+A01RZY2RnKOCapzjNF0mqA6xwRJpymqThMk55jCXRoLAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Project ID" = _t, #"Change type" = _t, Date = _t]),
        #"Changed Type" = Table.TransformColumnTypes(Source,{{"Project ID", type text}, {"Change type", type text}, {"Date", type date}}),
        #"Filtered Rows" = Table.SelectRows(#"Changed Type", each ([Change type] = "Error")),
        #"Grouped Rows" = Table.Group(#"Filtered Rows", {"Project ID"}, {{"Count", each Table.AddIndexColumn(_, "Index", 1, 1, Int64.Type)}}),
        #"Expanded Count" = Table.ExpandTableColumn(#"Grouped Rows", "Count", {"Change type", "Date", "Index"}, {"Change type", "Date", "Index"})
    in
        #"Expanded Count"

     

    Then create a Blank Query:

    Put all of these M functions into the Advanced Editor in the Blank Query:

     

    let
        Source = Table.NestedJoin(Table, {"Project ID", "Change type", "Date"}, #"Table (2)", {"Project ID", "Change type", "Date"}, "Table (2)", JoinKind.LeftOuter),
        #"Expanded Table (2)" = Table.ExpandTableColumn(Source, "Table (2)", {"Index"}, {"Index"}),
        #"Added Custom" = Table.AddColumn(#"Expanded Table (2)", "Custom Date", each 
            let 
                currentRow = _,
                projectId = currentRow[Project ID],
                changeType = currentRow[Change type],
                index = currentRow[Index],
                date = currentRow[Date]
            in 
            if changeType = "Error" then 
                if index = 1 then 
                    let 
                        sameProjectRows = Table.SelectRows(#"Expanded Table (2)", each [Project ID] = projectId),
                        filteredRows = Table.SelectRows(sameProjectRows, each [Change type] <> "Error" and [Date] < date),
                        maxDate = if 
                        Table.IsEmpty(filteredRows) then null else List.Max(Table.Column(filteredRows, "Date"))
                    in 
                        maxDate 
                else 
                    let 
                        sameProjectPreviousIndexRow = Table.SelectRows(#"Expanded Table (2)", each [Project ID] = projectId and [Change type] = "Error" and [Index] = index - 1),
                        previousDate = if Table.IsEmpty(sameProjectPreviousIndexRow) then null else sameProjectPreviousIndexRow{0}[Date]
                    in 
                        previousDate
                else 
                    null 
            ),
        #"Added Custom1" = Table.AddColumn(#"Added Custom", "Custom", each [Date] - [Custom Date]),
        #"Changed Type" = Table.TransformColumnTypes(#"Added Custom1",{{"Custom", type number}})
    in
        #"Changed Type"

     

    The final output is as below:


    Best Regards,
    Dino Tao
    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

    • RichArt's avatar
      RichArt
      Frequent Visitor

      Thank you for your help!

      Your end result looks good but somehow the expanding in the balnk query does not align the index numbers with me. I tried different kind of merges but i cannot get it done.  This is the result with copy paste your solution:

       

  • RichArt's avatar
    RichArt
    Frequent Visitor

    Hi dufoq3  and jgeddes 

     

    Thank you both so much for these fast reply's and solutions.  I am really gratefull there are still people who are willingly and able to help others, thanks!

     

    I set up multiple variations to double check and the options from the both of you are a solution and work great.

    Actually i will stick to the solution of jgeddes  since I am able to keep the other columns as well, probarly the solution from dufoq3 would be able to do so but i cannot find how at this moment.  I did not implement it on my actual data set but will be doing this tonight, i will keep you posted.

     

    Thank again!

    • dufoq3's avatar
      dufoq3
      Community Champion

      You're welcome, just kind advice: for future think in advance what your output should looks like and be careful with sample data - there have to be no mistakes.

      • RichArt's avatar
        RichArt
        Frequent Visitor

        Yes i noted that already as a learning from this topic 👍