Forum Discussion

sgt_pepper's avatar
sgt_pepper
Frequent Visitor
1 year ago
Solved

How to approach a monthly summary from a data start that has date start and end columns

Hi all,

 

I think this is really a Power Query question but ultimately i'll be using this in Power BI. Scratching my head with this one and would appreciate some advice. I have a dataset of up/down time periods for approximately 15 websites. See an illustrative example at the bottom of this post. I need to report the % of uptime for each website but with this data set - see 'Desired outcome' below. I'm struggling to get what I want. I feel I'm close with a combination of filtering to only 'down' periods, adding an 'End of month' column (named 'created_at_EOM' in the example below), then grouping the data by website_name, date_created, duration (aggregated as a sum). Some problems:

 

  • The request is that the reporting show % of uptime. I'm wondering if something like dividing the duration (shown in ms) by the number of milliseconds in the month? Not really sure if best approach.
  • For months where there is no down time, I don't have entries (I probably need to merge with a date table? Have struggled to do this with the multiple websites though).

Thanks in advance on any input! 🙂

 

Data set example

(The blanks are becuase of the most recent period, but I can easily fix those by inputting the current time with 'DateTime.LocalNow')

TestIDWebsiteNamestatuscreated_atended_atduration
1Website 1up2022-03-24T09:38:17+00:002022-04-03T09:28:43+00:00863426000
1Website 1down2022-04-03T09:28:43+00:002022-04-03T09:30:10+00:0087000
1Website 1up2022-04-03T09:30:10+00:002022-05-02T08:06:09+00:002500559000
1Website 1down2022-05-02T08:06:09+00:002022-05-02T08:07:29+00:0080000
1Website 1up2022-05-02T08:07:29+00:002022-05-29T10:29:54+00:002341345000
1Website 1down2022-05-29T10:29:54+00:002022-05-29T10:31:07+00:0073000
1Website 1up2022-05-29T10:31:07+00:002023-01-15T13:40:08+00:0019969741000
1Website 1down2023-01-15T13:40:08+00:002023-01-15T13:41:21+00:0073000
1Website 1up2023-01-15T13:41:21+00:002023-01-15T16:13:06+00:009105000
1Website 1down2023-01-15T16:13:06+00:002023-01-15T16:14:17+00:0071000
1Website 1up2023-01-15T16:14:17+00:00  
2Website 2up2017-02-08T11:15:45+00:002023-03-19T14:30:30+00:001.92684E+11
2Website 2down2023-03-19T14:30:30+00:002023-03-19T14:40:19+00:00589000
2Website 2up2023-03-19T14:40:19+00:002023-03-19T15:21:47+00:002488000
2Website 2down2023-03-19T15:21:47+00:002023-03-19T15:32:31+00:00644000
2Website 2up2023-03-19T15:32:31+00:00  
3Website 3up2023-08-22T05:04:23+00:002023-09-05T00:59:57+00:001194934000
3Website 3down2023-09-05T00:59:57+00:002023-09-05T01:00:03+00:006000
3Website 3up2023-09-05T01:00:03+00:002023-09-14T16:31:51+00:00833508000
3Website 3down2023-09-14T16:31:51+00:002023-09-14T16:33:02+00:0071000
3Website 3up2023-09-14T16:33:02+00:002023-12-01T03:44:51+00:006693109000
3Website 3down2023-12-01T03:44:51+00:002023-12-01T03:47:22+00:00151000
3Website 3up2023-12-01T03:47:22+00:002023-12-01T05:51:31+00:007449000
3Website 3down2023-12-01T05:51:31+00:002023-12-01T05:54:06+00:00155000
3Website 3up2023-12-01T05:54:06+00:002024-04-21T01:16:35+00:0012252149000
3Website 3down2024-04-21T01:16:35+00:002024-04-21T01:18:59+00:00144000
3Website 3up2024-04-21T01:18:59+00:002024-04-29T08:46:35+00:00718056000
3Website 3down2024-04-29T08:46:35+00:002024-04-29T08:47:46+00:0071000
3Website 3up2024-04-29T08:47:46+00:002024-08-28T07:57:26+00:0010451380000
3Website 3down2024-08-28T07:57:26+00:002024-08-28T07:59:44+00:00138000
3Website 3up2024-08-28T07:59:44+00:00  

 

Desired result

(The % aren't based on above, I've just put random figures in to demonstrate)

TestIDWebsiteNameperioduptime
1Website 1Dec-24100.00%
1Website 1Jan-2599.98%
1Website 1Feb-2599.96%
1Website 1Mar-25etc
2Website 2Dec-24etc
2Website 2Jan-25etc
2Website 2Feb-25etc
2Website 2Mar-25etc
3Website 3Dec-24etc
3Website 3Jan-25etc
3Website 3Feb-25etc
3Website 3Mar-25etc

 

  • sgt_pepper 

    Here's how I'd approach it:

    1. Ingest the data

    You already have:

    • WebsiteName

    • created_at

    • ended_at

    • duration

    • status (up/down)

      Create a full website + month matrix

      Because: if there's no downtime, your data has no record — but you still want to show 100% uptime.

      Steps:

      • Create a Dates Table with all months you care about.

      • Cross-join it with your distinct WebsiteName list.

    Result: even "no downtime" months will exist.

    Handle downtime periods

    Now, filter your original table to only status = "down" periods.

    From this, for each Website + Month:

    • Sum the duration (downtime in milliseconds).

      Join it together

      • Merge (left join) your Website-Month master list with the downtime sums.

    If no downtime, downtime = 0.

    Calculate % uptime

    Simple formula:
    Uptime % = (Milliseconds in month - Downtime) / Milliseconds in month
    Dates Table
    let
    StartDate = #date(2022, 1, 1),
    EndDate = Date.EndOfMonth(DateTime.Date(DateTime.LocalNow())),
    Dates = List.Dates(StartDate, Duration.Days(EndDate - StartDate) + 1, #duration(1,0,0,0)),
    TableDates = Table.FromList(Dates, Splitter.SplitByNothing(), {"Date"}),
    AddYearMonth = Table.AddColumn(TableDates, "YearMonth", each Date.ToText([Date], "MMM-yy"), type text),
    RemoveDay = Table.Distinct(Table.SelectColumns(AddYearMonth, {"YearMonth"}))
    in
    RemoveDay

    Just pull distinct WebsiteName from your fact table.
    Cross-join (all combinations):
    CrossJoin = Table.AddColumn(WebsiteList, "Custom", each RemoveDay),
    Expand = Table.ExpandTableColumn(CrossJoin, "Custom", {"YearMonth"})

    Downtime Aggregation

    • Filter status = "down".

    • Create a YearMonth column based on created_at.

    Group by WebsiteName, YearMonth, SUM of duration.

    Merge downtime into cross-join

    • Left-join the downtime sums onto your Website-Month table.

    If downtime is null, replace it with 0.

    Add Days in Month

    Add a column:
    DaysInMonth = Date.DaysInMonth(Date.FromText("01-" & [YearMonth]))
    Milliseconds in a month:
    MillisecondsInMonth = [DaysInMonth] * 24 * 60 * 60 * 1000

    Final Calculation

    Uptime %:
    UptimePercent = ([MillisecondsInMonth] - [Downtime]) / [MillisecondsInMonth]

    Did I answer your question? Mark my post as a solution! Appreciate your Kudos !!










4 Replies

  • sgt_pepper's avatar
    sgt_pepper
    Frequent Visitor

    Thanks all for the input. Through a combination of these reponses I've managed to get what I need. 🙂

  • Hi sgt_pepper The issue lies in calculating uptime from a dataset with start and end dates spanning multiple months. Begin by replacing null ended_at values with the current date to handle ongoing periods. Next, use a date table to align your data with monthly intervals and split durations accordingly. Finally, aggregate the data by month and website, calculating uptime as the proportion of up duration to the total time in the month.

  • Are you assuming that if it is not down then it is up? There are no other states ("not live yet", "decommissioned")?

     

    How granular do you need this to be?  By the second*, minute, or hour?  Keep in mind that uptime percentage per month becomes increasingly meaningless as months have different number of days.

     

    * meaning 31 million entries per year and website

     

    Your sample data indicates near 100% uptime.  Not sure what the business value is. You also have the same timestamp for the abutting events. That will further skew the result.

     

     

     

     

  • sgt_pepper 

    Here's how I'd approach it:

    1. Ingest the data

    You already have:

    • WebsiteName

    • created_at

    • ended_at

    • duration

    • status (up/down)

      Create a full website + month matrix

      Because: if there's no downtime, your data has no record — but you still want to show 100% uptime.

      Steps:

      • Create a Dates Table with all months you care about.

      • Cross-join it with your distinct WebsiteName list.

    Result: even "no downtime" months will exist.

    Handle downtime periods

    Now, filter your original table to only status = "down" periods.

    From this, for each Website + Month:

    • Sum the duration (downtime in milliseconds).

      Join it together

      • Merge (left join) your Website-Month master list with the downtime sums.

    If no downtime, downtime = 0.

    Calculate % uptime

    Simple formula:
    Uptime % = (Milliseconds in month - Downtime) / Milliseconds in month
    Dates Table
    let
    StartDate = #date(2022, 1, 1),
    EndDate = Date.EndOfMonth(DateTime.Date(DateTime.LocalNow())),
    Dates = List.Dates(StartDate, Duration.Days(EndDate - StartDate) + 1, #duration(1,0,0,0)),
    TableDates = Table.FromList(Dates, Splitter.SplitByNothing(), {"Date"}),
    AddYearMonth = Table.AddColumn(TableDates, "YearMonth", each Date.ToText([Date], "MMM-yy"), type text),
    RemoveDay = Table.Distinct(Table.SelectColumns(AddYearMonth, {"YearMonth"}))
    in
    RemoveDay

    Just pull distinct WebsiteName from your fact table.
    Cross-join (all combinations):
    CrossJoin = Table.AddColumn(WebsiteList, "Custom", each RemoveDay),
    Expand = Table.ExpandTableColumn(CrossJoin, "Custom", {"YearMonth"})

    Downtime Aggregation

    • Filter status = "down".

    • Create a YearMonth column based on created_at.

    Group by WebsiteName, YearMonth, SUM of duration.

    Merge downtime into cross-join

    • Left-join the downtime sums onto your Website-Month table.

    If downtime is null, replace it with 0.

    Add Days in Month

    Add a column:
    DaysInMonth = Date.DaysInMonth(Date.FromText("01-" & [YearMonth]))
    Milliseconds in a month:
    MillisecondsInMonth = [DaysInMonth] * 24 * 60 * 60 * 1000

    Final Calculation

    Uptime %:
    UptimePercent = ([MillisecondsInMonth] - [Downtime]) / [MillisecondsInMonth]

    Did I answer your question? Mark my post as a solution! Appreciate your Kudos !!