Forum Discussion

hirokichi's avatar
hirokichi
Regular Visitor
4 years ago
Solved

Performance of counting overlapped datetime

Hi there. The dataset consists of columns for ID, start time, and end time, with tens of thousands of rows. I would like to aggregate other rows that overlap in time based on the start time of each...
  • Anonymous's avatar
    Anonymous
    4 years ago

    Hi hirokichi 

     

    Groupby is slow, so did those 2 queries above achieve what you wanted? How about buffer the table in Query 2?

    let
        source = Csv.Document(File.Contents("C:\SampleTime.csv"),[Delimiter=",", Columns=3, Encoding=932, QuoteStyle=QuoteStyle.None]),
        header = Table.PromoteHeaders(source, [PromoteAllScalars=true]),
        convertType =Table.Buffer( Table.TransformColumnTypes(header,{{"Id", Int64.Type}, {"StartDate", type datetime}, {"EndDate", type datetime}})),
        addColumn = Table.AddColumn(convertType, "Overlap", each Table.RowCount(Table.SelectRows(convertType, (x) => x[StartDate] <= [StartDate] and [StartDate] <= x[EndDate])))
    in
        addColumn

     

    try query 3

    let
        source = Csv.Document(File.Contents("C:\SampleTime.csv"),[Delimiter=",", Columns=3, Encoding=932, QuoteStyle=QuoteStyle.None]),
        header = Table.PromoteHeaders(source, [PromoteAllScalars=true]),
        convertType = Table.Buffer( Table.TransformColumnTypes(header,{{"Id", Int64.Type}, {"StartDate", type datetime}, {"EndDate", type datetime}})),
        Custom = List.Buffer( Table.AddColumn(convertType, "Custom", each {[StartDate], [EndDate]})[Custom]),
        #"Added Custom1" = Table.AddColumn(convertType, "Overlap", (x)=> List.Count(List.Select( List.Transform(Custom, each _{0} <=x[StartDate] and _{1}>=x[StartDate]), each _=true)))
    in
        #"Added Custom1"