Forum Discussion
hirokichi
4 years agoRegular Visitor
Performance of counting overlapped datetime
Hi there. The dataset consists of columns for ID, start time, and end time, with tens of thousands of rows. I would like to aggregate other rows that overlap in time based on the start time of each...
- Anonymous4 years ago
Hi hirokichi
Groupby is slow, so did those 2 queries above achieve what you wanted? How about buffer the table in Query 2?
let source = Csv.Document(File.Contents("C:\SampleTime.csv"),[Delimiter=",", Columns=3, Encoding=932, QuoteStyle=QuoteStyle.None]), header = Table.PromoteHeaders(source, [PromoteAllScalars=true]), convertType =Table.Buffer( Table.TransformColumnTypes(header,{{"Id", Int64.Type}, {"StartDate", type datetime}, {"EndDate", type datetime}})), addColumn = Table.AddColumn(convertType, "Overlap", each Table.RowCount(Table.SelectRows(convertType, (x) => x[StartDate] <= [StartDate] and [StartDate] <= x[EndDate]))) in addColumntry query 3
let source = Csv.Document(File.Contents("C:\SampleTime.csv"),[Delimiter=",", Columns=3, Encoding=932, QuoteStyle=QuoteStyle.None]), header = Table.PromoteHeaders(source, [PromoteAllScalars=true]), convertType = Table.Buffer( Table.TransformColumnTypes(header,{{"Id", Int64.Type}, {"StartDate", type datetime}, {"EndDate", type datetime}})), Custom = List.Buffer( Table.AddColumn(convertType, "Custom", each {[StartDate], [EndDate]})[Custom]), #"Added Custom1" = Table.AddColumn(convertType, "Overlap", (x)=> List.Count(List.Select( List.Transform(Custom, each _{0} <=x[StartDate] and _{1}>=x[StartDate]), each _=true))) in #"Added Custom1"