Forum Discussion
Rate limited record inclusion help?
- 3 years ago
Hello, cmbyrd your (x)=> function is just a declaration. Ones you define it - call it with some x value to get true or false as a result. You would want to get latest entry_date with filter = "included". But you are in the process of creation of this column. So this approach won't work. Try the code below with List.Generate. The idea behind it is to get list of records of your table, iterate it one by one (always having time of the latest "approved" entry_date) and generating your "filter" column.
By the way your example is not correct. There are 53 minutes between 1st and 2nd entry. 2nd must be "Included".
Another options would be to use List.Accumulate. One could even filter out (skip) those "excluded" but you decided to keep them. There is also some room for algo optimization (e.g. find and select next record with entry_date + 5mins while travelling over "entry_date" column. But that's another story 😀
code updated - bug fixing
let raw = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("Xc3BDQAhCETRVjacTcRB40orhP7bWMHTev0vMGbUqBCqVDDkaVOZd2DyYvReNGSHlgS+bUQ4tm6LIGnyu4Mo4mc/czOtH1uK2B/k/gE=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [UPDATE_ID = _t, ENTRY_DATE = _t, Index = _t]), Source = Table.TransformColumnTypes(raw,{{"ENTRY_DATE", type datetime}, {"Index", Int64.Type}}), rec = List.Buffer(Table.ToRecords(Source)), generate_filter = List.Generate( () => rec{0} & [Filter = "Include", last_included = rec{0}[ENTRY_DATE], go = true], (x) => x[go], (x) => try let current_record = rec{x[Index] + 1}, good_to_go = Duration.Minutes(current_record[ENTRY_DATE] - x[last_included]) >= 5, add_fields = if good_to_go then [Filter = "Include", last_included = current_record[ENTRY_DATE], go = true] else [Filter = "Exclude", last_included = x[last_included], go = true] in current_record & add_fields otherwise [go = false] ), z = Table.RemoveColumns(Table.FromRecords(generate_filter), {"last_included", "go"}) in z
Hello, cmbyrd your (x)=> function is just a declaration. Ones you define it - call it with some x value to get true or false as a result. You would want to get latest entry_date with filter = "included". But you are in the process of creation of this column. So this approach won't work. Try the code below with List.Generate. The idea behind it is to get list of records of your table, iterate it one by one (always having time of the latest "approved" entry_date) and generating your "filter" column.
By the way your example is not correct. There are 53 minutes between 1st and 2nd entry. 2nd must be "Included".
Another options would be to use List.Accumulate. One could even filter out (skip) those "excluded" but you decided to keep them. There is also some room for algo optimization (e.g. find and select next record with entry_date + 5mins while travelling over "entry_date" column. But that's another story 😀
code updated - bug fixing
let
raw = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("Xc3BDQAhCETRVjacTcRB40orhP7bWMHTev0vMGbUqBCqVDDkaVOZd2DyYvReNGSHlgS+bUQ4tm6LIGnyu4Mo4mc/czOtH1uK2B/k/gE=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [UPDATE_ID = _t, ENTRY_DATE = _t, Index = _t]),
Source = Table.TransformColumnTypes(raw,{{"ENTRY_DATE", type datetime}, {"Index", Int64.Type}}),
rec = List.Buffer(Table.ToRecords(Source)),
generate_filter =
List.Generate(
() => rec{0} & [Filter = "Include", last_included = rec{0}[ENTRY_DATE], go = true],
(x) => x[go],
(x) =>
try
let
current_record = rec{x[Index] + 1},
good_to_go = Duration.Minutes(current_record[ENTRY_DATE] - x[last_included]) >= 5,
add_fields =
if good_to_go
then [Filter = "Include", last_included = current_record[ENTRY_DATE], go = true]
else [Filter = "Exclude", last_included = x[last_included], go = true]
in current_record & add_fields
otherwise [go = false]
),
z = Table.RemoveColumns(Table.FromRecords(generate_filter), {"last_included", "go"})
in
z
Thank you so much! Fiddled a bit to figure out exactly what was going on, I think I learned at least a little, and I've got this working after pulling my hair out for many hours.