Forum Discussion

cmbyrd's avatar
cmbyrd
Frequent Visitor
3 years ago
Solved

Rate limited record inclusion help?

Hi everyone.  I have a dataset containing updates to records.  When changes are made, and the user hits 'save', an entry is made for every field that is changed in the record.  Sometimes this results...
  • AlienSx's avatar
    3 years ago

    Hello, cmbyrd your (x)=> function is just a declaration. Ones you define it - call it with some x value to get true or false as a result. You would want to get latest entry_date with filter = "included". But you are in the process of creation of this column. So this approach won't work. Try the code below with List.Generate. The idea behind it is to get list of records of your table, iterate it one by one (always having time of the latest "approved" entry_date) and generating your "filter" column.

    By the way your example is not correct. There are 53 minutes between 1st and 2nd entry. 2nd must be "Included".

    Another options would be to use List.Accumulate. One could even filter out (skip) those "excluded" but you decided to keep them. There is also some room for algo optimization (e.g. find and select next record with entry_date + 5mins while travelling over "entry_date" column. But that's another story ğŸ˜€

    code updated - bug fixing

    let
        raw = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("Xc3BDQAhCETRVjacTcRB40orhP7bWMHTev0vMGbUqBCqVDDkaVOZd2DyYvReNGSHlgS+bUQ4tm6LIGnyu4Mo4mc/czOtH1uK2B/k/gE=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [UPDATE_ID = _t, ENTRY_DATE = _t, Index = _t]),
        Source = Table.TransformColumnTypes(raw,{{"ENTRY_DATE", type datetime}, {"Index", Int64.Type}}),
        rec = List.Buffer(Table.ToRecords(Source)),
        generate_filter = 
            List.Generate(
                () => rec{0} & [Filter = "Include", last_included = rec{0}[ENTRY_DATE], go = true],
                (x) => x[go],
                (x) => 
                    try
                        let 
                            current_record = rec{x[Index] + 1},
                            good_to_go = Duration.Minutes(current_record[ENTRY_DATE] - x[last_included]) >= 5,
                            add_fields = 
                            if good_to_go 
                            then [Filter = "Include", last_included = current_record[ENTRY_DATE], go =  true]
                            else [Filter = "Exclude", last_included = x[last_included], go =  true]
                        in current_record & add_fields
                    otherwise [go = false]
            ),
        z = Table.RemoveColumns(Table.FromRecords(generate_filter), {"last_included", "go"})
    in
        z