Forum Discussion

chetanhiwale's avatar
chetanhiwale
Icon for Resolver I rankResolver I
1 year ago
Solved

KQL update policy

I have 3 tables, src_table (data inserted from eventstream), target_table (transformed data). reference table. How a join can be applied in update policy query parameters for src_table and reference_...
  • Anonymous's avatar
    Anonymous
    1 year ago

    Hi chetanhiwale ,

     

    Please try the following alternatives to see if they work:

    1. Intermediate Table Approach

    (1)Continue using `src_table` with streaming ingestion enabled to receive real-time data.
    (2)Create an intermediate table (e.g., `intermediate_table`) without streaming ingestion.
    (3)Set up an update policy on `intermediate_table` to transform and join data from `src_table` and `reference_table`, and then store the result in `target_table`.

       .alter table intermediate_table policy update
       @'[{"IsEnabled": true, "Source": "src_table", "Query": "src_table | join kind=inner (reference_table) on id | project src_table.id, src_table.name, reference_table.additional_info", "IsTransactional": true}]'
    

     

    2.Use a scheduled query to periodically join `src_table` and `reference_table`, and insert the results into `target_table`. This way, you can keep streaming ingestion enabled on `src_table`.

    ```kql
       .create-or-alter function UpdateTargetTable() {
           src_table
           | join kind=inner (reference_table) on id
           | project src_table.id, src_table.name, reference_table.additional_info
           | into target_table
       }
    
       .create-or-alter function ScheduledUpdate() {
           UpdateTargetTable()
       }
    
       .create-or-alter function ScheduledUpdatePolicy() {
           .set-or-append target_table <| ScheduledUpdate()
       }
    
       .schedule function ScheduledUpdatePolicy() every 5m
       ```

     

    Best Regards,

    Neeko Tang

    If this post  helps, then please consider Accept it as the solution  to help the other members find it more quickly.