It would be great if you can define Create/Update/Delete actions from Event Processing. So depending of a field you can say i.e. "if this is field contains "delete" it's a delete operation, use the ID stored in field ID to identify the record to be deleted in your lakehouse". Same with Updates and Inserts. That would be super helpful to have a table which always contains the latest version of a record. This would be i.e. a great help when you receive events from D365 which can be new records, updates or deletes.
3 Comments
THANKS @Uri for the feedback. This is an interesting approach but it would require that you keep the full history of data in the KQL DB which is probably not the best idea, considering large volumes of data and the availability of standard SQL endpoints etc., a warehouse/lakehouse based approach might make more sense. But anyway, a step in the right direction :-)
UriBarashMicrosoft Employee
This is already partially feasible with a combination of materialized views and arg_max on the relevant columns. Please see how it is done for Cosmos DB change feeds here: Get the latest versions of Azure Cosmos DB documents. We will consider adding UX support or wizard, based on popuarity of the idea.
Full ingestion of CDC like feeds (Upserts and deletes) is not there yet.
- Status added:Needs Votes
