Forum Discussion
Saving events streaming data in a lakehouse
Hi,
I need to saving many events data from an Azure events hub, having an id, a version and an event_date as principal fields.
I need to maintain for each event, identified by the related id, only the last version, so a new version has to overwrite the old version.
I think to write the events in a lakehouse as a destination, then to copy the events from the lakehouse to an warehouse, in a staging area, and then using a merge statement to update the definitive events in the warehouse respect to the events in the staging area.
Based on this strategy, how could the event processor help me?
Could the event processor delete the events older than 30 minutes in the destination lakehouse, in order to copy in the warehouse only newer data?
Any suggests to me, please? Thanks
Hi pmscorca ,
Eventstream cannot process any data that has been passed to the destination. I recommend that you create a Dataflow Gen 2 after the eventstream. The Dataflow uses the lakehouse used by your eventstream's destination as the data source, and performs operations such as merge and delete obsolete data in Power Query online. Then use this lakehouse or create a new lakehouse as the destination of the Dataflow, and set the destination's Update method to Replace:
Best Regards,
Dino Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
4 Replies
I need to maintain for each event, identified by the related id, only the last version, so a new version has to overwrite the old version.What's the reasoning? Don't you want to keep history?
Hi, the user requirement is to do not keep the history of the events.
Hi pmscorca ,
Eventstream cannot process any data that has been passed to the destination. I recommend that you create a Dataflow Gen 2 after the eventstream. The Dataflow uses the lakehouse used by your eventstream's destination as the data source, and performs operations such as merge and delete obsolete data in Power Query online. Then use this lakehouse or create a new lakehouse as the destination of the Dataflow, and set the destination's Update method to Replace:
Best Regards,
Dino Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.Hi pmscorca ,
Sorry I need to correct that statement:Eventstream cannot process any data that has been passed to the destination.What this means is that you can no longer merge data that has been passed to the destination, but you can indeed set a retention period for the data:
But he can only set the minimum to 1 day, your requirement is 30min, so I'm afraid this can't fulfill your requirement, still please use the above method.
Best Regards,
Dino Tao
