topic Re: Saving events streaming data in a lakehouse in Eventstream

Saving events streaming data in a lakehouse

pmscorca — Fri, 11 Oct 2024 14:30:25 GMT

Hi,

I need to saving many events data from an Azure events hub, having an id, a version and an event_date as principal fields.

I need to maintain for each event, identified by the related id, only the last version, so a new version has to overwrite the old version.

I think to write the events in a lakehouse as a destination, then to copy the events from the lakehouse to an warehouse, in a staging area, and then using a merge statement to update the definitive events in the warehouse respect to the events in the staging area.

Based on this strategy, how could the event processor help me?

Could the event processor delete the events older than 30 minutes in the destination lakehouse, in order to copy in the warehouse only newer data?

Any suggests to me, please? Thanks

Re: Saving events streaming data in a lakehouse

lbendlin — Fri, 11 Oct 2024 17:43:14 GMT

I need to maintain for each event, identified by the related id, only the last version, so a new version has to overwrite the old version.

What's the reasoning? Don't you want to keep history?

Re: Saving events streaming data in a lakehouse

pmscorca — Fri, 11 Oct 2024 18:11:54 GMT

Hi, the user requirement is to do not keep the history of the events.

Re: Saving events streaming data in a lakehouse

Anonymous — Mon, 14 Oct 2024 02:54:20 GMT

Hi pmscorca ,

Eventstream cannot process any data that has been passed to the destination. I recommend that you create a Dataflow Gen 2 after the eventstream. The Dataflow uses the lakehouse used by your eventstream's destination as the data source, and performs operations such as merge and delete obsolete data in Power Query online. Then use this lakehouse or create a new lakehouse as the destination of the Dataflow, and set the destination's Update method to Replace:

Best Regards,
Dino Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Saving events streaming data in a lakehouse

Anonymous — Tue, 15 Oct 2024 03:19:24 GMT

Hi pmscorca ,

Sorry I need to correct that statement:

Eventstream cannot process any data that has been passed to the destination.

What this means is that you can no longer merge data that has been passed to the destination, but you can indeed set a retention period for the data:

But he can only set the minimum to 1 day, your requirement is 30min, so I'm afraid this can't fulfill your requirement, still please use the above method.

Best Regards,
Dino Tao