Forum Discussion
How to drop duplicates in event stream using event processor ?
Sure
in the Eventstream you can add a transformation task - here you will find the "group by" operator.
in this task you can add a windowing function and setup the needed aggregation
I hope you can find it 😊
Hi
You can make a windowing function with the "group by" transformation.
This will give you options in different windowing functions and you can add a time-span in which the window should look and remove the dublicates (group the rows).
It is not the best option to remove them here though - I would remove them in the Eventhouse using an update policy or a materialized view. In this way you are out of the windowing function and can have a faster stream of data to the Eventhouse. The update policy or materialized view is also quicker than the Eventstream for data processing 🙂
Hi datacoffee , thanks for the answer.
Yes , we can drop duplicates in eventhouse, but our need is to drop them in eventstream only.
In groupby, which operation are you suggesting , can you please mention ?
Using count, we can identify the duplicates but removing them seems a different task to me.
Sure
in the Eventstream you can add a transformation task - here you will find the "group by" operator.
in this task you can add a windowing function and setup the needed aggregation
I hope you can find it 😊
Yeah, datacoffee is correct. You can find the group by operation in the drop down list of transformation.
When you config it, you can select 'Average' as the aggregation method and which column to aggregate on. Then select all the remaining columns as Group operation by.
This is how you drop duplicates with current no code experience.
Once Eventstream can support users write SQL code (coming soon), you can use the following code to remove duplicates:
In the current no code experience, you can use 'group by' operator
WITH Temp AS (SELECT Value, DeviceIdFROM Input TIMESTAMP BY TimeGROUP BY Value, DeviceId, System.Timestamp())SELECTAVG(Value) AS AverageValue, DeviceIdINTO OutputFROM TempGROUP BY DeviceId,TumblingWindow(minute, 5)
