Forum Discussion
How to drop duplicates in event stream using event processor ?
Sure
in the Eventstream you can add a transformation task - here you will find the "group by" operator.
in this task you can add a windowing function and setup the needed aggregation
I hope you can find it 😊
Hi datacoffee , thanks for the answer.
Yes , we can drop duplicates in eventhouse, but our need is to drop them in eventstream only.
In groupby, which operation are you suggesting , can you please mention ?
Using count, we can identify the duplicates but removing them seems a different task to me.
Sure
in the Eventstream you can add a transformation task - here you will find the "group by" operator.
in this task you can add a windowing function and setup the needed aggregation
I hope you can find it 😊
Yeah, datacoffee is correct. You can find the group by operation in the drop down list of transformation.
When you config it, you can select 'Average' as the aggregation method and which column to aggregate on. Then select all the remaining columns as Group operation by.
This is how you drop duplicates with current no code experience.
Once Eventstream can support users write SQL code (coming soon), you can use the following code to remove duplicates:
In the current no code experience, you can use 'group by' operator
WITH Temp AS (SELECT Value, DeviceIdFROM Input TIMESTAMP BY TimeGROUP BY Value, DeviceId, System.Timestamp())SELECTAVG(Value) AS AverageValue, DeviceIdINTO OutputFROM TempGROUP BY DeviceId,TumblingWindow(minute, 5)
