Forum Discussion

chetanhiwale's avatar
chetanhiwale
Resolver I
1 year ago
Solved

How to drop duplicates in event stream using event processor ?

I am currently exploring what are the differnet ways to drop drop duplicates in eventstream via event processor.  I have understanding that by using groupby we can atleast identify the duplicates.  ...
Eventstream
  • datacoffee's avatar
    datacoffee
    1 year ago

    Sure

     

    in the Eventstream you can add a transformation task - here you will find the "group by" operator.

    in this task you can add a windowing function and setup the needed aggregation 

     

    I hope you can find it 😊

datacoffee's avatar
datacoffee
Most Valuable Professional
1 year ago

Sure

 

in the Eventstream you can add a transformation task - here you will find the "group by" operator.

in this task you can add a windowing function and setup the needed aggregation 

 

I hope you can find it 😊

Alicia_Li_MSFT's avatar
Alicia_Li_MSFT
Microsoft Employee
1 year ago

Yeah, datacoffee  is correct. You can find the group by operation in the drop down list of transformation.

 

When you config it, you can select 'Average' as the aggregation method and which column to aggregate on. Then select all the remaining columns as Group operation by. 

 

This is how you drop duplicates with current no code experience.

 

Once Eventstream can support users write SQL code (coming soon), you can use the following code to remove duplicates:

 

 

 

 

In the current no code experience, you can use 'group by' operator 

 

WITH Temp AS (
SELECT Value, DeviceId
FROM Input TIMESTAMP BY Time
GROUP BY Value, DeviceId, System.Timestamp()
)
 
 
SELECT
AVG(Value) AS AverageValue, DeviceId
INTO Output
FROM Temp
GROUP BY DeviceId,TumblingWindow(minute, 5)