Forum Discussion

chetanhiwale's avatar
chetanhiwale
Resolver I
1 year ago
Solved

How to drop duplicates in event stream using event processor ?

I am currently exploring what are the differnet ways to drop drop duplicates in eventstream via event processor.  I have understanding that by using groupby we can atleast identify the duplicates.  ...
Eventstream
  • datacoffee's avatar
    datacoffee
    1 year ago

    Sure

     

    in the Eventstream you can add a transformation task - here you will find the "group by" operator.

    in this task you can add a windowing function and setup the needed aggregation 

     

    I hope you can find it 😊

svelde's avatar
svelde
Super User
1 year ago

Hello @chetanhiwale ,

thank you for your question at this Fabric community forum.

If the ingested data is sent to an Eventhouse KQL databse table, you can also consider ignoring duplicates in the Eventhouse using a materialized view.

Consider this example based on the Weather data service:

.create materialized-view with(lookback=20m, lookback_column = "EventProcessedUtcTime", backfill=true, docString="Unique Weather data entries", folder="MaterializedViews") WeatherDataset on table WeatherData

{

  WeatherData

  | summarize arg_max(EventProcessedUtcTime, *) by dateTime

}

Here, the latest ingested rows with the same EventProcessedUtcTime are represented.

The table on which this view is based has a limited retention time to keep the database storage size at the minimum:

.alter-merge table WeatherData policy retention softdelete = 1h recoverability = disabled

Of course, this only works for this particular type of timeseries database.

 

If you find this reply to help with your problem, please consider hitting the upvote or accept button...