Forum Discussion
Identifying and filtering duplicates
I have tryed to solve this with DAX on/off for a couple of weeks now with limited success.
I have a large table that each day is filled with subscription data by a script, sometimes the script runs twice, and create duplicate data with different timestamps (DateTime collum).
I wish to identify the duplicates and filter them out in my visuals.
My idea was to first identify if there was a duplicate entry on a given date by counting SubscriptionID, since this should be uniqe each day, adding a True/False collum called "Have Duplicate". Then I would add a True/False collum called "Is Duplicate" determing this using the "Have Duplicate" collum and setting all entries after the first DateTime as True.
I can do a count on SubscriptionID just fine, idetifing if there is duplicates, but as soon as I try to use DateTime to set the "Is Duplicate" it breaks.
I have a DateTable that is related to the Date collum, and I suspect that it is the one breaking it, buy have run out of ideas to solve this.
Best Regards
Ole
- Anonymous6 years ago
Did you tried selecting all the columns except the timestamp column? Do you maybe have some sample data?
4 Replies
- AnonymousNot applicable
If you want to remove the duplicates, you can achieve this using Power Query.
Select all the columns except the date column, click Remove Rows > select "Remove Duplicates"
- OplinkNew Member
It was the first thing I tried, but since they are not really duplicates, just duplicate data, with different timestamps (DateTime), havn't I been able to get that approach to work for me.
I might be doing something wrong ?
- AnonymousNot applicable
Did you tried selecting all the columns except the timestamp column? Do you maybe have some sample data?