Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago

How to Easily Locate Duplicates in a Row After Merging Two Datasets ?

Good afternoon,   As the title suggests, I have combined two datasets based on a shared commonality (in this case account id #). Now that the datasets are merged, I have columns from the first dat...
Need Help
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Thanks for clarifying that query is case sensitive. I am pretty new to it, and I have figured out it'd be a little complicated to locate/identify all the duplicates - given that they don't have the same column names. 

 

I guess a follow up question would be what does someone in this situation do in order to identify a large number of duplicates? Going over each and every column manually is not feasible. 

edhans's avatar
edhans
Icon for Community Champion rankCommunity Champion
2 years ago

The key to a well performing model is few columns. The Vertipaq engine in Power BI loves very skinny, very tall tables. If you have too many columns to evaluate the issues with duplicates, then your main problem right now isn't duplicates. It is too many columns period.

I will spend hours (days!) ripping out columns out of a model to get more performance. The more columns you have the worse compression is, the larger your data dictionaries are, etc. I'm working with a model right now I dislike intensely because of how many columns the main fact table has, and it is only 24 columns. I really want to refactor it by doing some pivot operations and make the table taller and narrower, which I can do because of how the measures need to work, but I just need to find the time to do it.