Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
Anonymous
Not applicable

Very simple 'Remove Duplicates' not working

I need to very simply remove duplicates for one column in a query.  I want that one column to be the matching ID so I can match a 'many-to-one' relationship.  I have tried doing all of the below steps but still end up with duplicates and no way to get any relationship to work except for many-to-many.

 

After Selecting the column I want to remove duplicates on and Right Clicking that column --->

1.Remove Errors 

2.Filtered Rows (blanks) [This probably isn't doing anything to help?]

3.Trimmed Text

4.Cleaned Text

5.Remove Duplicates

 

after performing all of these steps, I still get duplicates in this field.  What else can I do to 'normalize' these varchar text lines?

 

I cannot upload a .pbix (company policy) but I can email anyone personally the pbix or images of the problem.


This is in a 'SQL Server Database' import data option. 

The field is a VarChar Text field and it is a "Manufacturer Number" with lots of various characters.. IE: -=,.!#$%^&* etc....

 

1 ACCEPTED SOLUTION
Anonymous
Not applicable

I received a couple email requests to send the PBIX out.  As I was trimming down the 77mb file to a more manageable size, I tested the data in excel and also change my query.  There definitely was duplicates in that column.  I have fixed the problem by removing the 'Group By' function in the SQL code.  I did not know Power BI would retain that 'group by' functionality after data had been retrieved.  This makes me wonder what else Power BI retains from SQL code.

View solution in original post

3 REPLIES 3
v-yuta-msft
Community Support
Community Support

@Anonymous ,

 


After Selecting the column I want to remove duplicates on and Right Clicking that column --->

1.Remove Errors 

2.Filtered Rows (blanks) [This probably isn't doing anything to help?]

3.Trimmed Text

4.Cleaned Text

5.Remove Duplicates


 

How do you trim the column? Please check if there's still some values in the column contains blank such as "#$%as1   ". 

 

Community Support Team _ Jimmy Tao

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Anonymous
Not applicable

I received a couple email requests to send the PBIX out.  As I was trimming down the 77mb file to a more manageable size, I tested the data in excel and also change my query.  There definitely was duplicates in that column.  I have fixed the problem by removing the 'Group By' function in the SQL code.  I did not know Power BI would retain that 'group by' functionality after data had been retrieved.  This makes me wonder what else Power BI retains from SQL code.

Greg_Deckler
Super User
Super User

Sometimes this involves upper and lower case issues. I'd be interested in seeing example/sample data if you want to send in a private message.



Follow on LinkedIn
@ me in replies or I'll lose your thread!!!
Instead of a Kudo, please vote for this idea
Become an expert!: Enterprise DNA
External Tools: MSHGQM
YouTube Channel!: Microsoft Hates Greg
Latest book!:
Power BI Cookbook Third Edition (Color)

DAX is easy, CALCULATE makes DAX hard...

Helpful resources

Announcements
Sept PBI Carousel

Power BI Monthly Update - September 2024

Check out the September 2024 Power BI update to learn about new features.

September Hackathon Carousel

Microsoft Fabric & AI Learning Hackathon

Learn from experts, get hands-on experience, and win awesome prizes.

Sept NL Carousel

Fabric Community Update - September 2024

Find out what's new and trending in the Fabric Community.