Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
5 years ago
Solved

I can't remove duplicates.

I cannot remove duplicates.  I tried changing to uppercase and then removing duplicates.  It says I have 12056 rows and 12054 distinct values.  I tried filtering null values through the python script...
Need Help
  • Anonymous's avatar
    Anonymous
    5 years ago

    Yes.  I used trim and it worked,

mahoneypat's avatar
mahoneypat
Microsoft Employee
5 years ago

You've already done the uppercase thing, which gets a lot of people.  Do you have some values with hidden characters?  If so, try adding a trim step before your Remove Duplicates step.

 

Also, if not already, you can make a table visual with that column and add it again to get the count of rows, then sort by that to see which ones have >1 and are causing your issue.

 

Pat

Pat

 

Anonymous's avatar
Anonymous
Not applicable
5 years ago

However, I still cannot remove them.  Power Query does not work and it has too many rows and won't let me filter.

  • mahoneypat's avatar
    mahoneypat
    Microsoft Employee
    5 years ago

    The 1000 rows is just the preview.  The Remove Duplicates step works on the whole dataset when it loads.  Were you able to make a table visual that shows with values are duplicated?

     

    Pat

     

    • Anonymous's avatar
      Anonymous
      Not applicable
      5 years ago

      I made the table.  However, I cannot delete the values.  I am currently just using a many to many relationship.  Since the two sets of duplicates are the same throughout the row, the many to many relationship functions the same as a one to many.

      • v-kelly-msft's avatar
        v-kelly-msft
        Community Support
        5 years ago

        Hi  Anonymous ,

         

        Have you changed the below setting to the entire data set in power query before removing duplicates?

         

         

        Best Regards,
        Kelly

        Did I answer your question? Mark my post as a solution!