Forum Discussion

andrew_sxs's avatar
andrew_sxs
Regular Visitor
3 years ago
Solved

Power Query not removing all duplicates

Hi, first time using this so lets see how it goes...   I am working ith a file of nearly 1mil rows and 14 columns.   I am trying to remove duplicates within a column (LOCAL_ITEM_DESCRIPTOR) colum...
Need Help
Show and Tell
  • Anonymous's avatar
    Anonymous
    3 years ago

    Hi andrew_sxs  - I would like to suggest that you Filter you population in Power Query before the remove Duplicates step.  If possible include an example where there are:

    • no duplicates to remove
    • duplicate has been removed successfully
    • duplicate is not removed

    This will help you visible see what is happening.  It is possible that you need Trim or change case on the column with the duplicates because the following are not the same:  "ABC" <> "AbC" or "ABC ".

Anonymous's avatar
Anonymous
Not applicable
3 years ago

Hi andrew_sxs  - I would like to suggest that you Filter you population in Power Query before the remove Duplicates step.  If possible include an example where there are:

  • no duplicates to remove
  • duplicate has been removed successfully
  • duplicate is not removed

This will help you visible see what is happening.  It is possible that you need Trim or change case on the column with the duplicates because the following are not the same:  "ABC" <> "AbC" or "ABC ".

  • andrew_sxs's avatar
    andrew_sxs
    Regular Visitor
    3 years ago

    trim function before remove duplicate worked... thanks 🙂