Forum Discussion

aleggore's avatar
aleggore
Frequent Visitor
4 years ago

Duplicate Values

I am finding duplicate values in my dataset after I try to refresh. There are no duplicate values showing when I run my query independently or in Power Query before I load to PowerBI. Has anyone seen this happen before and know of any resolution? 

3 Replies

  • Greg_Deckler's avatar
    Greg_Deckler
    Community Champion

    aleggore Extremely vague. Power Query only loads the first 1000 rows as preview. Are you appending some query to an existing table? Also, Power Query and DAX have different rules for what they consider duplicates. For example, DAX doesn't care about case while Power Query does. So, need more to go on in terms of what is going on.

     

    Sorry, having trouble following, can you post sample data as text and expected output?
    Not really enough information to go on, please first check if your issue is a common issue listed here: https://community.powerbi.com/t5/Community-Blog/Before-You-Post-Read-This/ba-p/1116882

    Also, please see this post regarding How to Get Your Question Answered Quickly: https://community.powerbi.com/t5/Community-Blog/How-to-Get-Your-Question-Answered-Quickly/ba-p/38490

    The most important parts are:
    1. Sample data as text, use the table tool in the editing bar
    2. Expected output from sample data
    3. Explanation in words of how to get from 1. to 2.

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi   aleggore ,

    Can't tell without seeing your query or data. Maybe you're expecting an inner join but used another join type?

    "There are no duplicate values showing when I run my query independently or in Power Query before I load to PowerBI. "

     

    The query editor only loads the first 1,000 records by default. You can force it to load all, then you'll get the error in the editor before closing.

     

    you can test for duplicates

    = Table.IsDistinct( table )

    = Table.IsDistinct( table, { "pk_column1", "pk_column2" } )

    or remove duplicates.

    = Table.Distinct( table )

    = Table.Distinct( table, { "pk_column1", "pk_column2" } ) 

     

    It might have to do with Power BI's case-insensitivity:Power BI And Case Sensitivity 

    For testing, you could add an index column (in Power Query) to check how the affected rows look like before and after loading.

     

    If it doesn't help, please provide more details with the error.

     

     Best Regards
    Community Support Team _ Polly

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

     

     

    aleggore

    • aleggore's avatar
      aleggore
      Frequent Visitor

      Turns out I was creating columns in my dataset in PowerBI that I should have been creating in Power Query. Making these changes, removed this duplicate situation. It was very odd. I'm a pretty new user so I'm still working through all the kinks 🙂