Forum Discussion

grggmrtn's avatar
grggmrtn
Post Patron
5 years ago

Remove duplicates without breaking query folding?

Is there a way to remove duplicates without breaking query folding? I need the folding (for incremental refresh), but I have a ton of duplicates (I go from 3 0378 271 rows to 10 472 after removing du...
Need Help
grggmrtn's avatar
grggmrtn
Post Patron
5 years ago

Yeah, exactly.

V-lianl-msft's avatar
V-lianl-msft
Community Support
5 years ago

Hi grggmrtn ,

 

This can happen if the date ranges are set NOT up to as follows: [Date] >= RangeStart and [Date] < RangeEnd. Is your upper bound set to be equal to or less than or just less than?

Refer to this thread:

https://community.powerbi.com/t5/Desktop/Problem-with-incremental-refresh-on-power-bi-premium/td-p/735818 

 

Best Regards,
Liang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

  • grggmrtn's avatar
    grggmrtn
    Post Patron
    5 years ago

    Hi V-lianl-msft thanks for the reply...

     

    Your answer would make sense if I had actually set ranges, but I haven't gotten to setting the ranges yet, because query folding on my dataset breaks. The final step in my query (remove duplicates) breaks the folding.

     

    So my question is, how can I remove the duplicates without breaking the folding?

    • justinh's avatar
      justinh
      Advocate IV
      5 years ago

      grggmrtn amitchandak 

      I could use this as well, did you ever find out how to do this? 

       

      Table.Distinct() breaks the fold. How do we tell the servers to return a distinct list after all our previous steps? 

      • justinh's avatar
        justinh
        Advocate IV
        5 years ago

        Never mind. Table.Distinct() only breaks the fold if you use it to remove duplicates from a subset of the tables columns, instead of the whole table. Using Table.Distinct() on the entire table does not break the query folding.