Forum Discussion
Practice Assessment Wrong? Power Query Sort and Remove Duplicates
- 1 year ago
Hi Tember,
Sorting and removing the duplicates is in general the correct answer. However, yes, it does not work all the time, and in my opinion, it is ambiguous, too as you stated.
In my projects, I never use this option, I rather go with inner joining with a duplicated version of the table having the max effective date grouped by customer id.
Regards,
Oktay
- 1 year ago
Yeah, the explanation sounds confusing. It could be a bug in the question/answer.
I would check out this blog: https://gorilla.bi/power-query/removing-duplicates/#advanced-techniques
Or write a native SQL query with ROW_NUMBER() PARTITION BY and the M function Value.NativeQuery() to maintain query folding if needed https://blog.crossjoin.co.uk/2021/02/21/query-folding-on-sql-queries-in-power-query-using-value-nativequery-and-enablefoldingtrue/
"Keep top rows" and "remove duplicates" are semantically different
I haven't seen the question in the assessment, so I'll pass this feedback over to the team who owns it.
Nevertheless, doing a "remove duplicates" doesn't keep or take into consideration the ordinal position of the rows. This means that if you are doing a "remove duplicates" on a subset of the columns from your table and not the whole table, then there's no guarantee that the result of the operation will be based on the ordinal position of the rows like you saw them in a previous step. That's where the more explicit definition of a explicit logic that you define where your logic can set a flag (or a value) for what rows to keep would be a far better approach. Typically you could reach this by using the rank, group by or merge methods in combination with a filter rows. If you have a specific example, we could go over it and how you could be more specific with your logic to keep exactly the rows based on your explicit logic
The Rank column (preview) function in Power Query Online is exciting!
Is it planned to support rank within partitions in the future?
And query folding? The indicator says "This step will be evaluated outside the data source".
Thanks!