Forum Discussion
Remove duplicates based on specific value
- 6 years ago
See the attached PBIX file. This is what I returned using Power Query.
EDIT: I see for this image I inadvertently formated TransactionEnd as Date. Should have been DateTime. I've fixed in the PBIX link, but not redoing the image. You may only want dates in your model, not times as well, so format that however you like. But Times are necessary for this to work since some of your records are on the same day and it needs a time to find the true latest value.
Here is what I did:
- Made sure the source table is not set to load.
- Created a reference to that table and called it "Successful Only" and kept only the Successful records and the ItemReference field.
- Created another reference to the source and called it "Final Table." Here is where the work starts.
- Starting with the Final Table, I merged the Successful Only table using the Item Reference field then expanded just the TransactionStatus field, which is now TransactionStatus.1. Now I either have null or Successful.
- Added a new "TransactionStatus Match" field that returns true or false if the status from the original table matches the new field. = [TransactionStatus] = [TransactionStatus.1]
- Filtered based on the TransactionStatus.1 field and the TransactionStatus Match field: each ([TransactionStatus.1] = null or [TransactionStatus Match] = true)
- I then grouped rows by the ItemReference and used the AllRows aggregation. You can see each embedded table has only those records related to the itemreference for that row.
- Then I filtered out to only return the record with the maximum date. This is a new column with the below formula:
- = Table.Max([All Rows],"transactionEnd")
- This returns a single record. I expanded that record and all fields except the TransactionStatus. We already have that.
- Removed all unnecessary columns, then changed the data types again. Using nested tables destroys the data types and you want them properly typed before you load.
- Load the Final Table into DAX for use.
- 6 years ago
Hi Anonymous
you can group on "itemReference", select "All" in the Operations and name the group-column "ItemData".
Then add this column:
if List.Contains([ItemData][TransactionStatus], "Successful") then [ItemData]{[TransactionStatus = "Successful"]} else Table.SelectRows([ItemData], (x) => x[transactionEnd] = List.Max([ItemData][transactionEnd])){0}Also see attached file
ImkeF , Can you help
- ImkeF6 years agoCommunity Champion
Hi Anonymous
you can group on "itemReference", select "All" in the Operations and name the group-column "ItemData".
Then add this column:
if List.Contains([ItemData][TransactionStatus], "Successful") then [ItemData]{[TransactionStatus = "Successful"]} else Table.SelectRows([ItemData], (x) => x[transactionEnd] = List.Max([ItemData][transactionEnd])){0}Also see attached file
- edhans6 years agoCommunity Champion
- ImkeF6 years agoCommunity Champion
Hi edhans
it's equivalent to:
each [transactionEnd]so referencing the "transactionEnd"-field in the currently iterated row/record.
But I cannot use this syntax sugar there, as this expression is written in an "AddColumns"-window. There it will be subject to an (outer) each already. So to to make it unambigous, I've used the "standard" function definition syntax ()=>.
https://docs.microsoft.com/en-us/powerquery-m/understanding-power-query-m-functions
Another practical example here: https://www.thebiccountant.com/2017/05/29/performance-tip-partition-tables-crossjoins-possible-powerquery-powerbi/