Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get Fabric Certified for FREE during Fabric Data Days. Don't miss your chance! Request now

Reply
M0
Regular Visitor

Removing duplicates using running index number?

Hi, 

I'm trying to keep the SKU with the oldest date and remove all other rows.

In Excel, I could've done it in this way as in the last column Formula.  I then keep the rows with the first occurrence of the SKU.

Skarmklipp.PNG

 

Can I replicate the =COUNTIF(A2,$A$2:A2) formula in PQ?

If not what else should I do to keep the right row?

 

At first I thought this would be easy to solve using the Remove Duplicates in PQ.  To my surprise this does not work as it does in Excel.  In PQ, it does not automatically save the first occurrence and remove the others (like in Excel).  Why is this different?

I think this is especially true when I have a large table. I still don't understand why.

 

1 ACCEPTED SOLUTION
ppm1
Solution Sage
Solution Sage

Here's one way to do it in the query editor, with a GroupBy and the Table.Max function.  To see how it works, just create a blank query, open the Advanced Editor and replace the text there with the M code below.

 

let
    Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMjQwMFTSUTLUN9I3MjAyBjIdlWJ1kMQNYeJOqOLGMHFnmLgRkGOEUI8iboxDHMl8oL2xAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [SKU = _t, #"Delivery Date" = _t, Other = _t]),
    #"Changed Type" = Table.TransformColumnTypes(Source,{{"SKU", Int64.Type}, {"Delivery Date", type date}, {"Other", type text}}),
    #"Grouped Rows" = Table.Group(#"Changed Type", {"SKU"}, {{"AllRows", each _, type table [SKU=nullable number, Delivery Date=nullable date, Other=nullable text]}}),
    Custom1 = Table.TransformColumns(#"Grouped Rows", {{"AllRows", each Table.Max(_, "Delivery Date")}}),
    #"Expanded AllRows" = Table.ExpandRecordColumn(Custom1, "AllRows", {"Delivery Date", "Other"}, {"Delivery Date", "Other"})
in
    #"Expanded AllRows"

 

Pat

Microsoft Employee

View solution in original post

2 REPLIES 2
M0
Regular Visitor

Thank you very much Pat!

I didn't use this exact solution, but it did help me to get around this problem, which is what I wanted 🙂

ppm1
Solution Sage
Solution Sage

Here's one way to do it in the query editor, with a GroupBy and the Table.Max function.  To see how it works, just create a blank query, open the Advanced Editor and replace the text there with the M code below.

 

let
    Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMjQwMFTSUTLUN9I3MjAyBjIdlWJ1kMQNYeJOqOLGMHFnmLgRkGOEUI8iboxDHMl8oL2xAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [SKU = _t, #"Delivery Date" = _t, Other = _t]),
    #"Changed Type" = Table.TransformColumnTypes(Source,{{"SKU", Int64.Type}, {"Delivery Date", type date}, {"Other", type text}}),
    #"Grouped Rows" = Table.Group(#"Changed Type", {"SKU"}, {{"AllRows", each _, type table [SKU=nullable number, Delivery Date=nullable date, Other=nullable text]}}),
    Custom1 = Table.TransformColumns(#"Grouped Rows", {{"AllRows", each Table.Max(_, "Delivery Date")}}),
    #"Expanded AllRows" = Table.ExpandRecordColumn(Custom1, "AllRows", {"Delivery Date", "Other"}, {"Delivery Date", "Other"})
in
    #"Expanded AllRows"

 

Pat

Microsoft Employee

Helpful resources

Announcements
Fabric Data Days Carousel

Fabric Data Days

Advance your Data & AI career with 50 days of live learning, contests, hands-on challenges, study groups & certifications and more!

October Power BI Update Carousel

Power BI Monthly Update - October 2025

Check out the October 2025 Power BI update to learn about new features.

FabCon Atlanta 2026 carousel

FabCon Atlanta 2026

Join us at FabCon Atlanta, March 16-20, for the ultimate Fabric, Power BI, AI and SQL community-led event. Save $200 with code FABCOMM.