Forum Discussion
Remove duplicates by prioritizing rows based on another column
- 4 years ago
Okay, please try this. I have included comments that explain each step.
BEFORE:
The goal is to remove rows 1, 3, 6, 8 and to have all columns present in the result.
RESULT:
SCRIPT:
There are two different options in the grouped step along with scenarios for when one would apply as opposed to the other, based on the specifics of the source data. Currently, option 2 is in use. To switch to option 1, add two forward slashes in front of Table.LastN and remove the two slashes at the beginning of let varTable and Table.FirstN.
let Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("jcy5DcAgEETRXjZGaHbBVwi+irDovw0vJiFAmOQjxGOeh0IIZIgB6BHiroW3YCsQ1kt+pGT+ISNXKiu9UTcEGaL1n40x5n/F7sfZGJ0q6HtwHoIMp10qOxV7XndjdB2CDK/dKKUX", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Latest = _t, #"NVE-Nr." = _t, Item = _t, Date = _t, #"LHM-Nr." = _t, Index = _t]), grouped = Table.Group( Source, // Column(s) containing the values from which you'd like duplicates removed. {"NVE-Nr."}, { { "Table", // Name of the new column. each //--------------------------------------------------------- // Option 1: Sort descending, then select the first result. //--------------------------------------------------------- // This will work if there is a maximum of only two rows per NVE-Nr. //let varTable = Table.Sort ( _, {{"LHM-Nr.", Order.Descending}}) in //Table.FirstN ( varTable, 1 ), //--------------------------------------------------------- // Option 2: Select the last LHM-Hr for each NVE-Nr. //--------------------------------------------------------- // This will work if the row to keep always appears last in the group. Table.LastN ( _, 1 ), type table } } ), expand = Table.ExpandTableColumn ( grouped, "Table", // Expand the tables in this column List.Difference ( // New column names Table.ColumnNames ( // are the column names Table.Combine ( grouped[Table] ) // in the nested tables ), Table.ColumnNames ( grouped ) // that do not appear in the grouped table. ) ) in expand
I might have a simpler solution. From this data set, I can see that there are only duplicates in NVE when LHM-Nr. is empty. In theory, you could just remove all rows where LHM-Nr. is empty and it would consequently remove duplicates from NVE. For this, you can click on the LHM-Nr. column, and on the top of Power Query click on Remove Rows+, then Remove Blank Rows. This should work.
Otherwise, you could just do what you said and sort by loading list. Since the rows move as a whole, you wouldn't mix any infos and could then remove duplicates, since the duplicate row without empty values would be on top. Hope this helps.
- Anonymous4 years agoNot applicable
Hi, JoaoPereira .
Thanks for the tip. Unfortanutely this will lead to data loss since I have data of over 12 000 rows.. Since I imported data from pdf and did the cleaning, I have some loading lists where Power Query didn't catch LHM numbers but only NVE numbers. If I delete all rows with empty LHM numbers it will delete this information as well, which I would like to avoid.
That's why I need a delicate solution so that it wouldn't lead to any information loss.
Regards,
- JoaoPereira4 years agoFrequent Visitor
Alright, the solution I came up with is a bit more complicated than it probably needed to be but it's all I got. Here is what I recommend:
Duplicate the LHM column. In the Transform section, divide column by Number of Characters (position 26) and by specific character (.). This will leave you with two columns, one with a 4 digit number (3422 for example) and a second column with the word pdf for the separator. Delete the pdf column.Now, duplicate the NVE column.
Now, join the duplicate LHM column with the 4 digits and the NVE duplicate column. Since every NVE is unique, the only duplicate values in this new column will be when the 4 digits from the LHM are identical AND when the NVE column was blank, therefore not adding anything to the values.
Finally, click on the joined column from both duplicates and erase duplicates. From here, you can delete the joined column since the functions work without it. There you have it, you'll have a data set with duplicates but only when NVE is not empty.