Forum Discussion
Remove duplicates that contains null in one particular column ,but both contains null keep one
- Anonymous2 years ago
Hi Analyst_in_prog ,
Here is my sample data:
Here is the whole M code in the Advanced Editor:let Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WclTSUfLJLweSFhYGQLJAKVYHIoom5ISk0BBFFE3IGUmhEYoompALTMgYRQimFyHqClNoAiRL8Yt5ZKZnoIu7IZlqhiqMLuYOFbM0ISwGswpZ3AOm1hS/mCdMzAxVDG4mmjjU+XDhWAA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [comments = _t, #"priority level" = _t, reportID = _t, eventType = _t]), #"Changed Type" = Table.TransformColumnTypes(Source,{{"comments", type text}, {"priority level", type text}, {"reportID", Int64.Type}, {"eventType", type text}}), #"Replaced Value" = Table.ReplaceValue(#"Changed Type","",null,Replacer.ReplaceValue,{"priority level"}), #"Grouped Rows" = Table.Group(#"Replaced Value", {"reportID"}, {{"AllNulls", each List.NonNullCount([priority level]) = 0, type logical}}), #"Merged" = Table.NestedJoin(#"Replaced Value", {"reportID"}, #"Grouped Rows", {"reportID"}, "GroupedData", JoinKind.LeftOuter), #"Expanded GroupedData" = Table.ExpandTableColumn(Merged, "GroupedData", {"AllNulls"}, {"AllNulls"}), #"Removed Duplicates" = Table.Distinct(#"Expanded GroupedData", {"reportID", "priority level"}), #"Filtered Rows" = Table.SelectRows(#"Removed Duplicates", each not ([priority level] = null and [AllNulls] = false)) in #"Filtered Rows"And the final output is as below:
My overall idea is to first add a column that marks the reportIDs where the priority level are both null as TRUE, then delete duplicate rows for both the reportID and priority level columns, and finally delete the rows where the priority level=null and marked as FALSE.
Best Regards,
Dino Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Hi Analyst_in_prog ,
Here is my sample data:
Here is the whole M code in the Advanced Editor:
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WclTSUfLJLweSFhYGQLJAKVYHIoom5ISk0BBFFE3IGUmhEYoompALTMgYRQimFyHqClNoAiRL8Yt5ZKZnoIu7IZlqhiqMLuYOFbM0ISwGswpZ3AOm1hS/mCdMzAxVDG4mmjjU+XDhWAA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [comments = _t, #"priority level" = _t, reportID = _t, eventType = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"comments", type text}, {"priority level", type text}, {"reportID", Int64.Type}, {"eventType", type text}}),
#"Replaced Value" = Table.ReplaceValue(#"Changed Type","",null,Replacer.ReplaceValue,{"priority level"}),
#"Grouped Rows" = Table.Group(#"Replaced Value", {"reportID"}, {{"AllNulls", each List.NonNullCount([priority level]) = 0, type logical}}),
#"Merged" = Table.NestedJoin(#"Replaced Value", {"reportID"}, #"Grouped Rows", {"reportID"}, "GroupedData", JoinKind.LeftOuter),
#"Expanded GroupedData" = Table.ExpandTableColumn(Merged, "GroupedData", {"AllNulls"}, {"AllNulls"}),
#"Removed Duplicates" = Table.Distinct(#"Expanded GroupedData", {"reportID", "priority level"}),
#"Filtered Rows" = Table.SelectRows(#"Removed Duplicates", each not ([priority level] = null and [AllNulls] = false))
in
#"Filtered Rows"
And the final output is as below:
My overall idea is to first add a column that marks the reportIDs where the priority level are both null as TRUE, then delete duplicate rows for both the reportID and priority level columns, and finally delete the rows where the priority level=null and marked as FALSE.
Best Regards,
Dino Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.