Forum Discussion
Remove rows based on same AND other column
- 1 year ago
The Source line, where you see the Base64 conversion, is what you get when you paste the data into a table in PowerBI/PowerQuery. You should replace the Source line with your own source. You can read the M code documentation for the Binary.FromText function, but that Base64 argument merely determines the type of Binary encoding to be used.
And either you didn't run the code, or I misunderstood your before and after.
Source Step:
And you wrote: remove rows 1,4 and keep 2,3,5,6,7
After running code:
Did I misunderstand what you wanted? Or did you not run the code to see what it did?
If I misunderstood, what rows did you wish to keep? As the code I provided clearly keeps 2,3,5,6 and 7
- Anonymous1 year ago
Hi FabvE ,
Thanks for ronrsnfld reply.
You can try the following codelet Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("ndBRC4IwEAfwr3LsOVCnlj1FBX6AICrEB7WjjXSTbeLXbwwJs0HQw8F2cL/9b0VBIrIiR4WVwRPqXgqNtrG3dWGVgaqWg4F2EA3b2V4WRGFAQ5rY81mjiki5Kgh9E7lUnb1c3ThvGNxR84cArqFGY1BNCJ0j1CGxL8dPKJ5DsYMSH3RwEAoYedvCiNAhmslI50bijHS50c1WzjUDqaCX6jmNbr+fX/+zB/34kNRBGx/kz0GzRY7yBQ==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [#"Unique ID" = _t, Operation = _t, #"From ID" = _t, #"Form Name" = _t, Timestamp = _t, User = _t]), AddFormID = Table.AddColumn(Source, "FormID", each if [Operation] = "CreateForm" then [From ID] else null), FillDownFormID = Table.FillDown(AddFormID,{"FormID"}), FilterCreateResponse = Table.SelectRows(FillDownFormID, each [Operation] = "CreateResponse"), AddCustom = Table.AddColumn(FilterCreateResponse, "HasCorrespondingForm", each List.Contains(FillDownFormID[FormID], [From ID])), RemoveInvalidResponses = Table.SelectRows(AddCustom, each [HasCorrespondingForm] = true), CombineTables = Table.Combine({Table.SelectRows(FillDownFormID, each [Operation] = "CreateForm"), RemoveInvalidResponses}), SortByUniqueID = Table.Sort(CombineTables,{{"Unique ID", Order.Ascending}}), RemoveHelperColumns = Table.RemoveColumns(SortByUniqueID,{"FormID", "HasCorrespondingForm"}) in RemoveHelperColumnsFinal output
Best regards,
Albert HeIf this post helps, then please consider Accept it as the solution to help the other members find it more quickly
- 1 year ago
Ok, after many tries and more fails I went with another approach which solved my problem.
I created an addinional query which contained only the created records:
let
Quelle = Table.SelectRows(Auswertung_raw, each [Operation] = "CreateForm")
in
QuelleI then used Merge queries to do a join between the initial table Auswertung_raw and the CreateForm-Table:
let
Quelle = Table.NestedJoin(Auswertung_raw, {"FormId"}, tblErstellteUmfragen, {"FormId"}, "tblErstellteUmfragen", JoinKind.LeftSemi),
#"Entfernte Duplikate" = Table.Distinct(Quelle, {"RecordId"})
in
#"Entfernte Duplikate"This gives me a table with all responses which have a corresponding CreateForm + the initial CreateForm-entries.
Nevertheless, thank you all - especially ronrsnfld- for your help and time! 👍
Ok, I did my best but still couldn't implement the code nor get it somehow to run.
My current query is this:
let
Quelle = Table.Combine({#"2023-10_12", #"2024-01_05", #"2024-06"}),
#"Remove Duplicates" = Table.Distinct(Quelle, {"RecordId"}),
#"Get JSON" = Table.AddColumn(#"Remove Duplicates", "JSON Block", each Json.Document([AuditData])),
#"Extend JSON Block" = Table.ExpandRecordColumn(#"Get JSON", "JSON Block", {"FormId", "FormName", "SourceApp", "FormsUserType", "OrganizationId"}, {"JSON Block.FormId", "JSON Block.FormName", "JSON Block.SourceApp", "JSON Block.FormsUserType", "JSON Block.OrganizationId"}),
#"Remove Columns" = Table.RemoveColumns(#"Extend JSON Block",{"AuditData"}),
#"Column isAnonymous" = Table.AddColumn(#"Remove Columns", "isAnonymous", each if Text.StartsWith([UserId], "urn:forms:anonymous") then true else false, type logical),
#"Sort Rows" = Table.Sort(#"Column isAnonymous",{{"CreationDate", Order.Descending}}),
#"SourceApp Mapping" = fBulkReplace(#"Sort Rows", tblSourceAppMapping, {"JSON Block.SourceApp"}),
#"FormsUserType Mapping" = fBulkReplace(#"SourceApp Mapping", tblFormsUserType, {"JSON Block.FormsUserType"})
in
#"FormsUserType Mapping"
- The JSON part splits a JSON part from the CSV source into separate columns for PBI, then removes the source column to keep the query simple and clean.
- An extra column isAnonymous is created to help identify anonymou replies.
- The last two actions replace values with mapped values from tables via a custom function fBulkReplace
Is there a way to put your nice code into this one?
I'm stuck.... 😐
You write, in part, that you couldn't get my code to run. What does that mean? When you pasted it into a blank query, did you not even get an error message? What happened?
- FabvE1 year agoHelper I
When I paste it 1:1 in a new query I just get the test data.
When I use the part after your source-row it runs forever and nothing happens.
s posted above I already hav a pre-filtered query which just needs this last step to get rid of the unwanted CreateResponse parts.
Atm my table looks like this:
Sorry for so many questions and postings. 😐
- ronrsnfld1 year agoSuper User
What you need to do is run your own query until you have a table which looks like your data table that you posted in your original question. Then, in the query I provided, you should be able to reference your own query in the Source line.
eg: Source = #"Your Query Name"
- FabvE1 year agoHelper I
Well I combined your code with mine and it's like this now:
let
Quelle = Table.Combine({#"2023-10_12", #"2024-01_05", #"2024-06"}),
#"Remove Duplicates" = Table.Distinct(Quelle, {"RecordId"}),
#"Get JSON" = Table.AddColumn(#"Remove Duplicates", "JSON Block", each Json.Document([AuditData])),
#"Extend JSON Block" = Table.ExpandRecordColumn(#"Get JSON", "JSON Block", {"FormId", "FormName", "SourceApp", "FormsUserType", "OrganizationId"}, {"JSON Block.FormId", "JSON Block.FormName", "JSON Block.SourceApp", "JSON Block.FormsUserType", "JSON Block.OrganizationId"}),
#"Remove Columns" = Table.RemoveColumns(#"Extend JSON Block",{"AuditData"}),
#"Column isAnonymous" = Table.AddColumn(#"Remove Columns", "isAnonymous", each if Text.StartsWith([UserId], "urn:forms:anonymous") then true else false, type logical),
#"Sort Rows" = Table.Sort(#"Column isAnonymous",{{"CreationDate", Order.Descending}}),
#"SourceApp Mapping" = fBulkReplace(#"Sort Rows", tblSourceAppMapping, {"JSON Block.SourceApp"}),
#"FormsUserType Mapping" = fBulkReplace(#"SourceApp Mapping", tblFormsUserType, {"JSON Block.FormsUserType"}),
#"Remove Rows" = Table.SelectRows(#"FormsUserType Mapping",(r)=>
r[Operation] = "CreateForm"
or (r[Operation] = "CreateResponse"
and List.Contains(Table.SelectRows(#"FormsUserType Mapping", each [JSON Block.FormId]=r[JSON Block.FormId])[Operation],"CreateForm")))
in
#"Remove Rows"It's running over 30mins now and doesn't seem to end...
When I select the step before (FormsUserType Mapping) it's quiet fast but the last step doesn't come to an end.