Forum Discussion
Grouping by three conditions
- 5 years ago
No. As I said earlier, null, blank/empty, and space are different. Here is the same code but I've replaced your <space> with ""
let Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("jZG7DoJAEEV/ZbK1xczs8lg6Gwtb7QgFRqIkCAmPwr93ARMgA0qyJAS4h3Nn4lhpbdFYY9RBMcO5K4DR3YO7LtUra595+YCsaDL3gBgI3XtGoDBiG2lSyWHJwIlxfeYNfFkyikPUhoEh7dnAfYMGjt1j9/81RhQKCJutEqsAkgCeN0jb1QIuirLAMjqW/18CpQNZOGW3qYQU6I8U2JGz7sjJ+1NOilf1PS/T+i3lKWIpP4dtyuMowV6omXzfmnF1M4th8j+255qwLyHzEezcQU/yVJJ8AA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID = _t, Date = _t, Status = _t, Result = _t, #"Record last updated" = _t]), #"Changed Type" = Table.TransformColumnTypes(Source,{{"Date", type date}, {"Record last updated", type datetime}}), #"Replaced Value" = Table.ReplaceValue(#"Changed Type"," ","",Replacer.ReplaceValue,{"Status", "Result"}), #"Grouped Rows" = Table.Group(#"Replaced Value", {"ID"}, {{"All Rows", each _, type table [ID=nullable text, Date=nullable date, Status=nullable text, Result=nullable text, Record last updated=nullable datetime]}}), #"Added Status" = Table.AddColumn( #"Grouped Rows", "Status", each Table.Max( Table.FromRecords( { Table.Max( Table.SelectRows([All Rows], each [Status] <> ""), each [Date] ) } ), each [Record last updated] )[Status] ), #"Added Result" = Table.AddColumn( #"Added Status", "Result", each Table.Max( Table.FromRecords( { Table.Max( Table.SelectRows([All Rows], each [Result] <> ""), each [Date] ) } ), each [Record last updated] )[Result] ), #"Removed Other Columns" = Table.SelectColumns(#"Added Result",{"ID", "Status", "Result"}) in #"Removed Other Columns"
ks1 - you can read a bit about List.Buffer here. You need to pay attention to how your data is coming into the model. I rarely sort data in Power Query for a few reasons:
- DAX doesn't care. It will change the sort anyway, or change how it is viewed in the Data view in Power BI. Same with the Data model in Excel.
- The sort can change based on earlier steps in your model. For example, if you were to merge data or append data into your data even after the sort operation, Power Query will internally optimize the code and may do things out of sequence to arrive at the same result faster. But it may change the sort order when it does this. Remember, this is a table of data, not a spreadsheet, so sorting isn't usually relevant. You have to use Table.Buffer, List.Buffer, or Binary.Buffer. Note that the point of buffering is to isolate the table at that point from external changes. You can read the documentation on Table.Buffer here, which is pretty sparse, but the key point is it "isolating it from external changes during evaluation."
- List.Buffer works slightly differently and does hold the list in memory for faster evaluation. I use List.Buffer all of the time within a List.Contains() where the query is folded back to the server. It fully generates the list, then creates the IN operator in SQL very quickly, vs one list item at a time.
- Bottom line is I generally only sort in PQ in two cases:
- I am dumping the result to an Excel table. There sort order can matter, and the Table.Sort function is always the last step in my query.
- I am using some of the List operations and doing some sort of running total based on date, volume, or something else, and I always use List.Buffer in those cases to ensure that all of my separate lists maintain the same order so when I put them back into a table, they are in the same order and become part of the correct record. I do this pretty rarely too as 99% of the time, running totals are best done in DAX, and there is no concept of sorting data in DAX at all. It is 100% based on how you filter the data. I try to apply that same principle to Power Query, which is what my proposed solution is based on.
So you can use the sort mechanism, but you need to understand that it does not work as a sort in a spreadsheet. It is somewhat volatile, and you need to know the impact of that and if/when/where to buffer.
This is why I used table filtering when I did this. I am controlling exactly what I am returning vs relying on it being in a specific order. That isn't to say you cannot sort and use List.First. As I said in my post earlier, I like mahoneypat 's response and it is a clever use of sorting. You just have to fully understand all of the ramifications of that.
As for the error you got in my code when you replaced with null, I don't know. Here is my full code where I am replacing your " " <space> with a null, then I just changed the comparison to <> null and it worked first try. Perhaps you missed something in your copy and paste.
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("jZG7DoJAEEV/ZbK1xczs8lg6Gwtb7QgFRqIkCAmPwr93ARMgA0qyJAS4h3Nn4lhpbdFYY9RBMcO5K4DR3YO7LtUra595+YCsaDL3gBgI3XtGoDBiG2lSyWHJwIlxfeYNfFkyikPUhoEh7dnAfYMGjt1j9/81RhQKCJutEqsAkgCeN0jb1QIuirLAMjqW/18CpQNZOGW3qYQU6I8U2JGz7sjJ+1NOilf1PS/T+i3lKWIpP4dtyuMowV6omXzfmnF1M4th8j+255qwLyHzEezcQU/yVJJ8AA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ID = _t, Date = _t, Status = _t, Result = _t, #"Record last updated" = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Date", type date}, {"Record last updated", type datetime}}),
#"Replaced Value" = Table.ReplaceValue(#"Changed Type"," ",null,Replacer.ReplaceValue,{"Status", "Result"}),
#"Grouped Rows" = Table.Group(#"Replaced Value", {"ID"}, {{"All Rows", each _, type table [ID=nullable text, Date=nullable date, Status=nullable text, Result=nullable text, Record last updated=nullable datetime]}}),
#"Added Status" =
Table.AddColumn(
#"Grouped Rows",
"Status",
each
Table.Max(
Table.FromRecords(
{
Table.Max(
Table.SelectRows([All Rows], each [Status] <> null),
each [Date]
)
}
),
each [Record last updated]
)[Status]
),
#"Added Result" =
Table.AddColumn(
#"Added Status",
"Result",
each
Table.Max(
Table.FromRecords(
{
Table.Max(
Table.SelectRows([All Rows], each [Result] <> null),
each [Date]
)
}
),
each [Record last updated]
)[Result]
),
#"Removed Other Columns" = Table.SelectColumns(#"Added Result",{"ID", "Status", "Result"})
in
#"Removed Other Columns"
Again, if you will share your data via some sort of onedrive/dropbox link, I can connect this to your data and ensure it works.
How to use M code provided in a blank query:
1) In Power Query, select New Source, then Blank Query
2) On the Home ribbon, select "Advanced Editor" button
3) Remove everything you see, then paste the M code I've given you in that box.
4) Press Done
5) See this article if you need help using this M code in your model.