Forum Discussion
Merge and update History table from Update table having new or updated values
- 4 years ago
I'd recommend checking out these relevant resources for the self-referencing logic:
Self Referencing Tables in Power Query
Inserting text manually in a custom column and should be visible on refresh of the report
To do the update, I appended the tables together, grouped by ProductKey taking the latest version of the data, expanded the non-manual columns, and then merged back in the manual columns from the beginning of the query.
Please see the attached.
I'd recommend checking out these relevant resources for the self-referencing logic:
Self Referencing Tables in Power Query
Inserting text manually in a custom column and should be visible on refresh of the report
To do the update, I appended the tables together, grouped by ProductKey taking the latest version of the data, expanded the non-manual columns, and then merged back in the manual columns from the beginning of the query.
Please see the attached.
- Anonymous4 years agoNot applicable
Hi AlexisOlson ,
This is so consice, but brilliant! 🙂
Can you help me with my below queries?
On opening Query editor, Updateable Table gives the following error on this line:
Formula.Firewall: Query 'UpdateableTable' (step 'AppendedQuery') references other queries or steps, so it may not directly access a data source. Please rebuild this data combination.AppendedQuery = Table.Combine({AddedVersion, NewData})- I see you have joined ProductKey column with Product column in step "Grouped Rows".
#"Grouped Rows" = Table.Group(#"Appended Query", {"ProductKey", "Product"}, {{"All Rows", each Table.Max(_, "Version"), type record}})In real data, the Product column may not be unique i.e. ProductKeys may have same Product names. How do we handle that? Is it possible to only use the ProductKey which is unique?
- Is there a way to make the column names selection dynamic, rather than hard-coded? There could be many standard columns in UpdateableTable.
#"Expanded All Rows" = Table.ExpandRecordColumn(#"Grouped Rows", "All Rows", {"Status", "Date", "Forecast", "Out of stock"}, {"Status", "Date", "Forecast", "Out of stock"})- Can Manual entry column names in last step be made dynamic?
Expanded Expanded All Rows = Table.ExpandTableColumn(#"Merged Queries", "Expanded All Rows", {"Manual1", "Manual2"}, {"Manual1", "Manual2"})Best,
Sifar
- AlexisOlson4 years agoSuper User
Read this for info on Formula.Firewall:
https://www.thepoweruser.com/2019/03/12/data-privacy-and-the-formula-firewall/
(The simplest solution is the 'Always ignore Privacy Level settings' under Query Options > Privacy.)
You can use whatever column or combination of columns you'd like to define what you consider unique. You can certainly drop the Product column. and only group over ProductKey.
Here's a bit more dynamic version of the query where I've added steps to generate lists of column names.
let Source = Excel.CurrentWorkbook(){[Name="UpdateableTable"]}[Content], #"Changed Type" = Table.TransformColumnTypes(Source,{{"ProductKey", Int64.Type}, {"Product", type text}, {"Status", type text}, {"Date", type date}, {"Forecast", Int64.Type}, {"Out of stock", type text}, {"Manual1", type text}, {"Manual2", type text}}), #"Added Custom" = Table.AddColumn(#"Changed Type", "Version", each 0, Int64.Type), #"Appended Query" = Table.Combine({#"Added Custom", NewData}), #"Grouped Rows" = Table.Group(#"Appended Query", {"ProductKey"}, {{"All Rows", each Table.Max(_, "Version"), type record}}), ColumnNames = List.Select(Table.ColumnNames(NewData), each _ <> "ProductKey"), #"Expanded All Rows" = Table.ExpandRecordColumn(#"Grouped Rows", "All Rows", ColumnNames, ColumnNames), #"Merged Queries" = Table.NestedJoin(#"Expanded All Rows", {"ProductKey"}, #"Changed Type", {"ProductKey"}, "Expanded All Rows", JoinKind.LeftOuter), ManualColumns = List.Difference(Table.ColumnNames(#"Changed Type"), Table.ColumnNames(NewData)), #"Expanded Expanded All Rows" = Table.ExpandTableColumn(#"Merged Queries", "Expanded All Rows", ManualColumns, ManualColumns) in #"Expanded Expanded All Rows"- Anonymous4 years agoNot applicable
Thanks AlexisOlson 🙂
- I used a similar method to create a list query that generates a list of column names from the Updateable table and then use List.Skip() to skip the ProductKey column and List.Difference().
- One question: Is it possible to make the column datatype transforms more dynamic for this line?
#"Changed Type" = Table.TransformColumnTypes(Source,{{"ProductKey", Int64.Type}, {"Product", type text}, {"Status", type text}, {"Date", type date}, {"Forecast", Int64.Type}, {"Out of stock", type text}, {"Manual1", type text}, {"Manual2", type text}}),I tried Table.Schema, and then used the "Name" & "TypeName" but couldnt wrap my head around how to get it in the same format.
SelectColumns = Table.SelectColumns( Table.Schema( ColTypes ),{"Name","TypeName"},MissingField.UseNull),where "ColTypes" is the Table.TransformColumnTypes().
P.S: Strange enough, when the query picks up the data types from the Excel Updateable table, most of the datatypes show up as "ABC 123" - though i have formatted the each of the table columns properly. Why is the query not able to recognize the data types from the table?