Forum Discussion
Auto and Manual Categories
Einomi this is standard self-referencing query stuff. But you must have some unique reference for every line of your statement (or date column will be enough if you load transactions for the whole day). I added index column but just for demo. In short: you have some table with "old" transactions in Excel (with auto and/or manual entries) - load it to PQ, load your source data, this way or another filter new transactions only (this is job for your unique identifier), apply your transformations (including auto category and subcategory stuff), combine old and new transactions and load resulting table back to Excel. Now you may manually edit that table and won't loose this info unless you manually delete rows from your table.
let
auto = Function.Invoke(
Record.FromList,
List.Reverse(
Table.ToColumns(
Table.CombineColumnsToRecord(
Excel.CurrentWorkbook(){[Name="auto"]}[Content],
"rec",
{"Category", "Subcategory"}
)
)
)
),
loaded_transactions = Table.Buffer(Excel.CurrentWorkbook(){[Name="loaded"]}[Content]),
loaded_ref = Record.FromList(List.Repeat({false}, Table.RowCount(loaded_transactions)), List.Transform(loaded_transactions[ref], Text.From)),
new_transactions = Table.AddColumn(
Table.SelectRows(statement, (x) => Record.FieldOrDefault(loaded_ref, x[ref], true)),
"cat",
(x) => Record.FieldOrDefault(auto, x[Counter Party], [])
),
xpand = Table.ExpandRecordColumn(new_transactions, "cat", {"Category", "Subcategory"}, null),
z = loaded_transactions & xpand
in
z
Hi,
Looks we are getting there. I have tried your file. However, the auto categorization does not look like to work.
I have tried to add one row to the mapping table and it did not refresh in the auto query, I even tried to change one subcategories in the mapping table and still did not reflect that change in the auto query.
The manual input where kept, I really need to be able to manual input and if null then the auto category, manual should override auto.
Finally, I am sure this is a bit basic but, I have read online in some specialized blogs that adding an index column may not be the best way to go as it recalculates and can mess up the data, what is your intake on this ?
Appreciate your time and your help
- AlienSx1 year agoSuper User
try this
let // load mapping table into dictoionary record auto = Function.Invoke( Record.FromList, List.Reverse( Table.ToColumns( Table.CombineColumnsToRecord( Excel.CurrentWorkbook(){[Name="auto"]}[Content], "rec", {"Category", "Subcategory"} ) ) ) ), // load old transactions, apply mapping to nulls loaded_transactions = Table.Buffer(Excel.CurrentWorkbook(){[Name="loaded"]}[Content]), add_mapping = Table.AddColumn( loaded_transactions, "map", (x) => if x[Category] is null and x[Subcategory] is null then Record.FieldOrDefault(auto, x[Counter Party], []) else Record.SelectFields(x, {"Category", "Subcategory"}) ), remove_old_columns = Table.RemoveColumns(add_mapping, {"Category", "Subcategory"}), // load new transactions loaded_ref = Record.FromList(List.Repeat({false}, Table.RowCount(loaded_transactions)), List.Transform(loaded_transactions[ref], Text.From)), new_transactions = Table.AddColumn( Table.SelectRows(statement, (x) => Record.FieldOrDefault(loaded_ref, x[ref], true)), "map", (x) => Record.FieldOrDefault(auto, x[Counter Party], []) ), // combine old and new, expand cat/subcat columns combine = remove_old_columns & new_transactions, expand = Table.ExpandRecordColumn(combine, "map", {"Category", "Subcategory"}, null) in expandRegarding index usage: again, I added index just for demo purposes. It plays "unique transaction reference" role in my file. Don't use index. Maybe you don't need unique ref at all - it depends upon the data you are loading. E.g. if you load historical data then you may use date column as you key. You need _something_ that identifies your transactions as "old" or "new".
- Einomi1 year agoHelper V
Whoa, we are getting so close 😊😁. I have managed to replicate this into my real data set. However, I struggled in 2 points.
You have 2 queries the statement one and the auto one, and I will use the same names to describe my steps and please tell me where I missed something.
1. Open blank workbook > Get Data > From SharePoint folder
2. Do all my transformations and add a unique column by merging a few ones (like you did with [ref])
3. Load back this query to Excel, add on that table the columns Category and Subcategory
4. Load back this query to PQ
5. Load my mapping (the one you called auto) table to PQSo I am ending up having three queries
statement (the one I got from SP and loaded to Excel)
auto (my mapping table loaded to PQ)
statement (2) (the one loaded from Excel with the two new columns Category and Subcategory)
Then I am stuck because in your case, your original query, also known as statement was loaded as a connection but if do the same, I will have an issue with the auto query at the step loaded_transactions, so how do we overcome this ?
Also is there a way to remove the [ref] column at the final step of the auto query ? I have tried but it messed ? Alternatively, can we do a merge query instead of creating a [ref] column ?Really appreciate your help.
- AlienSx1 year agoSuper User
Einomi just continue editing your original query statement (using statement(2) and auto). This article by Matt Allington illustrates one of possible self-referencing setup.
Regarding ref: well, Left Anti join may help