Forum Discussion
Auto and Manual Categories
Hi,
Looks we are getting there. I have tried your file. However, the auto categorization does not look like to work.
I have tried to add one row to the mapping table and it did not refresh in the auto query, I even tried to change one subcategories in the mapping table and still did not reflect that change in the auto query.
The manual input where kept, I really need to be able to manual input and if null then the auto category, manual should override auto.
Finally, I am sure this is a bit basic but, I have read online in some specialized blogs that adding an index column may not be the best way to go as it recalculates and can mess up the data, what is your intake on this ?
Appreciate your time and your help
try this
let
// load mapping table into dictoionary record
auto = Function.Invoke(
Record.FromList,
List.Reverse(
Table.ToColumns(
Table.CombineColumnsToRecord(
Excel.CurrentWorkbook(){[Name="auto"]}[Content],
"rec",
{"Category", "Subcategory"}
)
)
)
),
// load old transactions, apply mapping to nulls
loaded_transactions = Table.Buffer(Excel.CurrentWorkbook(){[Name="loaded"]}[Content]),
add_mapping = Table.AddColumn(
loaded_transactions,
"map",
(x) => if x[Category] is null and x[Subcategory] is null
then Record.FieldOrDefault(auto, x[Counter Party], [])
else Record.SelectFields(x, {"Category", "Subcategory"})
),
remove_old_columns = Table.RemoveColumns(add_mapping, {"Category", "Subcategory"}),
// load new transactions
loaded_ref = Record.FromList(List.Repeat({false}, Table.RowCount(loaded_transactions)), List.Transform(loaded_transactions[ref], Text.From)),
new_transactions = Table.AddColumn(
Table.SelectRows(statement, (x) => Record.FieldOrDefault(loaded_ref, x[ref], true)),
"map",
(x) => Record.FieldOrDefault(auto, x[Counter Party], [])
),
// combine old and new, expand cat/subcat columns
combine = remove_old_columns & new_transactions,
expand = Table.ExpandRecordColumn(combine, "map", {"Category", "Subcategory"}, null)
in
expand
Regarding index usage: again, I added index just for demo purposes. It plays "unique transaction reference" role in my file. Don't use index. Maybe you don't need unique ref at all - it depends upon the data you are loading. E.g. if you load historical data then you may use date column as you key. You need _something_ that identifies your transactions as "old" or "new".
- Einomi1 year agoHelper V
Whoa, we are getting so close 😊😁. I have managed to replicate this into my real data set. However, I struggled in 2 points.
You have 2 queries the statement one and the auto one, and I will use the same names to describe my steps and please tell me where I missed something.
1. Open blank workbook > Get Data > From SharePoint folder
2. Do all my transformations and add a unique column by merging a few ones (like you did with [ref])
3. Load back this query to Excel, add on that table the columns Category and Subcategory
4. Load back this query to PQ
5. Load my mapping (the one you called auto) table to PQSo I am ending up having three queries
statement (the one I got from SP and loaded to Excel)
auto (my mapping table loaded to PQ)
statement (2) (the one loaded from Excel with the two new columns Category and Subcategory)
Then I am stuck because in your case, your original query, also known as statement was loaded as a connection but if do the same, I will have an issue with the auto query at the step loaded_transactions, so how do we overcome this ?
Also is there a way to remove the [ref] column at the final step of the auto query ? I have tried but it messed ? Alternatively, can we do a merge query instead of creating a [ref] column ?Really appreciate your help.
- AlienSx1 year agoSuper User
Einomi just continue editing your original query statement (using statement(2) and auto). This article by Matt Allington illustrates one of possible self-referencing setup.
Regarding ref: well, Left Anti join may help
- Einomi1 year agoHelper V
Thanks for your answer. I am aware you are quite busy hence the quick reply 😁
However, can I send you my file via DM, because we are so so close and do not want to miss out.
After, I'll make a post to explain to new members or even advanced ones this rocking method 🙂
Please we are so close and I must miss something