Power BI is turning 10! Tune in for a special live episode on July 24 with behind-the-scenes stories, product evolution highlights, and a sneak peek at what’s in store for the future.
Save the dateEnhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.
Hi,
I download bank statements in CSV formats, and consolidate them in PQ. These are my steps
1. Load folder from SharePoints (using SharePoint.Contents)
2. Make all my transformations then load to Excel as table
3. Add a few columns in Excel, among them Category and Subcategory
4. I create a mapping table in Excel (attached) then load it PQ
5. I write some M code to reconcile some transactions automatically with Category and Subcategory.
My issue is, that I cannot map all transactions automatically, and I need some manual input and need to reconcile Category and Subcategory manually for the ones not able to do it automatically
What I need
I give sample transactions
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("rVxNjxy5Df0rDZ93tCKpz6MNb4IsECCIg1ycPXTs9u4g42ljZtaJ8+tDzbRUKqsmFlm+jV1dbH08Pj6SUr99+wLcjzb8iBbdix9e/PF8/vXmxH/85fjl4+n24XB3ene6/nx6f/hwd/54aI8xeTSUXvzyw9sXGDsDPx9vT4c3H68ffuN/vDneHO++HD5dbD2cD6vHV+BtNPbJSD+Klx+P/z3fPjuK9pgI0OT8aAByb+DTp/8zi/qUQsgmwOPrtn/9r+d/nu4eDq/uzv++3ZzGVx+4ggz2MhGbOkN/O93fHJ8dR30KwG/D0zQwdK//9PH65svh9fHz9f3mMNbPryhbqKPo9+TlzfW70+Hn82+39+ft+Xz9iSsIsU5IhQ8CH4x/mpLFHh/8HYfX59M2OpaHPAJIdQT93vz9/P744XxbPvOn28/nMu7OQvf0CmJwxrunZUWVCQSbjMOnaYR9S4oRXQW7V3tMtHbLiGI8RITVlNWDDm3OFXRet08hWONGFhBsExX3g3Gnp9GGzoYttL28vnt1++p5JqmP0YM3aYOJZh3GxUKFNK7Cm+Ovz6zA5Qm7fTB0YTGNr0LIZBC/Bw0maBBPeyBFbissiDyFI1Tlj9Wkvr2egSeR49PXkyIqJceh8el9AA13UvQNzU4BRgIgk54MQNSA0Scy7gKpfgleHe/e3Ry/3D/+efuvw4fT6b4sXff/VwAmPQkDC3vAhD776pJR49PEENhUF7PLiCwuwshsGq512Gi7n8vrf/xuLYY3hz9c3z2Dyq8/wlqDw1rCMQb8+frd3fn+/OFh21D/+ApcBhP99xlRCBf6Xjmaam4pZuOejIGKyMnHyh0r+GlG4zyvURoROEsEQMkZwFFFzclTRA4JFwCuFuPl7++vHw4//ecT+8v2VL76BDMqeyXCyEnzATKmqi0hiKfiYsqVFcBqMet5Py6acjdmHasWCHvkZeCIH1LTT064IoFq0C7f7+RAtzl2usvJJ4DIGr86m9OZIG8yNd93Ug9Bwtzra6eOFmST7aTHDkMOm24gxZQoLjlp0aLLzkrUC3lqCWUQgwvcMgbbr8a3xQ8U2eRyiy5OTzop8kLmpkvFK+mQc8iLEIN+FrMBnJO3ZmC1DOK8mloq2kNCFeH8ki5F+cYGjv3kR5edKzRQdL6G6qSgLMehNbZ45FQkHjnZqLSJ+j3BnFKXCOu8DIuUumysVwDMIUUT/EjCs0I7cKoANLLW7PsZXS1ggdTPIZjkxgA0WbLy1uSl/KeCQoFy3NjB6QpR9NWVkHYxFXqGQWr1N0UgTDZUv0RQWQBmBuNhxIKGZZBY3rs4etnc7jrHhJf9GNcl2+sxmujG/Z3AZg5oIo77MclTjr0ib3iFpCTFjmk3vGNa2DjvTYSN4CcPoymaRKMgmcrFgwnUirBKKYJElSRRwVGOw2cDQm9AU0MFv03Ys5LCuWTQ69Ud2MzA2HD0qd2IxtOIqXlYOo55yW+spBhU7GJNeK+1ldwUMfdVrmGo+3lH96Xw6Fr49PpB+NjiUUlPvYwxMEZXu1m2f31mV1OdfHExr3IxxKUU78Sj58ixArWXeiiWAnytU/QG5pV65lwYG9vpVgHSUoO3u9AAwOtJG5YUlbsYoavcLaZmJWIOS2+vX1tNWCcXa1wsxS3xYBLnyGEDKLJ4AEsG0++1vMAbIGyhfh62iK3n4lSwZcmFi+Tag7jSagwjDUzmZYihhYZ+YwSVGJejgUWy6beFcqauLyhfVSjtizC6jKw+ja2D4eXr6UrbNoyBRchHVPFpk8LbMDO+3cauyrcEWtd3Fd0m1bz1JsUNCpuKLj6kthN5J3+BYwKrBZmw1xhbuxQzVouiAHzErX2eFrgYs4lLXUQMFECb+8KKSH6UZmUaSVTmbAwv50cjAvqxOTQ9SepxRFbIaYzbksQtt6IC6iw4pKZlduEKy/EiO3KYsNIFpdFgxykJ6iRLI3Wv0z0Wt8exTAEVDcAoG1SuHx20gk3ULy066k99aM0wgaSuFQPCckXG5RQada/PsTsGy6IwtO0FabxGX4qQqa0CSPmPAgd8DOMEZDmQo/7syDINuRbz1FRykG9HZJGcxuXUdKRiF1bEy8obm03GjY2dDSuUmAyXNEi+sZGpsLoZdAa+ndmj887g0kYCTTygnGp1fAVMMYHGKmiLNhfigTzW4F6oZnlbUb16bKC08wTa+eByxKFEaxCRcGiQ8nvAzflOfxxAtb1IRZhS64WBPFgniLVZuXJ1SQebxV/Xxhd7GZbyJm2gVFW3t8Eacq0sJyZzKOx1kZJWCg5fa/7qLSUOhD5pqRc8QM2iV0s5WVwNaeS6iW5DWTIK45pPvEppOSKmoJaYbavP5z3uiN5uuqOUWVIOW2amT8z1bbCeX6bLIN6E8e2ZjhGvZD2IksRfzB+27TROr2AUrcRVGrSLDFhWmhBGZpqIvpYzUo9jiNDwfOiKfXINwll6o9cdyCRqOcwqCquyB8tRsB6Gs0qmezyMRuF7jSg04lshX5C2EtXusTISBqpN0+JBqAilJRgvSRlKY5dzzKbgmvMsBhRSi9r5qQK7ZkqWkljou42owQki5720HOZGAakxq5vlHLnkzXI7qzbWnGIrHkvroTUtUM8fMVAXUZZ9mD53CsvZK9sZmJIEuV7/KcFZPwlC77tMDmVR3rHojzQOQ1Lbj63tu/ILcXRidRlz0xuoJePUOkCQZcDMDIdamE9Kl4JyOgXiuB7Tea1ntWA33ENwXcj1xYo9TEX86ia4prP8cg6ZWhdsz1g4qG1663QuBJQNUquEiQ2Q5WzMLQKyGZjsdPiMrb+QtYydazMdhHwdy5kfbNVRIU8QsG8sp111IYuwAXO1AVN0mdq1JyuaeGbtQa2sIZw25BSbVrXaecNyudLSHmazYcsZpRUn72hrNJMoziUNCuPrgssalroKkXZJV2UmadTLiUV2Gp1BdliYRe3I8wLx0G6AWfU6PLeYMy6VL5VkuzNIQIHExkQ0xXHb9mUVvHXHTJlt/IaxackfOPGhDcqczNsT5+24HIjR8b2jaPJyPpHkhfpy3XKpepHQU3z0tWhSyqDi73fRJ0O5aWQSMDePHHCc+ux1Ag54NdxCP3GFDAqxv65LosDVSn/Yj0ICggTVVaH/dnmqkJZ7O74zpLsq4qAWg0sc0I8Kgm/ZcG9I0kJuit/tW59c1W4ROaRiZFqms9ot1SIThppMrJxnUr1g8FW9QFSCDxzGLseV8od1tWq6QslkT74eUELl4AMnY7RhQ1W0ibHrjzVTksMbLrVTcL3/Td/eywyHpY68azbR4RajTfwAQWm7Lr+dImXlci/UjXQobyp05a8+rEmbCtT0k93FrUjtxiuuwqz8SgVBaxtmscMhMkzRjwsjPO8AbTawazZYTjtXvEQR0pBCaj+R43eFmBjsVoSYblx6i/VyN3g56EPEdshx10QI42YEl0Uo2wopqx2ZTQAtq5HQUg7h6+AptDZq3OG4sPwGFKromF+AejZp5WjStDwsd91lAAfWCvWe/8rNdCdOOae9nGxeg3wyJWaJG90YXGYidewv/qi2IrRe88qE5nYFM1fJun75Hw==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Date = _t, #"Counter Party" = _t, Reference = _t, #"Amount (GBP)" = _t])
in
Source
the result should be something like this
where I have a single Category and Subcategory column, and for example the highlighted row need manual input, I want to be able to add manually the Category and Subcategory and still be able to refresh queries without losing all the auto ones. Please a code that optimizes the performance
Thanks for your time and expertise.
@ronrsnfld @lbendlin @MarkLaf @SundarRaj @Nasif_Azam @speedramps
Hi @Einomi ,
Just checking in to see if you were able to review the solution that @AlienSx mentioned sending on June 19. If the solution addressed your issue, could you please share it here for visibility and mark it as accepted? This will help other members who might be facing a similar issue.
Let me know if you need any help.
Thanks so much!
Best regards,
Shruti
@v-sdhruv would you be so kind and stop posting off-topic all across the board? Why don't you simply send PM to the topic starter? If you have anything to add regarding "Auto and Manual Categories" - feel free to post your solution. Otherwise please stop begging for "accept as solution". Right now this board is heavily flooded by local CST so that it becomes unusable and the place I don't want to spend my time anymore. Please share my message with your colleagues. Thank you for your attention to this matter.
Hi @Einomi ,
Just wanted to check if you were able to resolve the issue?
If any response has addressed your query, please accept it as a solution so other members can easily find it.
Thank You
Hi,
Thanks so much for your solution, seems to answers all my questions.
I hope last question :
- if I remove a CSV from the folder, when I refresh the query, those transactions will still appear, I assume because of Table.Buffer, so if you think of an idea to address this
Finally, either here or PM/e-mail, if you can briefly describe the steps because as much as I have tried to understand the steps, I struggled a bit and I usually have a good level, but you are a master 😊 and we need to learn from you 🙂
Thanks
Hi @AlienSx
I was a bit quiet this past weeks, my father was not doing good at all and he is still in hospital.
Any chance you can look at my previous message
@Einomi wrote:Hi,
Thanks so much for your solution, seems to answers all my questions.
I hope last question :
- if I remove a CSV from the folder, when I refresh the query, those transactions will still appear, I assume because of Table.Buffer, so if you think of an idea to address this
Finally, either here or PM/e-mail, if you can briefly describe the steps because as much as I have tried to understand the steps, I struggled a bit and I usually have a good level, but you are a master 😊 and we need to learn from you 🙂
Thanks
If you can let me know, we are really so so close 🙂
Thanks for your patience
@Einomi this is standard self-referencing query stuff. But you must have some unique reference for every line of your statement (or date column will be enough if you load transactions for the whole day). I added index column but just for demo. In short: you have some table with "old" transactions in Excel (with auto and/or manual entries) - load it to PQ, load your source data, this way or another filter new transactions only (this is job for your unique identifier), apply your transformations (including auto category and subcategory stuff), combine old and new transactions and load resulting table back to Excel. Now you may manually edit that table and won't loose this info unless you manually delete rows from your table.
let
auto = Function.Invoke(
Record.FromList,
List.Reverse(
Table.ToColumns(
Table.CombineColumnsToRecord(
Excel.CurrentWorkbook(){[Name="auto"]}[Content],
"rec",
{"Category", "Subcategory"}
)
)
)
),
loaded_transactions = Table.Buffer(Excel.CurrentWorkbook(){[Name="loaded"]}[Content]),
loaded_ref = Record.FromList(List.Repeat({false}, Table.RowCount(loaded_transactions)), List.Transform(loaded_transactions[ref], Text.From)),
new_transactions = Table.AddColumn(
Table.SelectRows(statement, (x) => Record.FieldOrDefault(loaded_ref, x[ref], true)),
"cat",
(x) => Record.FieldOrDefault(auto, x[Counter Party], [])
),
xpand = Table.ExpandRecordColumn(new_transactions, "cat", {"Category", "Subcategory"}, null),
z = loaded_transactions & xpand
in
z
Hi,
Looks we are getting there. I have tried your file. However, the auto categorization does not look like to work.
I have tried to add one row to the mapping table and it did not refresh in the auto query, I even tried to change one subcategories in the mapping table and still did not reflect that change in the auto query.
The manual input where kept, I really need to be able to manual input and if null then the auto category, manual should override auto.
Finally, I am sure this is a bit basic but, I have read online in some specialized blogs that adding an index column may not be the best way to go as it recalculates and can mess up the data, what is your intake on this ?
Appreciate your time and your help
try this
let
// load mapping table into dictoionary record
auto = Function.Invoke(
Record.FromList,
List.Reverse(
Table.ToColumns(
Table.CombineColumnsToRecord(
Excel.CurrentWorkbook(){[Name="auto"]}[Content],
"rec",
{"Category", "Subcategory"}
)
)
)
),
// load old transactions, apply mapping to nulls
loaded_transactions = Table.Buffer(Excel.CurrentWorkbook(){[Name="loaded"]}[Content]),
add_mapping = Table.AddColumn(
loaded_transactions,
"map",
(x) => if x[Category] is null and x[Subcategory] is null
then Record.FieldOrDefault(auto, x[Counter Party], [])
else Record.SelectFields(x, {"Category", "Subcategory"})
),
remove_old_columns = Table.RemoveColumns(add_mapping, {"Category", "Subcategory"}),
// load new transactions
loaded_ref = Record.FromList(List.Repeat({false}, Table.RowCount(loaded_transactions)), List.Transform(loaded_transactions[ref], Text.From)),
new_transactions = Table.AddColumn(
Table.SelectRows(statement, (x) => Record.FieldOrDefault(loaded_ref, x[ref], true)),
"map",
(x) => Record.FieldOrDefault(auto, x[Counter Party], [])
),
// combine old and new, expand cat/subcat columns
combine = remove_old_columns & new_transactions,
expand = Table.ExpandRecordColumn(combine, "map", {"Category", "Subcategory"}, null)
in
expand
Regarding index usage: again, I added index just for demo purposes. It plays "unique transaction reference" role in my file. Don't use index. Maybe you don't need unique ref at all - it depends upon the data you are loading. E.g. if you load historical data then you may use date column as you key. You need _something_ that identifies your transactions as "old" or "new".
Whoa, we are getting so close 😊😁. I have managed to replicate this into my real data set. However, I struggled in 2 points.
You have 2 queries the statement one and the auto one, and I will use the same names to describe my steps and please tell me where I missed something.
1. Open blank workbook > Get Data > From SharePoint folder
2. Do all my transformations and add a unique column by merging a few ones (like you did with [ref])
3. Load back this query to Excel, add on that table the columns Category and Subcategory
4. Load back this query to PQ
5. Load my mapping (the one you called auto) table to PQ
So I am ending up having three queries
statement (the one I got from SP and loaded to Excel)
auto (my mapping table loaded to PQ)
statement (2) (the one loaded from Excel with the two new columns Category and Subcategory)
Then I am stuck because in your case, your original query, also known as statement was loaded as a connection but if do the same, I will have an issue with the auto query at the step loaded_transactions, so how do we overcome this ?
Also is there a way to remove the [ref] column at the final step of the auto query ? I have tried but it messed ? Alternatively, can we do a merge query instead of creating a [ref] column ?
Really appreciate your help.
@Einomi just continue editing your original query statement (using statement(2) and auto). This article by Matt Allington illustrates one of possible self-referencing setup.
Regarding ref: well, Left Anti join may help
Thanks for your answer. I am aware you are quite busy hence the quick reply 😁
However, can I send you my file via DM, because we are so so close and do not want to miss out.
After, I'll make a post to explain to new members or even advanced ones this rocking method 🙂
Please we are so close and I must miss something
Hey @Einomi ,
Thank you for providing the screenshots and context. To help you reconcile automated and manual Category/Subcategory values into one seamless output for reporting while optimizing performance and minimizing merges.
Uses Table.Buffer to prevent repetitive evaluation,
Avoids multiple merges,
Keeps manual inputs intact,
Prioritizes auto-matching from your mapping table but allows override via manual entries.
let // Step 1: Buffer mapping table to improve performance MappingBuffered = Table.Buffer(tblMapping), // Step 2: Add Auto Category and Subcategory from mapping table AddAutoMappings = Table.AddColumns( tblTransactions, { "AutoCategory", each try Record.Field(Table.SelectRows(MappingBuffered, (r) => Text.Upper(r[Description]) = Text.Upper([Counter Party])){0}, "Category") otherwise null, "AutoSubcategory", each try Record.Field(Table.SelectRows(MappingBuffered, (r) => Text.Upper(r[Description]) = Text.Upper([Counter Party])){0}, "Subcategory") otherwise null } ), // Step 3: Merge Manual and Auto columns into Final ones AddFinalCategory = Table.AddColumn(AddAutoMappings, "FinalCategory", each if [Category] <> null and [Category] <> "" then [Category] else [AutoCategory]), AddFinalSubcategory = Table.AddColumn(AddFinalCategory, "FinalSubcategory", each if [Subcategory] <> null and [Subcategory] <> "" then [Subcategory] else [AutoSubcategory]), // Step 4: Keep only relevant columns for reporting FinalOutput = Table.SelectColumns(AddFinalSubcategory, {"Date", "Counter Party", "Reference", "Amount", "FinalCategory", "FinalSubcategory"}) in FinalOutput
If you found this solution helpful, please consider accepting it and giving it a kudos (Like) it’s greatly appreciated and helps others find the solution more easily.
Best Regards,
Nasif Azam
Hi,
Thanks for your time. Can you help with the full M code ? looks like it missing some code.
And if you can detail some steps, because we have to do self-merge a query if I am not mistaken to be able to input in Excel
Let me know
Hey @Einomi ,
I just sending you the steps of M codes. Let's try the full M code with steps and comments.
let // STEP 1: Load and buffer your Mapping Table MappingTable = Excel.CurrentWorkbook(){[Name="tblMapping"]}[Content], MappingBuffered = Table.Buffer(MappingTable), // STEP 2: Load your main Transactions Table (with Category/Subcategory editable manually) TransactionsOriginal = Excel.CurrentWorkbook(){[Name="tblTransactions"]}[Content], // STEP 3: Self-merge to preserve manual inputs (Category/Subcategory) TransactionsBuffered = Table.Buffer(TransactionsOriginal), PreserveManualInputs = Table.RemoveColumns( Table.NestedJoin(TransactionsBuffered, {"Counter Party", "Reference"}, TransactionsBuffered, {"Counter Party", "Reference"}, "Manual", JoinKind.LeftOuter), "Manual" ), AddManualColumns = Table.ExpandTableColumn( Table.NestedJoin(TransactionsBuffered, {"Counter Party", "Reference"}, TransactionsBuffered, {"Counter Party", "Reference"}, "Manual", JoinKind.LeftOuter), "Manual", {"Category", "Subcategory"}, {"ManualCategory", "ManualSubcategory"} ), // STEP 4: Add Auto Category/Subcategory from mapping AddAutoMappings = Table.AddColumns( AddManualColumns, { "AutoCategory", each try Record.Field(Table.SelectRows(MappingBuffered, (r) => Text.Upper(r[Description]) = Text.Upper([Counter Party])){0}, "Category") otherwise null, "AutoSubcategory", each try Record.Field(Table.SelectRows(MappingBuffered, (r) => Text.Upper(r[Description]) = Text.Upper([Counter Party])){0}, "Subcategory") otherwise null } ), // STEP 5: Decide Final Values → Manual wins over Auto AddFinalCategory = Table.AddColumn(AddAutoMappings, "FinalCategory", each if [ManualCategory] <> null and [ManualCategory] <> "" then [ManualCategory] else [AutoCategory]), AddFinalSubcategory = Table.AddColumn(AddFinalCategory, "FinalSubcategory", each if [ManualSubcategory] <> null and [ManualSubcategory] <> "" then [ManualSubcategory] else [AutoSubcategory]), // STEP 6: Output only relevant columns FinalOutput = Table.SelectColumns(AddFinalSubcategory, {"Date", "Counter Party", "Reference", "Amount", "FinalCategory", "FinalSubcategory"}) in FinalOutput
If you found this solution helpful, please consider accepting it and giving it a kudos (Like) it’s greatly appreciated and helps others find the solution more easily.
Best Regards,
Nasif Azam
Hi
Thanks for your time but I am bit still confused.
Can we write the M code from scratch becasue I do not see the original load of the transactions table.
Thanks
Hey @Einomi ,
You want to:
Load your transaction table (CSV or Excel),
Map Categories/Subcategories automatically from a mapping table,
Retain any manual edits you make in Excel (so they're not lost after refresh),
Combine both auto and manual in a single output for reporting,
Avoid repeated merges and improve performance using Table.Buffer.
let // STEP 1: Load the mapping table from Excel MappingTable = Excel.CurrentWorkbook(){[Name="tblMapping"]}[Content], MappingBuffered = Table.Buffer(MappingTable), // STEP 2: Load transactions from Excel (this is the table with editable Category/Subcategory) TransactionsRaw = Excel.CurrentWorkbook(){[Name="tblTransactions"]}[Content], TransactionsBuffered = Table.Buffer(TransactionsRaw), // STEP 3: Add auto Category/Subcategory using Mapping Table AddAutoColumns = Table.AddColumns( TransactionsBuffered, { "AutoCategory", each try Record.Field( Table.SelectRows(MappingBuffered, (r) => Text.Upper(r[Description]) = Text.Upper([Counter Party])){0}, "Category" ) otherwise null, "AutoSubcategory", each try Record.Field( Table.SelectRows(MappingBuffered, (r) => Text.Upper(r[Description]) = Text.Upper([Counter Party])){0}, "Subcategory" ) otherwise null } ), // STEP 4: Final logic – Manual input takes priority over auto AddFinalCategory = Table.AddColumn( AddAutoColumns, "FinalCategory", each if [Category] <> null and [Category] <> "" then [Category] else [AutoCategory] ), AddFinalSubcategory = Table.AddColumn( AddFinalCategory, "FinalSubcategory", each if [Subcategory] <> null and [Subcategory] <> "" then [Subcategory] else [AutoSubcategory] ), // STEP 5: Keep necessary columns FinalOutput = Table.SelectColumns( AddFinalSubcategory, {"Date", "Counter Party", "Reference", "Amount", "FinalCategory", "FinalSubcategory"} ) in FinalOutput
The query does not overwrite manual values; any edits you make directly in the tblTransactions table will be retained.
The "tblMapping" and "tblTransactions" should be named Excel tables in your workbook.
If you found this solution helpful, please consider accepting it and giving it a kudos (Like) it’s greatly appreciated and helps others find the solution more easily.
Best Regards,
Nasif Azam
Hi Nasif,
Thanks for your time, however I strongly suspect your code is generated by an AI, the layout is very similar and I am not sure you addresses my requirements.
Table.AddColumns does not exist in Power Query, only Table.AddColumn
Did you try your code ?
Please review all my messages and feel free to offer a suggestion 🙂
User | Count |
---|---|
11 | |
7 | |
5 | |
5 | |
4 |
User | Count |
---|---|
16 | |
14 | |
8 | |
6 | |
6 |