Forum Discussion
Power Query doesn't use 100% of the processor
- 3 years ago
I think I understand your requirements now. I'm pretty sure you can achieve some performnce improvements by leveraging Group, still, but there are a couple extra steps. To test performance better, I switched to randomly generated data to test 10k and 100k rows in the structure you specified above for your testing.
The below works pretty well, just takes 1-2 seconds to load. Approach is to merge grouped rows (when grouped on a column, that column becomes primary key, which improve join performance), filter merged grouped rows as needed, sum values for running total, then do a second group to get the min:
let Source = PerfTest_10k, MaterialGroups = Table.Group( Source, {"Material"}, {{ "Current qty", each _, type table [Material=nullable text, Date=nullable date, Stock movement qty=nullable number] }} ), MergeGroups = Table.NestedJoin( Source, "Material", MaterialGroups, "Material", "Groups", JoinKind.Inner ), ExpandGroups = Table.ExpandTableColumn(MergeGroups, "Groups", {"Current qty"}, {"Current qty"}), GetCurQtyRows = Table.TransformRows( ExpandGroups, (row)=> Record.TransformFields( row, { "Current qty", each let _t = Table.SelectRows( row[Current qty], each [Date] <= row[Date] ) in List.Sum( Table.Column(_t, "Stock movement qty") ) } ) ), GetCurQty = Table.FromRecords( GetCurQtyRows, type table [Material=text, Date=date, Stock movement qty=number, Current qty=number] ), GetMinQty = Table.Group( GetCurQty, {"Material"}, { { "Min qty", each List.Min([Current qty]), type number } } ) in GetMinQtyOutput:
The above doesn't work so great when you up the rows to 100k, though. For that I think you have to turn to DAX. This takes about 2 sec to work over 100k rows (probably there are ways to improve performance further on this). Note that [Running Total] and [Min Running Total] are measures:
Running Total = VAR _thisDt = MAX( PerfTest_100k[Date] ) VAR _matGroup = CALCULATETABLE( PerfTest_100k, REMOVEFILTERS( PerfTest_100k ), VALUES( PerfTest_100k[Material] ) ) VAR _curPrevRows = FILTER( _matGroup, PerfTest_100k[Date] <= _thisDt ) RETURN CALCULATE( SUM( PerfTest_100k[Stock movement qty] ), _curPrevRows ) Min Running Total = MINX( SUMMARIZE( PerfTest_100k, PerfTest_100k[Material], PerfTest_100k[Date] ), [Running Total] )Output (note it's all randomly generated which is why these numbers don't match output above):
In case interested and to show my work, here is the M for the test data. Below generates 10k rows for
PerfTest_10k. It's same code, but 10000 replaced with 100000 in line 4, for PerfTest_100k:
let Source = List.Generate( ()=>0, each _ < 10000, each _ + 1, each [ Material = Character.FromNumber( List.Min( { Int32.From( Number.RandomBetween(65, 91) ), // A-Z 90 } ) ), Date = Date.AddDays( #date(2022,1,1), List.Min({ Int32.From( Number.RandomBetween( 0, 365 ) ), // 1/1/2022-12/31/2022 364 } ) ), Stock movement qty = Int64.From( Number.RandomBetween( -100, 100 ) ) // -100 - +100 ] ), Ouput = Table.FromRecords( Source, type table [Material=text,Date=date,Stock movement qty=number] ) in Ouput
Hi _AlexandreRM_ ,
Really hard to give a definitive answer here, but a few things to think about:
- Some of it depends on whether the work you're asking Excel/Power Query to do is serial or parallelable i.e. whether the work has to be done one part after another, or whether it can be threaded across different processing cores. Please don't ask me how to find out if this is the case, I haven't got a clue. It would probably be a question for someone like Ehren.
- If you're using a 32-bit version of Excel, then there's a hard RAM cap of 2GB per Windows process instance which, I believe, would then be subdivided across your evaluation containers that handle your query transformations. In this instance, you could consider upgrading to Excel 64-bit to avoid the cap.
- If you would consider using Power Query within Power BI Desktop, then you could get both the RAM cap gains by using the 64-bit version, as well as super-charge the RAM usage on the evaluation containers by updating the app MaxEvaluationWorkingSetInMB registry entry. More details on that from Chris Webb here:
Pete
Hi BA_Pete , thank you very much for your tips.
- In my opinion all calculations could be paralell. Ehren , I mention you, if you have a clue, here is a simplified version of my query code (function f is a custom function taking a very long time to complete):
let
base = Table.Buffer(#"RRP1 + RRP4"),
baseMaterials = Table.Distinct(base, {"Numéro de produit", "Désignation du produit"}),
baseMaterialsJoinBase = Table.NestedJoin(baseMaterials, {"Numéro de produit"}, base, "Numéro de produit", "join", JoinKind.LeftOuter),
bufferJoinTable = Table.TransformColumns(baseMaterialsJoinBase, {"join", Table.Buffer}),
addMin = Table.AddColumn(bufferJoinTable, "Qty", each f([join], 0, 9999999999, 0))
in
addMin
- I just checked in my account data, I am already on a 64bits version,
- This is incredibly powerful, thank you for sharing!!! I defined the parameter in my PowerBI desktop, but I didn't found it in Excel Power Query, it seems Microsoft still didn't pushed this change, sadly.
Alexandre
- Ehren3 years agoMicrosoft Employee
What is the CPU/memory utilization of the Microsoft.Mashup.Container*.exe processes? They're the ones doing the bulk of the PQ work.
Also, it's possible the processing is slow because of data source access, which wouldn't show up in CPU/memory.
If you remove the addMin step, is everything faster? If so, it would be important to share what the f function does.
- _AlexandreRM_3 years agoHelper II
Hello Ehren,
The CPU/memory utilization of all Excel sub-processes is as follows:
(columns: process name, CPU, memory, hard disk, network)
Note that the CPU utilization is far higher than on my previous screenshot, even if I didn't changed anything. Still, PQ has a good romm for improvement.
The data sources are 2 Excel files on the Sharepoint, previously mergued in the #"RRP1 + RRP4" table.
Here is the code of function f, which is the reason of the query slowness (a recursive function applying to the [join] table). Its goal is to calculate the minimum value of forecasted stock, using all stocks entries and exits:
f = (codeTable as table, previousQt as number, minQt as number, i as number) => let currentQt = previousQt + codeTable{i}[#"Stock movement qty"], currentMinQt = if currentQt < minQt then currentQt else minQt, //former method, the new one with error handling seems more efficient: //result = if i = iMax then minQt else @f(codeTable, currentQt, currentMinQt, i + 1, iMax) result = try @f(codeTable, currentQt, currentMinQt, i + 1) otherwise currentMinQt in result,If you have an idea to improve this function, I would be stronly interested!
Alexandre
- Ehren3 years agoMicrosoft Employee
I'm not entirely sure what the purpose of the minQt calculation is, but other than that the recursion seems unnecessary. Buffering each and every table in the join column independently also seems like it could be detrimental perf-wise.
After performing your join, have you tried just summing the "Stock movement qty" column in the nested tables?