Forum Discussion
Power Query doesn't use 100% of the processor
- 3 years ago
I think I understand your requirements now. I'm pretty sure you can achieve some performnce improvements by leveraging Group, still, but there are a couple extra steps. To test performance better, I switched to randomly generated data to test 10k and 100k rows in the structure you specified above for your testing.
The below works pretty well, just takes 1-2 seconds to load. Approach is to merge grouped rows (when grouped on a column, that column becomes primary key, which improve join performance), filter merged grouped rows as needed, sum values for running total, then do a second group to get the min:
let Source = PerfTest_10k, MaterialGroups = Table.Group( Source, {"Material"}, {{ "Current qty", each _, type table [Material=nullable text, Date=nullable date, Stock movement qty=nullable number] }} ), MergeGroups = Table.NestedJoin( Source, "Material", MaterialGroups, "Material", "Groups", JoinKind.Inner ), ExpandGroups = Table.ExpandTableColumn(MergeGroups, "Groups", {"Current qty"}, {"Current qty"}), GetCurQtyRows = Table.TransformRows( ExpandGroups, (row)=> Record.TransformFields( row, { "Current qty", each let _t = Table.SelectRows( row[Current qty], each [Date] <= row[Date] ) in List.Sum( Table.Column(_t, "Stock movement qty") ) } ) ), GetCurQty = Table.FromRecords( GetCurQtyRows, type table [Material=text, Date=date, Stock movement qty=number, Current qty=number] ), GetMinQty = Table.Group( GetCurQty, {"Material"}, { { "Min qty", each List.Min([Current qty]), type number } } ) in GetMinQtyOutput:
The above doesn't work so great when you up the rows to 100k, though. For that I think you have to turn to DAX. This takes about 2 sec to work over 100k rows (probably there are ways to improve performance further on this). Note that [Running Total] and [Min Running Total] are measures:
Running Total = VAR _thisDt = MAX( PerfTest_100k[Date] ) VAR _matGroup = CALCULATETABLE( PerfTest_100k, REMOVEFILTERS( PerfTest_100k ), VALUES( PerfTest_100k[Material] ) ) VAR _curPrevRows = FILTER( _matGroup, PerfTest_100k[Date] <= _thisDt ) RETURN CALCULATE( SUM( PerfTest_100k[Stock movement qty] ), _curPrevRows ) Min Running Total = MINX( SUMMARIZE( PerfTest_100k, PerfTest_100k[Material], PerfTest_100k[Date] ), [Running Total] )Output (note it's all randomly generated which is why these numbers don't match output above):
In case interested and to show my work, here is the M for the test data. Below generates 10k rows for
PerfTest_10k. It's same code, but 10000 replaced with 100000 in line 4, for PerfTest_100k:
let Source = List.Generate( ()=>0, each _ < 10000, each _ + 1, each [ Material = Character.FromNumber( List.Min( { Int32.From( Number.RandomBetween(65, 91) ), // A-Z 90 } ) ), Date = Date.AddDays( #date(2022,1,1), List.Min({ Int32.From( Number.RandomBetween( 0, 365 ) ), // 1/1/2022-12/31/2022 364 } ) ), Stock movement qty = Int64.From( Number.RandomBetween( -100, 100 ) ) // -100 - +100 ] ), Ouput = Table.FromRecords( Source, type table [Material=text,Date=date,Stock movement qty=number] ) in Ouput
There are a few ways to do this without recursion, but some of them involve repeatedly summing all the previous rows, which can be inefficient.
Here's an approach that only scans the data once.
let
Source = ...,
#"Sorted Rows" = Table.Sort(Source,{{"Date", Order.Ascending}}),
#"Added Index" = Table.AddIndexColumn(#"Sorted Rows", "Index", 0, 1, Int64.Type),
RunningTotal = List.Generate(
() => [Index = 0, Total = #"Added Index"{0}[Stock movement qty], Min = Total],
each [Index] < Table.RowCount(#"Added Index"),
(previous) =>
let
newIndex = previous[Index] + 1
in
[
Index = newIndex,
Total = previous[Total] + #"Added Index"{newIndex}[Stock movement qty],
Min = List.Min({previous[Min], Total})
]),
Custom1 = Table.AddColumn(#"Added Index", "Current qty", each RunningTotal{[Index]}[Total], type number),
Custom2 = Table.AddColumn(Custom1, "Min qty", each RunningTotal{[Index]}[Min], type number)
in
Custom2
That said, these kinds of cumulative calculations are probably better done in DAX, after loading the data to the (Power Pivot) Data Model.
- Ehren3 years agoMicrosoft Employee
You can learn more about creating efficient M running totals here:
https://gorilla.bi/power-query/running-total/#fast-running-totals
- _AlexandreRM_3 years agoHelper II
Hello Ehren , sorry for the very late reply, I was in vacation.
I think the List.Generate is a good alternative, I never tried to use it with records to store multiple values at each iteration!
After tests, it seems that both solutions (recursivity and list generation) have almost the same (low) performance. I think it's just the fact to iterate manually over each row of a table which is time-consuming. For those who are interested in, here is the formula using List.Generate:
f = (codeTable as table) => let iMax = Table.RowCount(codeTable), qtList = List.Buffer(codeTable[#"Qté entrées/besoins"]), cumulativeStockList = List.Generate( () => [ i = 1, qt = qtList{0}, minQt = qtList{0} ], each [i] <= iMax, each [ i = [i] + 1, qt = [qt] + qtList{[i]}, minQt = List.Min({[qt] + qtList{[i]}, [minQt]}) ]) in List.Last(cumulativeStockList)[minQt],So, except if Microsoft add a way to access more efficiently rows content for custom formulas, I don't think this problem will be solved!
I will also take a look at DAX, but its interface in Excel isn't... user-friendly, let's say.
- Ehren3 years agoMicrosoft Employee
Just curious: do you see a perf difference if you omit the call to List.Buffer?
- _AlexandreRM_3 years agoHelper II
No, it's exactly the same.
But I'm already using Table.Buffer on codeTable before calling the function, and this has been a huge performance improvement (around 10 times faster with Table.Buffer(codeTable) than without).