Forum Discussion
Power Query doesn't use 100% of the processor
- 3 years ago
I think I understand your requirements now. I'm pretty sure you can achieve some performnce improvements by leveraging Group, still, but there are a couple extra steps. To test performance better, I switched to randomly generated data to test 10k and 100k rows in the structure you specified above for your testing.
The below works pretty well, just takes 1-2 seconds to load. Approach is to merge grouped rows (when grouped on a column, that column becomes primary key, which improve join performance), filter merged grouped rows as needed, sum values for running total, then do a second group to get the min:
let Source = PerfTest_10k, MaterialGroups = Table.Group( Source, {"Material"}, {{ "Current qty", each _, type table [Material=nullable text, Date=nullable date, Stock movement qty=nullable number] }} ), MergeGroups = Table.NestedJoin( Source, "Material", MaterialGroups, "Material", "Groups", JoinKind.Inner ), ExpandGroups = Table.ExpandTableColumn(MergeGroups, "Groups", {"Current qty"}, {"Current qty"}), GetCurQtyRows = Table.TransformRows( ExpandGroups, (row)=> Record.TransformFields( row, { "Current qty", each let _t = Table.SelectRows( row[Current qty], each [Date] <= row[Date] ) in List.Sum( Table.Column(_t, "Stock movement qty") ) } ) ), GetCurQty = Table.FromRecords( GetCurQtyRows, type table [Material=text, Date=date, Stock movement qty=number, Current qty=number] ), GetMinQty = Table.Group( GetCurQty, {"Material"}, { { "Min qty", each List.Min([Current qty]), type number } } ) in GetMinQtyOutput:
The above doesn't work so great when you up the rows to 100k, though. For that I think you have to turn to DAX. This takes about 2 sec to work over 100k rows (probably there are ways to improve performance further on this). Note that [Running Total] and [Min Running Total] are measures:
Running Total = VAR _thisDt = MAX( PerfTest_100k[Date] ) VAR _matGroup = CALCULATETABLE( PerfTest_100k, REMOVEFILTERS( PerfTest_100k ), VALUES( PerfTest_100k[Material] ) ) VAR _curPrevRows = FILTER( _matGroup, PerfTest_100k[Date] <= _thisDt ) RETURN CALCULATE( SUM( PerfTest_100k[Stock movement qty] ), _curPrevRows ) Min Running Total = MINX( SUMMARIZE( PerfTest_100k, PerfTest_100k[Material], PerfTest_100k[Date] ), [Running Total] )Output (note it's all randomly generated which is why these numbers don't match output above):
In case interested and to show my work, here is the M for the test data. Below generates 10k rows for
PerfTest_10k. It's same code, but 10000 replaced with 100000 in line 4, for PerfTest_100k:
let Source = List.Generate( ()=>0, each _ < 10000, each _ + 1, each [ Material = Character.FromNumber( List.Min( { Int32.From( Number.RandomBetween(65, 91) ), // A-Z 90 } ) ), Date = Date.AddDays( #date(2022,1,1), List.Min({ Int32.From( Number.RandomBetween( 0, 365 ) ), // 1/1/2022-12/31/2022 364 } ) ), Stock movement qty = Int64.From( Number.RandomBetween( -100, 100 ) ) // -100 - +100 ] ), Ouput = Table.FromRecords( Source, type table [Material=text,Date=date,Stock movement qty=number] ) in Ouput
Hello MarkLaf , thank you for looking for a solution!
Unfortunately, your code doesn't meet my need.
Here is an example of what I want to achieve. The example table down here could be one of the rows from your column "Grouped Original Table".
The result I want to get is in red, and the result of your code is in blue.
Columns in italic aren't in the native table, and are only here to illustrate the logic of the red result.
| Material | Date | Stock movement qty | Current qty | Min qty |
| A | 01/01/2023 | +30 | 30 | 30 |
| A | 01/02/2023 | +10 | 40 | 30 |
| A | 01/03/2023 | -15 | 25 | 25 |
| A | 01/04/2023 | -5 | 20 | 20 |
| A | 01/05/2023 | +40 | 60 | 20 |
To achieve this, I need to execute a running total to get the minimum cumulated stock over an ascending list of dates. Your code would have been perfect if I had the "Current qty" column in my native table, which is sadly not the case.
I think I understand your requirements now. I'm pretty sure you can achieve some performnce improvements by leveraging Group, still, but there are a couple extra steps. To test performance better, I switched to randomly generated data to test 10k and 100k rows in the structure you specified above for your testing.
The below works pretty well, just takes 1-2 seconds to load. Approach is to merge grouped rows (when grouped on a column, that column becomes primary key, which improve join performance), filter merged grouped rows as needed, sum values for running total, then do a second group to get the min:
let
Source = PerfTest_10k,
MaterialGroups = Table.Group(
Source,
{"Material"},
{{
"Current qty",
each _,
type table [Material=nullable text, Date=nullable date, Stock movement qty=nullable number]
}}
),
MergeGroups = Table.NestedJoin(
Source,
"Material",
MaterialGroups,
"Material",
"Groups",
JoinKind.Inner
),
ExpandGroups = Table.ExpandTableColumn(MergeGroups, "Groups", {"Current qty"}, {"Current qty"}),
GetCurQtyRows =
Table.TransformRows(
ExpandGroups,
(row)=> Record.TransformFields(
row,
{
"Current qty",
each let
_t = Table.SelectRows( row[Current qty], each [Date] <= row[Date] )
in
List.Sum( Table.Column(_t, "Stock movement qty") )
}
)
),
GetCurQty = Table.FromRecords(
GetCurQtyRows,
type table [Material=text, Date=date, Stock movement qty=number, Current qty=number]
),
GetMinQty = Table.Group(
GetCurQty,
{"Material"},
{
{ "Min qty", each List.Min([Current qty]), type number }
}
)
in
GetMinQty
Output:
The above doesn't work so great when you up the rows to 100k, though. For that I think you have to turn to DAX. This takes about 2 sec to work over 100k rows (probably there are ways to improve performance further on this). Note that [Running Total] and [Min Running Total] are measures:
Running Total =
VAR _thisDt = MAX( PerfTest_100k[Date] )
VAR _matGroup = CALCULATETABLE( PerfTest_100k, REMOVEFILTERS( PerfTest_100k ), VALUES( PerfTest_100k[Material] ) )
VAR _curPrevRows = FILTER( _matGroup, PerfTest_100k[Date] <= _thisDt )
RETURN
CALCULATE( SUM( PerfTest_100k[Stock movement qty] ), _curPrevRows )
Min Running Total =
MINX( SUMMARIZE( PerfTest_100k, PerfTest_100k[Material], PerfTest_100k[Date] ), [Running Total] )
Output (note it's all randomly generated which is why these numbers don't match output above):
In case interested and to show my work, here is the M for the test data. Below generates 10k rows for
PerfTest_10k. It's same code, but 10000 replaced with 100000 in line 4, for PerfTest_100k:
let
Source = List.Generate(
()=>0,
each _ < 10000,
each _ + 1,
each [
Material = Character.FromNumber(
List.Min( {
Int32.From( Number.RandomBetween(65, 91) ), // A-Z
90
} )
),
Date = Date.AddDays(
#date(2022,1,1),
List.Min({
Int32.From( Number.RandomBetween( 0, 365 ) ), // 1/1/2022-12/31/2022
364 } )
),
Stock movement qty = Int64.From( Number.RandomBetween( -100, 100 ) ) // -100 - +100
]
),
Ouput = Table.FromRecords( Source, type table [Material=text,Date=date,Stock movement qty=number] )
in
Ouput
- _AlexandreRM_3 years agoHelper II
Hello MarkLaf , just... wouah !
I tried your code without any modification in a blank new file, and it worked almost instantly.
After I tried it in my real query, and it has incredibly improved its performance : previously it took around 25 mins to complete, and now only 1 minute !!! (I couldn't reach the duration of 2 seconds, because the root table comes itself from a few others simple queries).
I have now a strongly improved query, and I also learned some stuff I didn't know about M syntax. For example, I discovered that groupping can be used without aggregation function, to just group the rows of a table on a given key (previously I was joining the table on itself to achieve this, which is... less clean).
So, thank you VERY much for your message. You helped me a lot !
- MarkLaf3 years agoSuper User
Glad that worked for you!
You may also want to test with different parallelism settings. Increasing may help, and sometimes (counter-intuitively) turning off parallelism can improve performance, especially when you have a cascade of transformation queries.