Forum Discussion
Power Query doesn't use 100% of the processor
- 3 years ago
I think I understand your requirements now. I'm pretty sure you can achieve some performnce improvements by leveraging Group, still, but there are a couple extra steps. To test performance better, I switched to randomly generated data to test 10k and 100k rows in the structure you specified above for your testing.
The below works pretty well, just takes 1-2 seconds to load. Approach is to merge grouped rows (when grouped on a column, that column becomes primary key, which improve join performance), filter merged grouped rows as needed, sum values for running total, then do a second group to get the min:
let Source = PerfTest_10k, MaterialGroups = Table.Group( Source, {"Material"}, {{ "Current qty", each _, type table [Material=nullable text, Date=nullable date, Stock movement qty=nullable number] }} ), MergeGroups = Table.NestedJoin( Source, "Material", MaterialGroups, "Material", "Groups", JoinKind.Inner ), ExpandGroups = Table.ExpandTableColumn(MergeGroups, "Groups", {"Current qty"}, {"Current qty"}), GetCurQtyRows = Table.TransformRows( ExpandGroups, (row)=> Record.TransformFields( row, { "Current qty", each let _t = Table.SelectRows( row[Current qty], each [Date] <= row[Date] ) in List.Sum( Table.Column(_t, "Stock movement qty") ) } ) ), GetCurQty = Table.FromRecords( GetCurQtyRows, type table [Material=text, Date=date, Stock movement qty=number, Current qty=number] ), GetMinQty = Table.Group( GetCurQty, {"Material"}, { { "Min qty", each List.Min([Current qty]), type number } } ) in GetMinQtyOutput:
The above doesn't work so great when you up the rows to 100k, though. For that I think you have to turn to DAX. This takes about 2 sec to work over 100k rows (probably there are ways to improve performance further on this). Note that [Running Total] and [Min Running Total] are measures:
Running Total = VAR _thisDt = MAX( PerfTest_100k[Date] ) VAR _matGroup = CALCULATETABLE( PerfTest_100k, REMOVEFILTERS( PerfTest_100k ), VALUES( PerfTest_100k[Material] ) ) VAR _curPrevRows = FILTER( _matGroup, PerfTest_100k[Date] <= _thisDt ) RETURN CALCULATE( SUM( PerfTest_100k[Stock movement qty] ), _curPrevRows ) Min Running Total = MINX( SUMMARIZE( PerfTest_100k, PerfTest_100k[Material], PerfTest_100k[Date] ), [Running Total] )Output (note it's all randomly generated which is why these numbers don't match output above):
In case interested and to show my work, here is the M for the test data. Below generates 10k rows for
PerfTest_10k. It's same code, but 10000 replaced with 100000 in line 4, for PerfTest_100k:
let Source = List.Generate( ()=>0, each _ < 10000, each _ + 1, each [ Material = Character.FromNumber( List.Min( { Int32.From( Number.RandomBetween(65, 91) ), // A-Z 90 } ) ), Date = Date.AddDays( #date(2022,1,1), List.Min({ Int32.From( Number.RandomBetween( 0, 365 ) ), // 1/1/2022-12/31/2022 364 } ) ), Stock movement qty = Int64.From( Number.RandomBetween( -100, 100 ) ) // -100 - +100 ] ), Ouput = Table.FromRecords( Source, type table [Material=text,Date=date,Stock movement qty=number] ) in Ouput
I think I understand your requirements now. I'm pretty sure you can achieve some performnce improvements by leveraging Group, still, but there are a couple extra steps. To test performance better, I switched to randomly generated data to test 10k and 100k rows in the structure you specified above for your testing.
The below works pretty well, just takes 1-2 seconds to load. Approach is to merge grouped rows (when grouped on a column, that column becomes primary key, which improve join performance), filter merged grouped rows as needed, sum values for running total, then do a second group to get the min:
let
Source = PerfTest_10k,
MaterialGroups = Table.Group(
Source,
{"Material"},
{{
"Current qty",
each _,
type table [Material=nullable text, Date=nullable date, Stock movement qty=nullable number]
}}
),
MergeGroups = Table.NestedJoin(
Source,
"Material",
MaterialGroups,
"Material",
"Groups",
JoinKind.Inner
),
ExpandGroups = Table.ExpandTableColumn(MergeGroups, "Groups", {"Current qty"}, {"Current qty"}),
GetCurQtyRows =
Table.TransformRows(
ExpandGroups,
(row)=> Record.TransformFields(
row,
{
"Current qty",
each let
_t = Table.SelectRows( row[Current qty], each [Date] <= row[Date] )
in
List.Sum( Table.Column(_t, "Stock movement qty") )
}
)
),
GetCurQty = Table.FromRecords(
GetCurQtyRows,
type table [Material=text, Date=date, Stock movement qty=number, Current qty=number]
),
GetMinQty = Table.Group(
GetCurQty,
{"Material"},
{
{ "Min qty", each List.Min([Current qty]), type number }
}
)
in
GetMinQty
Output:
The above doesn't work so great when you up the rows to 100k, though. For that I think you have to turn to DAX. This takes about 2 sec to work over 100k rows (probably there are ways to improve performance further on this). Note that [Running Total] and [Min Running Total] are measures:
Running Total =
VAR _thisDt = MAX( PerfTest_100k[Date] )
VAR _matGroup = CALCULATETABLE( PerfTest_100k, REMOVEFILTERS( PerfTest_100k ), VALUES( PerfTest_100k[Material] ) )
VAR _curPrevRows = FILTER( _matGroup, PerfTest_100k[Date] <= _thisDt )
RETURN
CALCULATE( SUM( PerfTest_100k[Stock movement qty] ), _curPrevRows )
Min Running Total =
MINX( SUMMARIZE( PerfTest_100k, PerfTest_100k[Material], PerfTest_100k[Date] ), [Running Total] )
Output (note it's all randomly generated which is why these numbers don't match output above):
In case interested and to show my work, here is the M for the test data. Below generates 10k rows for
PerfTest_10k. It's same code, but 10000 replaced with 100000 in line 4, for PerfTest_100k:
let
Source = List.Generate(
()=>0,
each _ < 10000,
each _ + 1,
each [
Material = Character.FromNumber(
List.Min( {
Int32.From( Number.RandomBetween(65, 91) ), // A-Z
90
} )
),
Date = Date.AddDays(
#date(2022,1,1),
List.Min({
Int32.From( Number.RandomBetween( 0, 365 ) ), // 1/1/2022-12/31/2022
364 } )
),
Stock movement qty = Int64.From( Number.RandomBetween( -100, 100 ) ) // -100 - +100
]
),
Ouput = Table.FromRecords( Source, type table [Material=text,Date=date,Stock movement qty=number] )
in
Ouput
Hello MarkLaf , just... wouah !
I tried your code without any modification in a blank new file, and it worked almost instantly.
After I tried it in my real query, and it has incredibly improved its performance : previously it took around 25 mins to complete, and now only 1 minute !!! (I couldn't reach the duration of 2 seconds, because the root table comes itself from a few others simple queries).
I have now a strongly improved query, and I also learned some stuff I didn't know about M syntax. For example, I discovered that groupping can be used without aggregation function, to just group the rows of a table on a given key (previously I was joining the table on itself to achieve this, which is... less clean).
So, thank you VERY much for your message. You helped me a lot !
- MarkLaf3 years agoSuper User
Glad that worked for you!
You may also want to test with different parallelism settings. Increasing may help, and sometimes (counter-intuitively) turning off parallelism can improve performance, especially when you have a cascade of transformation queries.