Forum Discussion

_AlexandreRM_'s avatar
_AlexandreRM_
Helper II
3 years ago
Solved

Power Query doesn't use 100% of the processor

Hello,   In this example I'm using Power Query in Excel, but I gess it whould be the same on PowerBI.   I have currently a very complex query taking an average of 30 minuts to execute, while my p...
  • MarkLaf's avatar
    MarkLaf
    3 years ago

    I think I understand your requirements now. I'm pretty sure you can achieve some performnce improvements by leveraging Group, still, but there are a couple extra steps. To test performance better, I switched to randomly generated data to test 10k and 100k rows in the structure you specified above for your testing.

     

    The below works pretty well, just takes 1-2 seconds to load. Approach is to merge grouped rows (when grouped on a column, that column becomes primary key, which improve join performance), filter merged grouped rows as needed, sum values for running total, then do a second group to get the min:

     

    let
        Source = PerfTest_10k,
        MaterialGroups = Table.Group(
            Source, 
            {"Material"}, 
            {{
                "Current qty", 
                each _, 
                type table [Material=nullable text, Date=nullable date, Stock movement qty=nullable number]
            }}
        ),
        MergeGroups = Table.NestedJoin( 
            Source, 
            "Material", 
            MaterialGroups, 
            "Material", 
            "Groups", 
            JoinKind.Inner 
        ),
        ExpandGroups = Table.ExpandTableColumn(MergeGroups, "Groups", {"Current qty"}, {"Current qty"}),
        GetCurQtyRows = 
        Table.TransformRows(  
            ExpandGroups, 
            (row)=> Record.TransformFields( 
                row, 
                {
                    "Current qty", 
                    each let 
                        _t = Table.SelectRows( row[Current qty], each [Date] <= row[Date] ) 
                    in 
                        List.Sum( Table.Column(_t, "Stock movement qty") ) 
                } 
            ) 
        ),
        GetCurQty = Table.FromRecords( 
            GetCurQtyRows, 
            type table [Material=text, Date=date, Stock movement qty=number, Current qty=number] 
        ),
        GetMinQty = Table.Group(
            GetCurQty, 
            {"Material"}, 
            {
                { "Min qty", each List.Min([Current qty]), type number }
            }
        )
    in
        GetMinQty

     

    Output:

     

    The above doesn't work so great when you up the rows to 100k, though. For that I think you have to turn to DAX. This takes about 2 sec to work over 100k rows (probably there are ways to improve performance further on this). Note that [Running Total] and [Min Running Total] are measures:

     

    Running Total = 
    VAR _thisDt = MAX( PerfTest_100k[Date] )
    VAR _matGroup = CALCULATETABLE( PerfTest_100k, REMOVEFILTERS( PerfTest_100k ), VALUES( PerfTest_100k[Material] ) )
    VAR _curPrevRows = FILTER( _matGroup, PerfTest_100k[Date] <= _thisDt )
    RETURN
    CALCULATE( SUM( PerfTest_100k[Stock movement qty] ), _curPrevRows )
    
    
    Min Running Total = 
    MINX( SUMMARIZE( PerfTest_100k, PerfTest_100k[Material], PerfTest_100k[Date] ), [Running Total] )

     

    Output (note it's all randomly generated which is why these numbers don't match output above):

    In case interested and to show my work, here is the M for the test data. Below generates 10k rows for

    PerfTest_10k. It's same code, but 10000 replaced with 100000 in line 4, for PerfTest_100k:

     

    let
        Source = List.Generate( 
            ()=>0,
            each _ < 10000, 
            each _ + 1, 
            each [
                Material = Character.FromNumber( 
                    List.Min( { 
                        Int32.From( Number.RandomBetween(65, 91) ), // A-Z
                        90 
                    } ) 
                ), 
                Date = Date.AddDays( 
                    #date(2022,1,1), 
                    List.Min({ 
                        Int32.From( Number.RandomBetween( 0, 365 ) ), // 1/1/2022-12/31/2022
                        364 } ) 
                    ),
                Stock movement qty = Int64.From( Number.RandomBetween( -100, 100 ) ) // -100 - +100
            ]
        ),
        Ouput = Table.FromRecords( Source, type table [Material=text,Date=date,Stock movement qty=number] )
    in
        Ouput