Forum Discussion
Performance : calculating Moving average
- Anonymous7 years ago
Anonymous -
The query is slow because there are no grouping indexes. Without index(es) to group the rows, it will iterate the entire table for each row.
The solution that I tried earlier didn't group them into big enough groups. You will need to optimize the exact number and size of grouping, but the concept is this:
//Create the Table Moving Average DAX = ADDCOLUMNS( GENERATESERIES(1,20000,1), "MyValue", CEILING(RAND()*100,1) ) //Rename the first column idx, for clarity //Calculated Columns: //This consists of a pair of groups at each level, to cover forward and backward. You can adjust the number and size of groups, but the smallest group needs to cover the necessary range. GroupLarge1 = rounddown(DIVIDE('Moving Average DAX'[idx], 500),0) GroupLarge2 = rounddown(DIVIDE('Moving Average DAX'[idx] + 250, 500),0) GroupSmall1 = rounddown(DIVIDE('Moving Average DAX'[idx], 20),0) GroupSmall2 = rounddown(DIVIDE('Moving Average DAX'[idx] + 10, 20),0) //Measure: // Moving Average = var cur_idx = MAX('Moving Average DAX'[idx]) var cur_grp1 = MAX('Moving Average DAX'[GroupSmall1]) var cur_grp2 = MAX('Moving Average DAX'[GroupSmall2]) var mytablarge = DISTINCT( UNION( ALLEXCEPT('Moving Average DAX', 'Moving Average DAX'[GroupLarge1]), ALLEXCEPT('Moving Average DAX', 'Moving Average DAX'[GroupLarge2]) ) ) //If you want to add more grouping levels, you'll need to create a pair of variables for each level and create a table variable for each level var mytabsmall = DISTINCT( UNION( FILTER(mytablarge, [GroupSmall1] = cur_grp1), FILTER(mytablarge, [GroupSmall2] = cur_grp2) ) ) return AVERAGEX( FILTER( mytabsmall, AND([idx] -10<= cur_idx , [idx] +10>= cur_idx) ), [MyValue] )
Well, Just 1 table and 20000 rows, so this is really a simple structure :-)
Is there any way to get those # of rows lower? If you use Averagex and then Filter you are iterating it twice, which is why you seeing performance degradation.
- Anonymous7 years agoNot applicable
Hi,
the table on which is performed the average depends on each row from the source table. And in my humble opinion , this moving subset table has to be filtered, not summarized, as it s not an aggregated table. But i Might be wrong :-)
Anonymous
Hi,
Well, the data has to stay at that granularity. (Basically, it is what my client wanted). So source table cannot be aggregated or filtered.
As i ve said in my original post, i am surprised that such a powerful language as DAX can't handle this kind of problem otherwise than by solutions consisting in the reduction of the source table. (After all 20000 rows isn't that much).
- Anonymous7 years agoNot applicable
Anonymous -
The query is slow because there are no grouping indexes. Without index(es) to group the rows, it will iterate the entire table for each row.
The solution that I tried earlier didn't group them into big enough groups. You will need to optimize the exact number and size of grouping, but the concept is this:
//Create the Table Moving Average DAX = ADDCOLUMNS( GENERATESERIES(1,20000,1), "MyValue", CEILING(RAND()*100,1) ) //Rename the first column idx, for clarity //Calculated Columns: //This consists of a pair of groups at each level, to cover forward and backward. You can adjust the number and size of groups, but the smallest group needs to cover the necessary range. GroupLarge1 = rounddown(DIVIDE('Moving Average DAX'[idx], 500),0) GroupLarge2 = rounddown(DIVIDE('Moving Average DAX'[idx] + 250, 500),0) GroupSmall1 = rounddown(DIVIDE('Moving Average DAX'[idx], 20),0) GroupSmall2 = rounddown(DIVIDE('Moving Average DAX'[idx] + 10, 20),0) //Measure: // Moving Average = var cur_idx = MAX('Moving Average DAX'[idx]) var cur_grp1 = MAX('Moving Average DAX'[GroupSmall1]) var cur_grp2 = MAX('Moving Average DAX'[GroupSmall2]) var mytablarge = DISTINCT( UNION( ALLEXCEPT('Moving Average DAX', 'Moving Average DAX'[GroupLarge1]), ALLEXCEPT('Moving Average DAX', 'Moving Average DAX'[GroupLarge2]) ) ) //If you want to add more grouping levels, you'll need to create a pair of variables for each level and create a table variable for each level var mytabsmall = DISTINCT( UNION( FILTER(mytablarge, [GroupSmall1] = cur_grp1), FILTER(mytablarge, [GroupSmall2] = cur_grp2) ) ) return AVERAGEX( FILTER( mytabsmall, AND([idx] -10<= cur_idx , [idx] +10>= cur_idx) ), [MyValue] ) - Bian7 years agoHelper II
Have you tried using summarize()? Not at my PBI computer so cant test it out but you should be able to do a AverageX over a sumarized table. The summariezed table would still be used for every row but is much smaller the your entire dataset.
- Anonymous7 years agoNot applicable
Anonymous ,
I got the idea, this is really nice :-)
Thanks for your time and explanations !
Piu