Forum Discussion
Non Distinct Count Calculation Efficiency
- 4 years ago
Thanks for testing those out!
Good to get an idea of relative performance, and glad some of them are performing better, but it feels as though performance should be much better!
Period Table filtering
Looking again at the code you posted earlier to determine the date range to filter, have I understood correctly that 'Period Table' your sole date table?
I think you could improve performance by rewriting the logic to filter 'Period Table' like this:
VAR curPeriodIndex = -- I would generally prefer MAX rather than MIN -- Doesn't affect performance but makes more sense if filtering -- on multiple Periods. MAX ( 'Period Table'[Period Index] ) VAR startPeriod = curPeriodIndex - 11 RETURN CALCULATE ( < Some Measure >, ALL ( 'Period Table' ), 'Period Table'[Period Index] >= startPeriod, 'Period Table'[Period Index] <=curPeriod )This version removes filters on 'Period Table' then applies filters to the Period Index column (rather than Date column). The original version using FILTER ( ALL ( 'Period Table' ), ... ) is an iteration over the entire 'Period Table' which can be expensive.
Would you be able to post a model diagram, or a PBIX with an empty 'All Order Table', and I can generate a fact table at my end?
Repeat Customers calculation itself
Going back to the different DAX options for Repeat Customers, some ideas occurred to me, that I probably should have thought of earlier!
Version 5
Uses GENERATE to remove Customers whose first & last order are the same.
If FirstOrder = LastOrder, then EXCEPT ( FirstOrder, LastOrder ) is empty, and that Customer's row won't appear in result.
Repeat Customers Version 5 = VAR RepeatCustomers = GENERATE ( VALUES ( 'All Order Table'[Customer] ), VAR FirstOrder = FIRSTNONBLANK ( 'All Order Table'[OrderNbr], 0 ) VAR LastOrder = LASTNONBLANK ( 'All Order Table'[OrderNbr], 0 ) RETURN EXCEPT ( FirstOrder, LastOrder ) ) VAR NumRepeatCustomers = COUNTROWS ( RepeatCustomers ) RETURN NumRepeatCustomersVersion 6
Use HASONEVALUE to see if there is not exactly one OrderNbr for a given Customer. This might be optimised to stop counting when it knows there are 2+ values.
Repeat Customers Version 6 = VAR RepeatCustomers = FILTER ( VALUES ( 'All Order Table'[Customer] ), NOT CALCULATE ( HASONEVALUE ( 'All Order Table'[OrderNbr] ) ) ) VAR NumRepeatCustomers = COUNTROWS ( RepeatCustomers ) RETURN NumRepeatCustomersVersion 7
Same as Version 6 but use COUNTROWS (this is really the same logic as Version 3):
Repeat Customers Version 7 = VAR RepeatCustomers = FILTER ( VALUES ( 'All Order Table'[Customer] ), NOT CALCULATE ( COUNTROWS ( 'All Order Table' ) ) = 1 ) VAR NumRepeatCustomers = COUNTROWS ( RepeatCustomers ) RETURN NumRepeatCustomersI'm hoping some of this gets us closer to acceptable performance!
Regards,
Owen
Nice, Owen!! Great variations here!
My notes:
First of all, notes about the period table:
CALCULATE (
< Some Measure >,
ALL ( 'Period Table' ),
'Period Table'[Period Index] >= startPeriod,
'Period Table'[Period Index] <=curPeriod
)Typically, instead of the above, which I have used in the past, I have been using
CALCULATE (
< Some Measure >,
DATESBETWEEN('Period Table'[Day],startOfPeriod,endOfPeriod)
) This seems to be slightly faster. I also have been using it when I might have some other filter active that I don't want to strip away, and only want to strip away the date (so avoids using ALL). I tested the two variations above on your version #5, and performance was slightly improved (6.3 seconds vs 7.2 seconds) - that was giving the wrong data, but the efficiency test was still accurate. I'll explain more below
I also want to be clear that I'm also using this for the "all customer" calculation here:
VAR allCust =
CALCULATE(
DISTINCTCOUNT('All Order Table'[Customer]),
DATESBETWEEN('Period Table'[Day],startOfPeriod,endOfPeriod)
)When running this on its own, it takes only about 1.5 seconds. So I know there isn't a problem with this part of the code.
So speaking of the versions, Version 5 was definitely the best yet at 6.3 seconds!
Just to be clear, here is what the code looks like with the modification
VAR RepeatCustomers =
GENERATE (
CALCULATETABLE(
VALUES ( 'All Order Table'[Customer] ),
DATESBETWEEN('Period Table'[Day],startOfPeriod,endOfPeriod)
),
VAR FirstOrder =
FIRSTNONBLANK ( 'All Order Table'[OrderNbr], 0 )
VAR LastOrder =
LASTNONBLANK ( 'All Order Table'[OrderNbr], 0 )
RETURN
EXCEPT ( FirstOrder, LastOrder )
)
VAR NumRepeat =
COUNTROWS ( RepeatCustomers )This took only 6.2 seconds. But then I realized the data was incorrect. Again, because the FIRSTNONBLANK was removing the filter context of DATESBETWEEN. So I had to add another calculate there. This threw us back down to 26 seconds calc time, and it looked like this:
VAR RepeatCustomers =
GENERATE (
CALCULATETABLE(
VALUES ( 'All Order Table'[Customer] ),
DATESBETWEEN('Period Table'[Day],startOfPeriod,endOfPeriod)
),
VAR FirstOrder =
CALCULATETABLE(
FIRSTNONBLANK ( 'All Order Table'[OrderNbr], 0 ),
DATESBETWEEN('Period Table'[Day],startOfPeriod,endOfPeriod)
)
VAR LastOrder =
CALCULATETABLE(
LASTNONBLANK ( 'All Order Table'[OrderNbr], 0 ),
DATESBETWEEN('Period Table'[Day],startOfPeriod,endOfPeriod)
)
RETURN
EXCEPT ( FirstOrder, LastOrder )
)
VAR NumRepeat =
COUNTROWS ( RepeatCustomers )
This version blew my mind a little bit, Owen! I didn't realize you could write VAR INSIDE a calculated virtual table?!?! It really threw me off. Can you help me understand a little bit how this works?
I think the "GENERATE" calc throws me off a little bit, because I haven't used it much before. Only generate series.
I see that you first create a one column unique table of customers. Then my hunch here is that the FirstOrder "VAR" table gets applied at the row level to the unique customers? Or is it creating another table next to the first and applying a filter from the first of the customer? This is throwing me a bit. And then the EXCEPT is working on FirstOrder and LastOrder table, but then it applies the filter back to the customer table?
Version 6 and Version 7 - these I understand how you're calculating, but again, really smart usage here. Version 6 took about 19 seconds, but 6 was only at 14 seconds. I would not have thought of using calculate (didn't think it was possible) as a filter argument! Genius! Unfortunately, again, I had to add the DATESBETWEEN context to get this to work. And I had to add it to the VALUES table and to the CALCULATE formula. So to be clear, here is the final version:
VAR RepeatCustomers =
FILTER (
CALCULATETABLE(
VALUES ( 'All Order Table'[Customer] ),
DATESBETWEEN('Period Table'[Day],startOfPeriod,endOfPeriod)
),
NOT CALCULATE ( COUNTROWS ( 'All Order Table' ),DATESBETWEEN('Period Table'[Day],startOfPeriod,endOfPeriod) ) = 1
)
VAR NumRepeat =
COUNTROWS ( RepeatCustomers )
As for building out a PBIX, happy to do so, but I can't post the files on the forum - it won't let me.. Thought you could. Anyway, I can send you a PM and could email it to you, or if there is another way to upload here, happy to do that as well.