Forum Discussion

mahenkj2's avatar
mahenkj2
Icon for Solution Sage rankSolution Sage
5 years ago
Solved

DAX calculated column to get running sum which resets on particular conditions

I need help in writing a DAX calculated column to get running sum which resets on particular conditions. sample data is as below:

 

Created Date                 Index    clean trigger  machine    type    counter      Comments

05/01/2019 15:06:4711111clean trigger + type change resets the counter to 1
05/01/2019 17:38:1620101counter is maintained at previous calculation when trigger and type are 0
05/01/2019 22:01:2730101 
06/01/2019 03:13:0740101 
06/01/2019 06:44:0950101 
06/01/2019 08:02:3960101 
06/01/2019 10:58:3170112alone change in type triggers increment in counter
06/01/2019 11:32:0680102 
06/01/2019 14:48:3090102 
06/01/2019 19:26:21100102 
06/01/2019 21:07:37111111clean trigger + type change resets the counter to 1
06/01/2019 22:01:41120112 
06/01/2019 23:09:44130102 
07/01/2019 02:03:59140113alone change in type triggers increment in counter
07/01/2019 02:37:54150103 
07/01/2019 07:27:37160103 
07/01/2019 12:39:22171111 
07/01/2019 13:20:19180101 
07/01/2019 13:42:59190101 
07/01/2019 17:00:16200112 
07/01/2019 17:20:32210102 
07/01/2019 22:25:01220102 
08/01/2019 05:07:33230102 
08/01/2019 05:12:25240102 
08/01/2019 08:46:19250211now there is machine change also present, trigger is not present
08/01/2019 09:02:42260201 
08/01/2019 10:18:31270201 
08/01/2019 15:29:39280201 
08/01/2019 18:29:02290201 
08/01/2019 19:16:47300201 
08/01/2019 19:46:35310212 
08/01/2019 22:14:24320202 
08/01/2019 22:50:52330202 
09/01/2019 05:33:32340202 
09/01/2019 09:04:43351211 
09/01/2019 10:03:37360212 
09/01/2019 10:44:10370202 
09/01/2019 14:10:18380213 
09/01/2019 14:38:07390203 
09/01/2019 20:47:29400203 
10/01/2019 00:18:21411211 
10/01/2019 01:07:44420201 
10/01/2019 01:30:54430201 
10/01/2019 06:15:20440212 
10/01/2019 10:01:45450202 
10/01/2019 14:40:09460202 
10/01/2019 16:10:05470202 
10/01/2019 18:51:48480202 

 

Clean trigger resets counter to 1, and very incidence of type=1 increments counter to +1. If there are trigger and type are 0, they keep counter at last calculated value. As a main group, machine column is there, for each machine, the above calculation is executed independently.

I know to use VAR and countx related formulae to calculate running sum which filters the limited rows and then counts/sums to get the running sum, but here filtering trigger column does not make sense as I need to filter just last occurrence of trigger=1. I think, FIRSTNONBLANK kind of function be of help here, but I am unable to get it well.

 

Please advise.

  • Hi again mahenkj2 

     

    I've attached a PBIX illustrating a possible Power Query method using List.Generate, which I'm hoping refreshes in an acceptable time.

     

    This is the code for the query called Counter that generates the counter values. This query relies on a table called YourTableSource with columns Index, clean triggermachine, and type:

     

    let
        // Select required columns from source table
        // Table.Buffer used but may not be necessary
        SourceTable = Table.Buffer(YourTableSource[[Index],[clean trigger],[machine],[type]]),
        RowCount = Table.RowCount(SourceTable),
        GenerateCount =
          List.Generate(
            // Function with no arguments to generate first counter record
            // Record contains [Index, machine, counter]
            ()=>let CurrentRow = SourceTable{[Index= 1]}, CurrentType = CurrentRow[type], CurrentMachine = CurrentRow[machine], CurrentCounter = if CurrentType = 1 then 1 else 0
                in [Index = 1, machine = CurrentMachine, counter = CurrentCounter],
            // Condition to continue loop
            each [Index] <= RowCount,
            // Function to produce next counter record
            each let CurrentRow = SourceTable{[Index = [Index]+1]},
             CurrentType = CurrentRow[type], CurrentMachine = CurrentRow[machine], CurrentCleanTrigger = CurrentRow[clean trigger],
             CurrentCounter = (if CurrentMachine <> [machine] or CurrentCleanTrigger = 1 then 0 else [counter]) + CurrentType
             in [Index = [Index]+1, machine = CurrentMachine, counter = CurrentCounter]    
        ),
        // Convert into table to be joined with main table
        #"Converted to Table" = Table.FromList(GenerateCount, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
        #"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"Index", "counter"}, {"Index", "counter"}),
        #"Changed Type" = Table.TransformColumnTypes(#"Expanded Column1",{{"Index", Int64.Type}, {"counter", Int64.Type}})
    in
        #"Changed Type"

     

    The output counter values appear correct but please test with your actual data.

     

    The important parts to check are the 1st and 3rd arguments of List.Generate.

    The final table is called YourTable, which joins YourTableSource and Counter.

     

    On the original DAX version, the poor performance is essentially due to the quadratic relationship between number of sets to be included in the calculation and the number of rows, which eats up RAM. We could possibly rewrite, but I still think Power Query performance would be better.

     

    Let me know how this goes!

    Regards,

    Owen

9 Replies

  • Hi mahenkj2 

    Just an initial comment: this might be better handled in Power Query, using List.Generate.

     

    However, you can write a calculated column in DAX like this:

    counter =
    VAR CurrentIndex = YourTable[Index]
    VAR CurrentMachineStartIndex =
        CALCULATE (
            MIN ( YourTable[Index] ),
            ALLEXCEPT ( YourTable, YourTable[machine] )
        )
    VAR LastCleanTriggerIndex =
        CALCULATE (
            MAX ( YourTable[Index] ),
            ALLEXCEPT ( YourTable, YourTable[machine] ),
            YourTable[clean trigger] = 1,
            YourTable[Index] <= CurrentIndex
        )
    VAR LastResetIndex =
        MAX ( CurrentMachineStartIndex, LastCleanTriggerIndex )
    VAR TypeCumulativeCountSinceLastReset =
        CALCULATE (
            SUM ( YourTable[type] ),
            ALLEXCEPT ( YourTable, YourTable[machine] ),
            YourTable[Index] >= LastResetIndex,
            YourTable[Index] <= CurrentIndex
        )
    RETURN
        TypeCumulativeCountSinceLastReset

    At least this works with your sample data and my reading of the rules, but there may well be some tweaking required if I misunderstood something.

    At least this general structure should work.

     

    Regards,

    Owen

    • mahenkj2's avatar
      mahenkj2
      Icon for Solution Sage rankSolution Sage

      Thanks. I am now testing it, it is crashing my Powerbi file, so now trying to work out through DAX studio. There code can execute but I have to now check the validity of the results.

       

      I was even intending to request a Power query solution, incase that is more efficient. Calculated column have limitation that it is not so visual, second it can not be referenced later in power query in another table. If possible, can you also add a power query solution please! I would like to give a try to both.

      • OwenAuger's avatar
        OwenAuger
        Icon for Super User rankSuper User

        You're welcome! I had wondered how large the real table is, as these sorts of calculations in DAX can be rather memory-intensive.

         

        I will look at the Power Query version using List.Generate when I have a few minutes later and get back to you. I expect it to perform better and probably be a bit more intuitive.

         

        Kind regards,

        Owen

  • mahenkj2's avatar
    mahenkj2
    Icon for Solution Sage rankSolution Sage

    OwenAugerI could use the solution offered in my actual work. Thanks a lot. In power query, it looks quite lighter than in DAX.

    It would be more helpful, if you can explain that how the code is actually working. I have tough time to really understand it and so I just replicated it.

    There are many resources on DAX/PowerBI, is there something for M as well?

    • OwenAuger's avatar
      OwenAuger
      Icon for Super User rankSuper User

      That's good news 🙂

       

      Sorry, I tried to post this reply some days ago, but I was unexpectedly logged out and the post disappeared.

       

      The key part of the query is List.Generate.

      This function works a bit like a for-loop, and creates a list of items, adding one for each iteration of the loop. The arguments it takes are :

      1. Initial value
      2. Condition to continue loop
      3. Function to derive next value from current value
      4. Optional argument which I didn't use, which specifies the value to be returned based on the values produced in steps 1 & 3 (e.g. a subset of fields from a record rather than the whole record)

      1. The Initial value is this, which constructs a record from the first row of SourceTable:

      ()=>let CurrentRow = SourceTable{[Index= 1]}, CurrentType = CurrentRow[type], CurrentMachine = CurrentRow[machine], CurrentCounter = if CurrentType = 1 then 1 else 0
      in [Index = 1, machine = CurrentMachine, counter = CurrentCounter]

      2. The condition to continue loop is this, which checks if the Index of the current row <= the total number of rows:

      each [Index] <= RowCount,

      3. The function to derive the next value from the current value (which is a record) is:

      each let CurrentRow = SourceTable{[Index = [Index]+1]},
               CurrentType = CurrentRow[type], CurrentMachine = CurrentRow[machine], CurrentCleanTrigger = CurrentRow[clean trigger],
               CurrentCounter = (if CurrentMachine <> [machine] or CurrentCleanTrigger = 1 then 0 else [counter]) + CurrentType
               in [Index = [Index]+1, machine = CurrentMachine, counter = CurrentCounter]    
          )

       This is the most important part, as it contains the logic for resetting and incrementing the counter.

       

      There are various resources out there on M.

      A good article on List.Generate is here on Chris Webb's blog.

      Also the Microsoft documentation is not too bad these days:

      https://docs.microsoft.com/en-us/powerquery-m/m-spec-introduction

       

      I also have a book M is for (Data) Monkey which I found quite good.

       

      Kind regards,

      Owen