Forum Discussion

Mi_80's avatar
Mi_80
Helper I
1 year ago

percentile calculating incorrectly

Hi, I am trying to use percentilex to calculate quartiles but the numbers are sometimes inaccurate and sometimes correct.  My data has a lot of outliers, is it possible this impacts the calculation? also it is sometimes incorrect after filters are added.

 

My data

I want the percentile to be for all the sums of the amount by group e.g. A --> sum amount 30

Group      Item    Amount        status          region

A                A1          10              active            amer  

A               a2             20            active               apj

B                B1             10             inactive             emea 

c                  c1            5               active                 amer

c                   c2           10             inactive                amer

 

the measure: 

PERCENTILEX.INC(CALCULATETABLE(SUMMARIZE(table,table[Group],"total",SUM(table[amount])),Filter(ALLSELECTED(table),table[status]="Active"  && table[Region]== SELECTEDVALUE(table[Region]))),[total],0.25)
 
for example with these random values tested below for summed amounts it gives me 18779.67 for  lower quartile (0.25) and 731077.14 for 0.75 when correct numbers should be 13678 and 706691, slicer is selected to a region 
 
6995006
2940950
2771286
2527232
2442400
2427449
2149473
1973790
1389482
1360260
933270
756507
755463
657920
605432
575615
473103
415675
374740
326447
324970
261365
238375
232109
227532
193430
148829
130320
117885
88682
85625
76048
66805
64546
64337
38500
29841
28835
23881
3475
663
401
0
0
0
0
0
0
0
0
0
0

 

Thanks

 

2 Replies

  • Hi Mi_80 ,

    The presence of outliers and applied filters can significantly impact the results of the PERCENTILEX.INC calculation in DAX, especially if your dataset contains extreme values or a large number of zeros. These factors can skew percentile calculations, resulting in inaccurate results. Additionally, the use of ALLSELECTED in your measure ensures that slicer and filter context is respected, which might further alter the dataset used for percentile calculations if filters are applied incorrectly or interact in unexpected ways.

    To address these issues, start by examining the intermediate summarized data. Using SUMMARIZE and CALCULATETABLE, you can review the total amounts grouped by Group to ensure they are calculated and filtered as intended:

    SUMMARIZED_TABLE = 
    CALCULATETABLE(
        SUMMARIZE(
            table,
            table[Group],
            "total", SUM(table[Amount])
        ),
        Filter(
            ALLSELECTED(table),
            table[status] = "Active" && 
            table[Region] = SELECTEDVALUE(table[Region])
        )
    )
    

    This intermediate table helps verify that the grouped totals align with your expectations. If outliers are present, you can dynamically filter them from the dataset. For example, exclude totals outside a statistically derived range:

    FILTERED_TABLE = 
    FILTER(
        SUMMARIZED_TABLE,
        [total] > 0 && [total] < SOME_THRESHOLD
    )
    

    SOME_THRESHOLD can be dynamically defined based on your data, such as using averages or standard deviations. Once the data is verified and refined, calculate the percentile while ensuring the [total] column is consistently evaluated:

    PERCENTILE = 
    PERCENTILEX.INC(
        FILTERED_TABLE,
        [total],
        0.25
    )
    

    To further troubleshoot, test the calculation without slicers or filters to confirm baseline accuracy:

    TEST_PERCENTILE = 
    PERCENTILEX.INC(
        ADDCOLUMNS(
            SUMMARIZE(table, table[Group]),
            "total", SUM(table[Amount])
        ),
        [total],
        0.25
    )
    

    This approach isolates the effects of slicers and filters and helps identify discrepancies. If inaccuracies persist, visualizing the data distribution with a histogram or boxplot can provide additional insights into the impact of outliers. These steps will help refine your measure and improve the accuracy of the percentile calculations.

     

    Best regards,

    • Mi_80's avatar
      Mi_80
      Helper I

      Thanks.

       

      I checked all the calcuated tables and they seems to give the right results and played around with removing the o and higher values but the calculations can't be improved.
      I would like to try and custom calculate the percentile and have created a measure to rank so that I can take correcsponding amounts for the group at that rank.  My measure to rank works but now I cannot use it in a lookup so I can get the amount:

      Rankx(ALLSELECTED(table[Group]),[Total Amount],,asc,Skip)
      This will give me the groups ranked correctly according to total amount but I cannot use this to get the corresponding amounts.
      Any ideas? Thanks