Subtracting two equal measures gives wrong result

[LuukP]

Hi everyone,

I have a strange situation where I am subtracting two variables of equal value and they provide the wrong result.  (Note that when the two variables are different amounts, the subtraction works correctly.)

Here is a screenshot of the New Card Visual showing the data being returned for a specific employee with the wrong result highlighted:

Here is the measure for the Other calculation:

Each measure is exactly the same data type (Decimal number with 2 decimal points.  If I change to more than 2 decimal points, they still return 2 decimal points e.g. 155.350 so it's not a data type issue.)

Any idea what is going wrong here?

Thanks in advance!

Luuk

[Greg_Deckler · Accepted solution]

@LuukP Looks like a floating point/accuracy kind of issue. Perhaps try changing the data type from Decimal to Fixed? Or add a check that if value is less than .0000000000001 or something and if so return 0.

[LuukP]

@Greg_Deckler @rsbin Thanks for your responses.  Rounding the two variables solved the issue.

[rsbin · Accepted solution]

@LuukP ,

Expand your Measures to 18 decimal places and I believe you may see the difference.

To get around this, use the ROUND function in your Measures to actually round your  values to 2 decimal places, rather than just "display" 2 decimal.

Hope this helps resolve your issue.

Regards,

[Greg_Deckler · Accepted solution]

@LuukP Looks like a floating point/accuracy kind of issue. Perhaps try changing the data type from Decimal to Fixed? Or add a check that if value is less than .0000000000001 or something and if so return 0.