Forum Discussion
Helper ISubtracting two equal measures gives wrong result
Hi everyone,
I have a strange situation where I am subtracting two variables of equal value and they provide the wrong result. (Note that when the two variables are different amounts, the subtraction works correctly.)
Here is a screenshot of the New Card Visual showing the data being returned for a specific employee with the wrong result highlighted:
Here is the measure for the Other calculation:
Each measure is exactly the same data type (Decimal number with 2 decimal points. If I change to more than 2 decimal points, they still return 2 decimal points e.g. 155.350 so it's not a data type issue.)
Any idea what is going wrong here?
Thanks in advance!
Luuk
LuukP Looks like a floating point/accuracy kind of issue. Perhaps try changing the data type from Decimal to Fixed? Or add a check that if value is less than .0000000000001 or something and if so return 0.
3 Replies
- LuukP
Greg_Deckler rsbin Thanks for your responses. Rounding the two variables solved the issue.
- Greg_Deckler
Community Champion
LuukP Looks like a floating point/accuracy kind of issue. Perhaps try changing the data type from Decimal to Fixed? Or add a check that if value is less than .0000000000001 or something and if so return 0.
- rsbin
LuukP ,
Expand your Measures to 18 decimal places and I believe you may see the difference.
To get around this, use the ROUND function in your Measures to actually round your values to 2 decimal places, rather than just "display" 2 decimal.
Hope this helps resolve your issue.
Regards,
