Forum Discussion

LuukP's avatar
LuukP
Icon for Helper I rankHelper I
2 years ago
Solved

Subtracting two equal measures gives wrong result

Hi everyone,   I have a strange situation where I am subtracting two variables of equal value and they provide the wrong result.  (Note that when the two variables are different amounts, the subtra...
Need Help
  • Greg_Deckler's avatar
    Greg_Deckler
    2 years ago

    LuukP Looks like a floating point/accuracy kind of issue. Perhaps try changing the data type from Decimal to Fixed? Or add a check that if value is less than .0000000000001 or something and if so return 0.

  • rsbin's avatar
    rsbin
    2 years ago

    LuukP ,

    Expand your Measures to 18 decimal places and I believe you may see the difference.

    To get around this, use the ROUND function in your Measures to actually round your  values to 2 decimal places, rather than just "display" 2 decimal.

    Hope this helps resolve your issue.

    Regards,

rsbin's avatar
rsbin
Icon for Community Champion rankCommunity Champion
2 years ago

LuukP ,

Expand your Measures to 18 decimal places and I believe you may see the difference.

To get around this, use the ROUND function in your Measures to actually round your  values to 2 decimal places, rather than just "display" 2 decimal.

Hope this helps resolve your issue.

Regards,