Forum Discussion
Solved
Index Query Step Not Working As Expected
Hello, I have a step that adds an index starting from an large integer, and then adds on a decimal increment. There are specific reasons to do with uniqueness to use this combination. However, Pow...
I'm pretty sure this is a floating point data type issue.
The largest precision that can be represented in a Decimal Number type is 15 digits long. The decimal separator can occur anywhere in the number. The Decimal Number type corresponds to how Excel stores its numbers. Note that a binary floating-point number can't represent all numbers within its supported range with 100% accuracy. Thus, minor differences in precision might occur when representing certain decimal numbers.
The last digits you see are artifacts of decimal-binary conversion that occur beyond the 15-digit precision limit.
I'm pretty sure this is a floating point data type issue.
The largest precision that can be represented in a Decimal Number type is 15 digits long. The decimal separator can occur anywhere in the number. The Decimal Number type corresponds to how Excel stores its numbers. Note that a binary floating-point number can't represent all numbers within its supported range with 100% accuracy. Thus, minor differences in precision might occur when representing certain decimal numbers.
The last digits you see are artifacts of decimal-binary conversion that occur beyond the 15-digit precision limit.
Thanks! Good to know and I think then nothing to worry about in this context.