precision

Is there a reliable way in JavaScript to obtain the number of decimal places of an arbitrary number?

by

Historical note: the comment thread below may refer to first and second implementations. I swapped the order in September 2017 since leading with a buggy implementation caused confusion. If you want something that maps “0.1e-100” to 101, then you can try something like function decimalPlaces(n) { // Make sure it is a number and use … Read more

Precision in C floats

by

“6 digits after the decimal point” is nonesnse, and your example is a good demonstration of this. This is an exact specification of the float data type. The precision of the float is 24 bits. There are 23 bits denoting the fraction after the binary point, plus there’s also an “implicit leading bit”, according to … Read more

Determine precision and scale of particular number in Python

by

Getting the number of digits to the left of the decimal point is easy: int(log10(x))+1 The number of digits to the right of the decimal point is trickier, because of the inherent inaccuracy of floating point values. I’ll need a few more minutes to figure that one out. Edit: Based on that principle, here’s the … Read more

Why is a round-trip conversion via a string not safe for a double?

by

I found the bug. .NET does the following in clr\src\vm\comnumber.cpp: DoubleToNumber(value, DOUBLE_PRECISION, &number); if (number.scale == (int) SCALE_NAN) { gc.refRetVal = gc.numfmt->sNaN; goto lExit; } if (number.scale == SCALE_INF) { gc.refRetVal = (number.sign? gc.numfmt->sNegativeInfinity: gc.numfmt->sPositiveInfinity); goto lExit; } NumberToDouble(&number, &dTest); if (dTest == value) { gc.refRetVal = NumberToString(&number, ‘G’, DOUBLE_PRECISION, gc.numfmt); goto lExit; } DoubleToNumber(value, … Read more