How to deal with Number precision in Actionscript?

by

This is my generic solution for the problem (I have blogged about this here):

var toFixed:Function = function(number:Number, factor:int) {
  return Math.round(number * factor)/factor;
}

For example:

trace(toFixed(0.12345678, 10)); //0.1
  • Multiply 0.12345678 by 10; that gives us 1.2345678.
  • When we round 1.2345678, we get 1.0,
  • and finally, 1.0 divided by 10 equals 0.1.

Another example:

trace(toFixed(1.7302394309234435, 10000)); //1.7302
  • Multiply 1.7302394309234435 by 10000; that gives us 17302.394309234435.
  • When we round 17302.394309234435 we get 17302,
  • and finally, 17302 divided by 10000 equals 1.7302.

Edit

Based on the anonymous answer below, there is a nice simplification for the parameter on the method that makes the precision much more intuitive. e.g:

var setPrecision:Function = function(number:Number, precision:int) {
 precision = Math.pow(10, precision);
 return Math.round(number * precision)/precision;
}

var number:Number = 10.98813311;
trace(setPrecision(number,1)); //Result is 10.9
trace(setPrecision(number,2)); //Result is 10.98
trace(setPrecision(number,3)); //Result is 10.988 and so on

N.B. I added this here just in case anyone sees this as the answer and doesn’t scroll down…

Leave a Comment

tech