To understand computations done by a computer and therefore also to understand the computations performed by the Redbex Application Server it is sometimes useful to know some basic rules of how a computer handles the comparison of double values.

While the result of integer computations can usually be represented in computer systems using 32 or 64 bits of memory the representation of floating point numbers that result from computations may produce quantities that cannot be represented exactly in computer systems. The result of a floating point calculation is therefore often rounded to fit into a finite representation within a computer system. For example the quantity 0.1 cannot be represented exactly in a computer system since its binary representation is periodic.

The Application Server makes extensive use of the double (64bit) or float (32bit) floating point representations for floating point numbers. Wherever numbers that are represented in either one of these representations are used in calculations the Application Server has do deal with the rounding error.

In most cases the user will not see any effect of this internal rounding and the system will nicely do all the calculation work and will deliver reliable calculation results. However in some border cases and mostly when comparing two double values you might experience results that are founded in the rounded internal representation.

The Application Server tries to deal with this insufficiency by handling comparisons of double values using a (very small) error constant. Table 1 shows examples of comparisons done with an error constant. Wherever necessary this documentation will therefore specify the error constant used.

Table 1: Comparison of double values with an error constant