[fpc-devel] FormatFloat bug

[Graeme Geldenhuys]

On 5/30/07, Jonas Maebe <jonas.maebe at elis.ugent.be> wrote:
>
> 0.005, as extended, is actually 0.004999999999999999999898356046329484
> As double, it is actually 0.0050000000000000001

For now I have managed to implement a work-around in my code which
passes all unit tests I could throw at it.

I first look at what precision is required from the user, by looking
at the format string (eg: '.  I then use a tiSetPrecision function
that rounds the Extended (or Double or Single) value passed in to the
correct amount of decimals using a default rounding method (my default
is Common Rounding aka HalfUp). It then passes the new extended which
contains the already rounded value to the FormatFloat() so no round
needs to be done, only formatting.  This works perfectly and gives
consistent results under both FPC and Delphi.

The rounding routine is based on the code posted to the Borland
Developer Networks forums for all to use.

Here is an extract which highlights the exact problem we are talking about:
---------------------------------------
These routines round input values to fit as closely as possible to an
output number with desired number of decimal fraction digits.

Because, in general, numbers with decimal fractions cannot be exactly
represented in IEEE-754 floating binary point variables, error limits
are used to determine if the input numbers are intended to represent
an exact decimal fraction rather than a nearby value. Thus an error
limit will be taken into account when deciding that a number input
such as 1.295, which internally is represented 1.29499 99999 …, really
should be considered exactly 1.295 and that 0.29999 99999 ... really
should be interpreted as 0.3 when applying rounding rules.

Nine rounding rules are implemented.
--------------------------------

The implementation as used in the tiOPF project can be viewed here
(around line 3200):   http://tinyurl.com/2jawq9
The function comment also includes the original link to the BDN page.


I think the Free Pascal developers will find this code very useful.
Hopefully it can find it's way into the FPC compiler.


-- 
Graeme Geldenhuys
General error, hit any user to continue.