[fpc-devel] Possible bug in fpc_round_real for softfloat?

[Alexander Hofmann]

Dear all,

while investigating a bug in an application designed for ARM with
floating point emulation enabled, I stumbled upon the following problem:

When rounding large numbers to int64, some actually get rounded to
something negative, e.g:

 round(1.5000000000000000E+018) => 1500000000000000000
 round(1.5000005497558139E+018) => -491856199680

Tested with trunk and 3.0.0 on a raspberry pi (armsf).

At first I though it's specific to ARM, but it can be reproduced also on
e.g. x86_64 by copying fpc_round_real from rtl/genmath.inc and using
this directly. By the way, the above fractional numbers differ by only
one bit, which is bit 32 (or the sign bit in a 32bit number).

I think the culprit is in line 1342 of genmath.inc, i.e.

            result:=((int64(hx) shl 32) or float64low(d)) shl (j0-52);

float64low(d) will return the lower 32 bit of the float as longint,
where the sign is negative in case of the second number. The compiler
seems to expand this to a 64bit signed integer, by keeping the number
not the bits - thus the invalid result. If this line is changed to

            result:=((int64(hx) shl 32) or *dword*(float64low(d))) shl
(j0-52);

both floats are rounded correctly. It needs to be an unsigned number, it
does not work with an explicit cast to int64.

Note that this code-path will only be put into action if the exponent of
the base-two fractional number is larger 51 and the float thus lacks the
fractional part. In my point of view, the rest of the code is not prone
to this error; in line 1361 of genmath.inc the result of float64low is
shifted right first, which will always leave a 0 at the position of the
"sign bit".

I am, however, not sure if this is because my compilers all have been
compiled e.g. with the wrong switches (leading to some obscure
optimization) or if this is indeed an error. I didn't find anything
about this on the list or the net.

Attached is a little program illustrating the problem, most of the code
is a 1:1 copy from genmath.inc


With best regards,

Alex

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freepascal.org/pipermail/fpc-devel/attachments/20200218/5a120816/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: round_test.pas
Type: text/x-pascal
Size: 3090 bytes
Desc: not available
URL: <http://lists.freepascal.org/pipermail/fpc-devel/attachments/20200218/5a120816/attachment.pas>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freepascal.org/pipermail/fpc-devel/attachments/20200218/5a120816/attachment.sig>