[fpc-pascal] Loss of precision when using math.Max()

[Sven Barth]

Am 29.06.2018 um 18:45 schrieb Alan Krause:
> I stumbled upon something the other day that was causing numerical 
> differences between compiled Delphi and FPC code. Executing the 
> following sample console application illustrates the issue clearly:
>
> program test;
>
> uses
>   math, SysUtils;
>
> var
>   arg1 : double;
>   arg2 : double;
>   res  : double;
> begin
>   arg1 := 100000.00;
>   arg2 := 72500.51;
>   writeln( 'arg1 = ' + FormatFloat( '0.00000000', arg1 ) );
>   writeln( 'arg2 = ' + FormatFloat( '0.00000000', arg2 ) );
>
>   res := arg1 - arg2;
>   writeln( 'arg1 - arg2 = ' + FormatFloat( '0.00000000', res ) );
>   writeln( 'Max( arg1 - arg2, 0 ) = ' + FormatFloat( '0.00000000', 
> Max( res, 0 ) ) );
>   writeln( 'Max( arg1 - arg2, 0.0 ) = ' + FormatFloat( '0.00000000', 
> Max( res, 0.0 ) ) );
> end.
>
> --- begin output (Linux x86_64) ---
>
> arg1 = 100000.00000000
> arg2 = 72500.51000000
> arg1 - arg2 = 27499.49000000
> *Max( res, 0 ) = 27499.49023438*
> Max( res, 0.0 ) = 27499.49000000
>
> --- end output ---
>
> I am guessing that the integer value of zero is causing the wrong 
> overloaded function to be called? I was able to solve the problem in 
> my code by replacing the 0 with 0.0.

The compiler converts the 0 to the type with the lowest precision that 
can hold the value (or the largest if none can hold it exactly). For 0 
this is already satisfied by Single, so the compiler essentially has the 
parameter types Double and Single. For some reason (I don't know whether 
it's due to a bug or by design) it picks the Single overload instead of 
the Double one.
Someone who knows more about the compiler's overload handling would need 
to answer why it favors (Single, Single) over (Double, Double) for 
(Double, Single) parameters (or (Single, Double), the order doesn't 
matter here).

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freepascal.org/pipermail/fpc-pascal/attachments/20180629/3957cb23/attachment.html>