[fpc-pascal] Loss of precision when using math.Max()
Alan Krause
alank at shermanloan.com
Fri Jun 29 18:45:31 CEST 2018
I stumbled upon something the other day that was causing numerical
differences between compiled Delphi and FPC code. Executing the following
sample console application illustrates the issue clearly:
program test;
uses
math, SysUtils;
var
arg1 : double;
arg2 : double;
res : double;
begin
arg1 := 100000.00;
arg2 := 72500.51;
writeln( 'arg1 = ' + FormatFloat( '0.00000000', arg1 ) );
writeln( 'arg2 = ' + FormatFloat( '0.00000000', arg2 ) );
res := arg1 - arg2;
writeln( 'arg1 - arg2 = ' + FormatFloat( '0.00000000', res ) );
writeln( 'Max( arg1 - arg2, 0 ) = ' + FormatFloat( '0.00000000', Max(
res, 0 ) ) );
writeln( 'Max( arg1 - arg2, 0.0 ) = ' + FormatFloat( '0.00000000', Max(
res, 0.0 ) ) );
end.
--- begin output (Linux x86_64) ---
arg1 = 100000.00000000
arg2 = 72500.51000000
arg1 - arg2 = 27499.49000000
*Max( res, 0 ) = 27499.49023438*
Max( res, 0.0 ) = 27499.49000000
--- end output ---
I am guessing that the integer value of zero is causing the wrong
overloaded function to be called? I was able to solve the problem in my
code by replacing the 0 with 0.0.
Thanks,
Alan
--
Alan Krause
*President @ Sherman & Associates, Inc.*
Office: (760) 634-1700 Web: https://www.shermanloan.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freepascal.org/pipermail/fpc-pascal/attachments/20180629/c145be1c/attachment.html>
More information about the fpc-pascal
mailing list