[fpc-devel] rounding precision error with integer and floating point division

[Seth Grover]

In a project I'm working on making platform portable, I was getting a
different answer for a calculation on i386 vs. x86_64. I boiled it
down to this:

===============================
program project1;

{$mode objfpc}{$H+}

uses
  Classes,
  SysUtils;

var
  iHour : longword;
  r64 : double;
begin
  iHour := 17608383;

  r64 := (iHour / 24.0);
  Writeln (FloatToStrf(r64,ffFixed,15,4));

  r64 := (double(iHour) / 24.0);
  Writeln (FloatToStrf(r64,ffFixed,15,4));
end.
===============================

Output when compiled for i386-linux:
733682.6250
733682.6250

Output when compiled for x86_64-linux:
733682.6875
733682.6250

Notice that in x86_64, unless you explicitly cast the longword as a
double before the division, you get some sort of rounding error or
loss of precision. For what it's worth, in C (compiled with gcc) both
64-bit and 32-bit give the same (correct) answer, 733682.6250.

In 32-bit, the two assignments generate the following machine code
(both give the same answer):

# [17] r64 := (iHour / 24.0);
  movl  %eax,U_P$PROJECT1_IHOUR
  movl  %eax,-48(%ebp)
  movl  $0,-44(%ebp)
  fildq  -48(%ebp)
  fdivs  _$PROJECT1$_Ld2
  fstpl  U_P$PROJECT1_R64

# [20] r64 := (double(iHour) / 24.0);
  movl  U_P$PROJECT1_IHOUR,%eax
  movl  %eax,-48(%ebp)
  movl  $0,-44(%ebp)
  fildq  -48(%ebp)
  fdivl  _$PROJECT1$_Ld3
  fstpl  U_P$PROJECT1_R64

In 64-bit, here is the machine code generated (the first one gives the
wrong answer, the second one is right):

# [17] r64 := (iHour / 24.0);
  movl  U_P$PROJECT1_IHOUR,%eax
  cvtsi2ssq  %rax,%xmm0
  divss  _$PROJECT1$_Ld2,%xmm0
  cvtss2sd  %xmm0,%xmm0
  movsd  %xmm0,U_P$PROJECT1_R64

# [20] r64 := (double(iHour) / 24.0);
  movl  U_P$PROJECT1_IHOUR,%eax
  cvtsi2sdq  %rax,%xmm0
  divsd  _$PROJECT1$_Ld3,%xmm0
  movsd  %xmm0,U_P$PROJECT1_R64

Although I can avoid the problem by casting the longword to a double
point before the division, this seems like it should be done
automatically to make results across architectures consistent. What
are your thoughts? Should I log an issue in mantis?

Thanks,

-SG

--
This email is fiction. Any resemblance to actual events
or persons living or dead is purely coincidental.

Seth Grover