[fpc-devel] Memory Streams unable to handle reads past 2GB

[Michael Van Canneyt]

On Tue, 6 Nov 2012, Andrew wrote:

> On 12-11-06 02:50 AM, michael.vancanneyt at wisa.be wrote:
>> Since you can only read 2GB (Count is a longint) in a single read
>> operation, it makes no sense to return a 64-bit result.
>> 
>> Note that the operating system only allows 2Gb reads anyway, even on
>> 64-bit
>> systems.
>> 
>
> Hi Michael,
>
> Thanks but that's not the problem.  I can't even write 2bytes to a 2GB 
> memory/file stream.
>
> See: http://bugs.freepascal.org/view.php?id=23284 for attached test case 
> shows simple math exactly like read/write in streams
>
> This is another glitch that I must overcome.  My sync app can't handle zip 
> files >=~2GB
>
> The problem is with overflow.  FPC raises an exception (as it should).
> I realize this is going to affect a couple base classes too.


All you demonstrate is that it is possible to create overflows.

There is nothing wrong with the base classes. They work perfectly. 
I've read/written 4Gb files. Probably you do something with the 
result which is overflowing your variables.

Michael.

See the below program.

It writes 4Gb to disk and writes the resulting number of bytes, 
which is larger than integer, but which I store in an Int64.

When I run the program:
araminta: >./testw
Wrote 4294968320 bytes

As expected. With Range checking.

-----------------------------------------------------------------------
{$mode objfpc}{$h+}
program testw;

uses classes;

var
   F : TFileStream;
   // 4 Mb buffer.
   Buf : Array[0..4*1024*1024] of byte;
   I : Integer;
   S : Int64;

  begin
    FillChar(Buf,Sizeof(Buf),' ');
    f:=TFileStream.Create('test.dat',fmCreate);
    try
      For I:=1 to 1024 do
        S:=S+F.Write(Buf,SizeOf(Buf));
    Finally
      F.Free;
    end;
    Writeln('Wrote ',S,' bytes');
end.