[fpc-devel]Problems with AnsiStrings (memory leak)

[Florian Klaempfl]

At 15:50 19.11.00 +0100, you wrote:
>Hello,
>
>I encountered a problem using AnsiStrings. It isn't a real bug, but
>can make problems, like it does for me.
>
>I noticed it with Free Pascal version 1.1, snapshot 17/11/2000 under
>Linux (i386).
>
>The problem is that when altering AnsiStrings the greatest available
>memory block is becoming smaller and smaller, but much smaller than it
>is really needed.
>
>For example, I made the following example program that creates a
>string of 5000 chars by adding one char at a time. I gave the program
>1 mb of heap memory but it runs out of memory.
>
>Here is this program:
>
>-----------------8<------------------8<-------------------------------
>
>{$MODE Delphi}       // I prefer Delphi programming style
>{$M 16387, 1000000}  // 16 kb stack, 1 mb heap
>
>
>program StringTest;
>
>var
>   S        : AnsiString;  // test-string
>   i        : Integer;     // count-variables
>   StartMem : Integer;     // total free memory at startup
>   StartMax : Integer;     // greatest free memory block atstartup
>   c        : Char;
>
>begin
>   { This disables the automatic heap growing when more memory is needed.
>     As with the $M switch defined, the program has 1 mb of heap available,
>     that should be enough for a 5,000 chars string... }
>   HeapError := NIL;
>
>
>   writeln('ANSISTRINGS TEST');
>   writeln;
>   writeln('Please note heap griwing is disabled. ',
>           'The program may only use 1 mb of heap.');
>
>   StartMem := MemAvail;
>   StartMax := MaxAvail;
>   writeln('Memory at program start: ', StartMem, ' bytes free, ',
>            StartMax, ' in largest block.');
>
>   writeln('Hit RETURN to continue...');
>   ReadLn;
>
>   c:='.';
>
>   S:='start';
>   For i:=1 to 5000 do
>   begin
>     S := S + c; // simple instruction to add one single character to a string
>
>     writeln('String length = ', Length(S),' chars;  memory: ',
>             MemAvail,' free, ',MaxAvail,' largest block.');
>   end;
>
>   writeln('memory loss: ',StartMem-MemAvail, ' bytes, ',
>           StartMax-MaxAvail, ' in largest block');
>
>   writeln('done.');
>   writeln('S="',S,'"');
>   writeln('That is, ', Length(S),' chars.');
>   writeln('Memory at program end: ', MemAvail, ' bytes free, ',
>           MaxAvail, ' in largest block.');
>
>end.
>
>-----------------8<------------------8<-------------------------------
>
>
>And on my machine it generates the following output:
>
>ANSISTRINGS TEST
>
>Please note heap griwing is disabled. The program may only use 1 mb of heap.
>Memory at program start: 999920 bytes free, 999920 in largest block.
>Hit RETURN to continue...
>String length = 6 chars;  memory: 999856 free, 999856 largest block.
>String length = 7 chars;  memory: 999856 free, 999824 largest block.
>String length = 8 chars;  memory: 999856 free, 999824 largest block.
>String length = 9 chars;  memory: 999856 free, 999824 largest block.
>String length = 10 chars;  memory: 999856 free, 999824 largest block.
>[...]
>String length = 3958 chars;  memory: 995904 free, 7904 largest block.
>String length = 3959 chars;  memory: 995904 free, 3984 largest block.
>String length = 3960 chars;  memory: 995904 free, 7904 largest block.
>String length = 3961 chars;  memory: 995904 free, 3984 largest block.
>String length = 3962 chars;  memory: 995904 free, 7904 largest block.
>String length = 3963 chars;  memory: 995904 free, 3984 largest block.
>String length = 3964 chars;  memory: 995904 free, 7904 largest block.
>String length = 3965 chars;  memory: 995904 free, 3984 largest block.
>String length = 3966 chars;  memory: 995904 free, 7904 largest block.
>String length = 3967 chars;  memory: 995904 free, 3984 largest block.
>Runtime error 203 at 0x080503B0
>   0x080503B0
>   0x0804FF40
>   0x0804C1F6
>   0x0804C696
>   0x0804C379
>   0x080539BF
>   0x080480B0
>   0x00000000
>
>
>As you can see, the used memory begins to grow exponencially (well, to
>be exact, the free memory sinks correctly).
>
>Can you correct this behavior? I have a big problem with a web server
>application that has to process large strings and in some cases it
>runs out of memory, even with 15 mb heap.

This is nearly impossible to fix because the problem is the memory 
fragmentation:
there are a lot of memory blocks allocated and deallocated but the
new blocks never fit into the available free blocks because there
is new data concatenated.

I suggest that you estimate first, how long you string will get and then
you do a setlength(s,<estimated length>);

If you can't estimate the length, but you know that you strings are quite 
large,
say several kB, you do a manual setlength in some kB steps (untested):

aktsize:=1024;
setlength(s,1024);
For i:=1 to 5000 do
   begin
     S := S + c; // simple instruction to add one single character to a string
     if length(s)=aktsize then
       begin
          inc(aktsize,1024);
          setlength(s,aktsize);
       end;
     writeln('String length = ', Length(S),' chars;  memory: ',
             MemAvail,' free, ',MaxAvail,' largest block.');
   end;

This will also improve the performance of your program a lot because much
fewer memory (re)allocations are performed and because setlength can use 
reallocmem
while s:=s+c always performs a copy with move.