[fpc-pascal] heap size growing

[Burkhard Carstens]

Am Mittwoch, 14. Januar 2009 04:50 schrieb Seth Grover:
> I've got a fairly large system (originally written in Delphi, then
> moved to Kylix, and now finally in FPC 2.2.2) which is doing a lot of
> different things. Various threads are running which execute reports,
> communiate with other devices, query data, process notifications,
> send and receive data on sockets, etc.
>
> For analysis of how its performing, I've got a thread which prints
> out the information from GetFPCHeapStatus every five minutes. What I
> notice is that CurrHeapUsed is not growing at all, it stays pretty
> much the same over hours of running. However, CurrHeapSize is pretty
> much continuously growing. For example (cu is current heap used, cs
> is current heap size, cf is current heap free, mu is max heap used,
> ms is max heap size):
>
> heap status: cu=1,407,664, cs=34,930,688, cf=33,523,024,
> mu=33,621,024, ms=35,749,888
> heap status: cu=1,411,264, cs=66,715,648, cf=65,304,384,
> mu=33,635,232, ms=67,239,936
> heap status: cu=1,409,296, cs=98,435,072, cf=97,025,776,
> mu=33,635,232, ms=98,467,840
> heap status: cu=1,411,952, cs=161,873,920, cf=160,461,968,
> mu=33,674,176, ms=161,873,920
> heap status: cu=1,409,056, cs=193,593,344, cf=192,184,288,
> mu=33,674,176, ms=193,593,344
> heap status: cu=1,409,360, cs=193,593,344, cf=192,183,984,
> mu=33,674,176, ms=193,626,112
> heap status: cu=1,409,552, cs=225,312,768, cf=223,903,216,
> mu=33,674,176, ms=225,345,536
> heap status: cu=1,408,704, cs=257,032,192, cf=255,623,488,
> mu=33,674,176, ms=257,064,960
> etc.
>
> top shows:
> PID   USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  P
> COMMAND 15196 root      19   0 2323m 1.8g 2780 S    0 24.6   0:00.68
> 4 processname
>
> free -m shows:
>              total       used       free     shared    buffers    
> cached Mem:          7611       7356        254          0         66
>       5023 -/+ buffers/cache:       2266       5344
> Swap:         4094          0       4094
>
> If I were allocating memory and not deallocating it, ie., if it were
> a memory leak, it seems CurrHeapUsed would be growing, right?
>
> It seems I should be concerned... what's the best direction to go
> from here? HEAPTRC?
>
> -SG

I had the same problem. You could try to enable "BESTMATCH" in the heap 
manager by either compiling the rtl with "-dBESTMATCH" or changing
"{ define BESTMATCH}" to "{$define BESTMATCH}" in rtl/inc/heap.inc and 
see if the situation improves. For me, it improved but didn't solve the 
problem completely. I had to create my own mem pool for some frequently 
allocated/freed structures.

IIRC this was caused by usage pattern like this:
* free a huge chunk (a) of mem -> chunk is returned to heap manager
* allocate small chunk -> this results in heap manager splitting chunk 
(a) to return the small piece (b) 
* now allocating again a huge chunk (same size as (a)) results in heap 
manager requesting a new chunk of mem from OS because the remainder of 
(a)-(b) is not sufficient.
.. well, in short: memory fragmentation
With BESTMATCH enabled, the heap manager tries harder to find a small 
free block for (b) before splitting (a) ..

However, I am not completely sure if this is the same problem .. it 
could also be caused by the reworked heap manager, which now handles 
mem allocation per thread (if that's allready in 2.2.2 ??) ..

regards
 Burkhard