[fpc-other] Re: A question or two regarding the FPC [heap size growing]

[Giuliano Colla]

This can be an answer to mission critical reliability. A small bug may 
affect this application, while another could run forever.

Giuliano


-------- Messaggio originale  --------
Oggetto: Re: [fpc-pascal] heap size growing
Data: Mon, 11 May 2009 12:00:52 +0200
Da: Luca Olivetti <luca at ventoso.org>
Rispondi-A: FPC-Pascal users discussions <fpc-pascal at lists.freepascal.org>
A: FPC-Pascal users discussions <fpc-pascal at lists.freepascal.org>
Referenze: 
<4f7c444e0901131950o6403773y9d3ab8897f0b9cb9 at mail.gmail.com> 
<200901141302.01344.fpc at bcsoft.de> 
<D87EE373-C301-43FA-8782-A0A3AFAD112C at elis.ugent.be>

En/na Jonas Maebe ha escrit:

[sorry to revive an old thread, but I'm having a similar problem and I
prefer to follow-up than to start a new thread. Since it's old, I'm
quoting it without trimming]

> 
> On 14 Jan 2009, at 13:02, Burkhard Carstens wrote:
> 
>> Am Mittwoch, 14. Januar 2009 04:50 schrieb Seth Grover:
>>>
>> I had the same problem. You could try to enable "BESTMATCH" in the heap
>> manager by either compiling the rtl with "-dBESTMATCH" or changing
>> "{ define BESTMATCH}" to "{$define BESTMATCH}" in rtl/inc/heap.inc and
>> see if the situation improves. For me, it improved but didn't solve the
>> problem completely. I had to create my own mem pool for some frequently
>> allocated/freed structures.
> 
> In the general/average case, "best fit" produces slightly worse results 
> than "first fit" (which is the default). The reason is that's you're 
> more likely to end up with a bunch of unusably small blocks over time 
> than with first fit (there are of course usage patterns in which this 
> does not hold).
> 
>> IIRC this was caused by usage pattern like this:
>> * free a huge chunk (a) of mem -> chunk is returned to heap manager
>> * allocate small chunk -> this results in heap manager splitting chunk
>> (a) to return the small piece (b)
>> * now allocating again a huge chunk (same size as (a)) results in heap
>> manager requesting a new chunk of mem from OS because the remainder of
>> (a)-(b) is not sufficient.
>> .. well, in short: memory fragmentation
>> With BESTMATCH enabled, the heap manager tries harder to find a small
>> free block for (b) before splitting (a) ..
>>
>> However, I am not completely sure if this is the same problem ..
> 
> It probably is.
> 
>> it
>> could also be caused by the reworked heap manager, which now handles
>> mem allocation per thread (if that's allready in 2.2.2 ??) ..
> 
> No, it is not. It will only be released in 2.4.0. And it will not solve 
> the problem, unless the differently-sized memory blocks are only 
> allocated in different threads.
> 
> You can also try using your platform's libc memory manager to see 
> whether it deals better with this usage pattern (add "uses cmem" to the 
> uses clause of your main program). Or, as mentioned above, use your own 
> memory pool (like the default memory manager already does for small 
> allocations).

I'm having a similar problem with the heap: I'm capturing an mjpeg
stream from an ip camera (well, actually I'm capturing 50 streams from
50 cameras) using TJpegImage (from the LCL).
The problem is that memory usage grows, slowly but steadily, and after a
while (1 day, 1 week, it depends) the program starts failing (with "No
buffer space available").
Seeing this thread, I used GetFPCHeapStatus and, effectively,
CurrHeapSize is growing but CurrHeapUsed isn't, so apparently
TJpegImage.LoadFromStream (that's what I'm using) causes heap fragmentation.
FYI, using cmem (this program is running under windows xp) overall
memory usage stays constant (though I haven't tried it yet in the field).
Is there any undesirable side effect due to the use of cmem?

Bye
-- 
Luca

_______________________________________________
fpc-pascal maillist  -  fpc-pascal at lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-pascal


-- 
Giuliano Colla

Whenever people agree with me, I always feel I must be wrong (O. Wilde)