[fpc-pascal] TFPList.Expand was: Maximum of memory which can be used by single program in 32 bit Windows

[Michael Van Canneyt]

On Fri, 20 May 2016, Marco van de Voort wrote:

> In our previous episode, LacaK said:
>> As new nodes are appended TFPList.Expand is called
>> And there is:
>>    if FCapacity > 127 then Inc(IncSize, FCapacity shr 2);
>>
>> So if I have in list 1 000 000 items, then at once list is expanded by
>> 250 000, which causes in one step out of memory
>> So my question is: can I somehow control increment count ? I think that
>> ATM no.
>> So second question is can TFPList.Expand be modified, that for large
>> FCapacity will be used smaller increment ? :-)
>
> TMemoryStream also has this behaviour, and indeed I override it in my own
> applications to let the growth taper off after 128MB.  If you have a lot of
> midsized memory streams, it otherwise results in having 25%/2 = 12.5%
> unused memory lieing around.
>
> But of course all datastructures based on a single array (and thus inviting
> very large block allocations) are fundamentally flawed because of it in the
> first place.

Which is why I recommended going the SAX way and store data manually.

There is no single good and simple catch-all mechanism.

Michael.