[fpc-pascal] TFPList.Expand was: Maximum of memory which can be used by single program in 32 bit Windows

[Michael Van Canneyt]

On Fri, 20 May 2016, LacaK wrote:

>
>> 
>>> Btw is there any limit for number of memory block, which can be requested 
>>> ?
>>> As far as my XML file has milions of nodes, may be that for each node is
>>> created instance of TDOMNode class (or TDOMElement I do not know), which
>>> in turn can cause that there is large amount of relative small memory
>>> blocks ...
>> 
>> There is no limit, other than the fact that every block has some overhead, 
>> and hence you can allocate less memory in total if you allocate tons of 
>> small blocks instead of a few large ones. It's just 16 or 32 bytes per 
>> block though, so even with millions of blocks you won't lose more than a 
>> few tens of megabytes.
>> 
>
> I think that, I found root of my problem.
> As far as I have million nodes under few nodes, there is maintained 
> TDOMNodeList, which stores nodes in TFPList
> As new nodes are appended TFPList.Expand is called
> And there is:
>  if FCapacity > 127 then Inc(IncSize, FCapacity shr 2);
>
> So if I have in list 1 000 000 items, then at once list is expanded by 250 
> 000, which causes in one step out of memory
> So my question is: can I somehow control increment count ? I think that ATM 
> no.
> So second question is can TFPList.Expand be modified, that for large 
> FCapacity will be used smaller increment ? :-)

This is of course possible, but I think that you would be served better by
either
- going to 64-bit. Today you have million nodes, tomorrow maybe 2, and then
   it will be a problem just as well.
- Using SAX and not store the whole DOM in memory, and at the same time
   use your own memory allocation routines for your structures.

Michael.