[fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

[Sven Barth]

Am 14.10.2010 04:15, schrieb Andrew Brunner:
> On Wed, Oct 13, 2010 at 3:24 PM, Michael Van Canneyt
> <michael at freepascal.org>  wrote:
>
>>> Is it possible I have the pthread library in some sort of debug mode
>>> that is slowing down the process of thread creation?
>>
>> I seriously doubt it.
>> What you could do to test, is write your program using direct Pthread calls.
>> This way we can ascertain whether it is the FPC or Pthread code which is the
>> bottleneck.
>
> Right.  I'm going to do more reading on the POSIX threading system.  I
> did get to trace into the threading unit under linux and the first
> thing I noticed was that a mutex was used to suspend and create the
> thread instead of using the ThreadManager.Suspend and Resume features.
>   My local copy has removed the semaphore and I instantly noticed a
> speed increase in thread creation due to the lack of the extra
> semaphore per thread.
>
> Based on what I see as far as performance goes, the current version of
> threading under Unix takes 2 semaphores per thread. One by use in the
> threads.inc file and at least one by the pthreads.so !
>

It could be that "RTL initialization" of the thread slows down as well 
(just a possibility). You might want to disable the calls to 
"InitThread" and "DoneThread" in "ThreadMain" inside in 
rtl/unix/cthreads.pp. But be careful: now you must not use Pascal's I/O, 
Heap and Exceptions inside your thread function, cause they aren't 
initialized now (use direct syscalls like fpwrite and allocate the 
memory manually).

>> If need be, I can dig up my texts for the Kylix book, it describes how to do
>> this in Object Pascal. I suspect the code would still be pretty much the
>> same.
>
> I would say let's try to obtain source to pthreads or something.  I'd
> bet we can just do a straight shoot into something from there.  If
> it's open source i'd bet we can bother them perhaps for a newer
> version more high-scale friendly.
>
>> Yes, write an object pascal version of it, which accesses the kernel
>> directly and bypasses the C library.
>
> That's exactly what I'm thinking.  There are only like 36 methods to
> implement.  Depending on how hard will be to hook into the kernel...
> The only thing is we're going not going to have diversity as found in
> the pthreads.so.  I'd bet they have tons of code for Darwin, Debian,
> Redhat, etc.  I guess it's unknown at this point but well worth the
> time to explore.

The problem with an own version of pthreads is that those threads will 
be limited to Pascal code only, cause other (C based) libraries will 
still use pthreads.

An interesting lecture on this topic is in the wiki of Wine, cause they 
had to implement their own threading implementation as well (the old 
pthreads library wasn't capable enough for the needs of the WinAPI). 
eYou can find the article here:
http://www.winehq.org/docs/winedev-guide/threading
(the interesting paragraph is "POSIX threading vs. kernel threading").

Regards,
Sven