[fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

[Michael Van Canneyt]

On Wed, 13 Oct 2010, Andrew Brunner wrote:

> On Wed, Oct 13, 2010 at 3:24 PM, Michael Van Canneyt
> <michael at freepascal.org> wrote:
>
>>> Is it possible I have the pthread library in some sort of debug mode
>>> that is slowing down the process of thread creation?
>>
>> I seriously doubt it.
>> What you could do to test, is write your program using direct Pthread calls.
>> This way we can ascertain whether it is the FPC or Pthread code which is the
>> bottleneck.
>
> Right.  I'm going to do more reading on the POSIX threading system.  I
> did get to trace into the threading unit under linux and the first
> thing I noticed was that a mutex was used to suspend and create the
> thread instead of using the ThreadManager.Suspend and Resume features.
> My local copy has removed the semaphore and I instantly noticed a
> speed increase in thread creation due to the lack of the extra
> semaphore per thread.
>
> Based on what I see as far as performance goes, the current version of
> threading under Unix takes 2 semaphores per thread. One by use in the
> threads.inc file and at least one by the pthreads.so !
>
>> If need be, I can dig up my texts for the Kylix book, it describes how to do
>> this in Object Pascal. I suspect the code would still be pretty much the
>> same.
>
> I would say let's try to obtain source to pthreads or something.  I'd
> bet we can just do a straight shoot into something from there.  If
> it's open source i'd bet we can bother them perhaps for a newer
> version more high-scale friendly.
>
>> Yes, write an object pascal version of it, which accesses the kernel
>> directly and bypasses the C library.
>
> That's exactly what I'm thinking.  There are only like 36 methods to
> implement.  Depending on how hard will be to hook into the kernel...

The kernel thing is easy, just a vfork() call, if I'm correct.
It's all the rest that is difficult: synchronization, mutex, 
semaphores :)

> The only thing is we're going not going to have diversity as found in
> the pthreads.so.  I'd bet they have tons of code for Darwin, Debian,
> Redhat, etc.  I guess it's unknown at this point but well worth the
> time to explore.

I don't think the various Linux distros work different in this respect.
They all use pthreads now, and they all use the same kernel interface.

>> It has been on many people's wish list for ages.
>
> I presently can create 1,500 threads in 2 min 27seconds. That's 2 1/2
> threads per second... 25min! Pathetic...

Remains the question why anyone would want to create 1500 threads ?
Any number of threads above the number of CPUs is no longer efficient
anyway (give or take some corner cases with lots of I/O).

Michael.