[fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

[Andrew Brunner]

On Wed, Oct 13, 2010 at 3:24 PM, Michael Van Canneyt
<michael at freepascal.org> wrote:

>> Is it possible I have the pthread library in some sort of debug mode
>> that is slowing down the process of thread creation?
>
> I seriously doubt it.
> What you could do to test, is write your program using direct Pthread calls.
> This way we can ascertain whether it is the FPC or Pthread code which is the
> bottleneck.

Right.  I'm going to do more reading on the POSIX threading system.  I
did get to trace into the threading unit under linux and the first
thing I noticed was that a mutex was used to suspend and create the
thread instead of using the ThreadManager.Suspend and Resume features.
 My local copy has removed the semaphore and I instantly noticed a
speed increase in thread creation due to the lack of the extra
semaphore per thread.

Based on what I see as far as performance goes, the current version of
threading under Unix takes 2 semaphores per thread. One by use in the
threads.inc file and at least one by the pthreads.so !

> If need be, I can dig up my texts for the Kylix book, it describes how to do
> this in Object Pascal. I suspect the code would still be pretty much the
> same.

I would say let's try to obtain source to pthreads or something.  I'd
bet we can just do a straight shoot into something from there.  If
it's open source i'd bet we can bother them perhaps for a newer
version more high-scale friendly.

> Yes, write an object pascal version of it, which accesses the kernel
> directly and bypasses the C library.

That's exactly what I'm thinking.  There are only like 36 methods to
implement.  Depending on how hard will be to hook into the kernel...
The only thing is we're going not going to have diversity as found in
the pthreads.so.  I'd bet they have tons of code for Darwin, Debian,
Redhat, etc.  I guess it's unknown at this point but well worth the
time to explore.

> It has been on many people's wish list for ages.

I presently can create 1,500 threads in 2 min 27seconds. That's 2 1/2
threads per second... 25min! Pathetic...