[fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

[Vinzent Höfler]

On Fri, 15 Oct 2010 20:15:51 +0200, Andrew Brunner
<andrew.t.brunner at gmail.com> wrote:

> On Fri, Oct 15, 2010 at 12:01 PM, Vinzent Höfler
> <JeLlyFish.software at gmx.net> wrote:
>
>>> Exactly.  It was poor implementation.  You should have had a global
>>> barrier onExecute.  That would unlock the thread after everything you
>>> needed was readable.
>>
>> If it's global, it would unlock *any* thread at that time.
>
> Yes, belonging to a system with of collection of said threads where
> "managing" system oversees any worker threads.

Adding more administration won't solve any issues.

>> Sure, it's not the optimum design. But using one global barrier each
>> thread could block each other on creation, potentially introducing
>> excessive context switches. Which is not optimal, neither.
>
> Ok.  Say a project manager is looking to give FPC a try for a highly
> scale system that requires managing
> 1 million concurrent operations across say 30,000 threads.  Say it's a
> 10 core system with 32Gigs ram.  The first thing they are going to do
> is compare a general case with TThreads.  As is - it's slow as crap.

Yes.

> They aren't going to want to re-implement the threading system.  They
> aren't going to take FPC seriously as a viable compiler for parallel
> computing.

They don't need to. Just throw away the abstraction and call BeginThread.
According to you this will be magnitudes faster, because it's more or
less a direct call to the underlying pthreads subsystem.

>> Maybe it's less expensive, yet this blocking is actually required.
>
> It is not required if threading is implemented that ALREADY has a
> barrier in this case see pthreads source. It does.

You CAN NOT use this barrier whatever they may do in the pthreads
implementation you're referring to. It may even be solved different
on another platform.

Simple rule: If there's no API for it, you can't do it.

>> IMO, the problem is neither FPC nor pthreads,
>> the problem is the required semantics of TThread.
>
> The problem is not with TThread.  The problem is with Suspend and
> Resume features.

Which are features of TThread.

>> The faster alternative is bound to fail with "FreeOnTerminate := True".
>
> How did you draw this conclusion?

Simple. I wrote the bug report.

>> Perhaps - instead of looking into the pthreads library - you should
>> better take a look at the TThreads implementation and try to understand
>> *why* all the code there actually *is* there.
>
> I'm going to look at severe linear inefficiencies.  That is my nature.  
> :-)

But by looking at *some* specific implemention of pthreads you're starting
at the wrong end, I'd say.

>> Yeah, right. Creating the semaphore to wait in the execute method is
>> bound to be faster than creating it before thread creation. Sure.
>
> LOL.  Again, how in the world did you make the assumption, especially
> given all the code I put out there regarding a manager overseeing all
> the threads, that I would create a semaphore inside a worker thread's
> execute body?

You wanted to suspend the thread?

>  Did you read the uThreads.pas unit for this discussion?

Where can I find it? I only read your changes to the RTL and you seem
to assume that neither TThread.Create nor TThread.Destroy can run
concurrently with ThreadFunc (i.e. TThread.Execute). You do not seem
to understand that the variables accessed here *may* change during
execution of any of those functions.

IOW: Your "optimization" is wrong in the sense that it will not work
reliably.

>> Yes. But if you prefer crashing the application in favor of speed just
>> use C. :->
>
> I would most likely prefer objective-c.

Does that lead to crashes that often? ;)

>>> IMO, cThreads has a lot of RFI. (Room for improvement).
>>
>> No. TThread has. Last time I looked into CThreads it was a more or
>> less thin binding to the pthreads API.
>
> TThread has RFI.  But are you then suggesting you want to improve w/o
> using libc !?!

I suggest to remove the bottlenecks causing the slowdown at the
point where they are created. Alas, I was long enough involved with
that stuff that I came to the conclusion that the original Delphi-API
is seriously flawed and can not be implemented both efficiently *and*
correctly. So, I'd rather see a FPThread class than hacking some
pthreads libraries.

> IMO, using whatever we can would be great.
>
>>>>> There is no need to block the thread from execution.
>>>> Actually there currently is. Otherwise you ask for trouble.
>
> We're going to disagree here.  It's just that simple.

You agreement is not necessary to see that synchronisation is
required in some way. But understanding all possible impacts
of call sequences /is/.

The problem is that the code does not do what it is supposed to do
if the TThread.Destroy is called while TThread.Create is still
running. And that *can* and *will* happen. It happened to me, so
it will to others.

One reason for that is that the Destroy method is designed to be
called from either the executing thread (FreeOnTerminate = True) or
some other external thread explicitely destroying it
(FreeOnTerminate = False). To prevent race conditions you *will*
actually *require* synchronisation, whether you agree or not.

If you dig deeper, you will see that the whole concept is flawed,
but that's not the FPC teams fault, it's the bloody compatibility
with Delphi.

> If you found
> yourself in the ranks with developers saying "What would anyone need
> 30,000 threads" camp, please ignore this entire discussion.  I beg
> this with authority :-)

Why would anyone need 30'000 threads? That would mean that 2'767 cores
are idling. ;) (You need one CPU left to kill them all.)

>> Try calling pthread_destroy() with an uninitialized handle. Maybe
>> that changes your way of thinking.
>
> No, I pretty much question EVERYTHING.

Then rewrite everything and don't get lost in the details of an
inefficient and flawed implementation (your opinion). I'd agree
with the inefficient only.

>> *What* barrier?
>
> Seriously?  There are four possibly five (depending on prospective)
> barriers mentioned in the course of this entire discussion.  List them
> and I will be able to oblige.

I mean that magic barrier in the pthreads library which you seem to
want to use for whatever reason.


Vinzent.