[fpc-pascal] Re: Multi-threaded project with few locks (no Thread.waitfor). Memory consumption keeps increasing on Ubuntu 10.10 x64

[Sven Barth]

On 16.10.2010 18:30, Marco van de Voort wrote:
> In our previous episode, Sven Barth said:
>
>>>>> First we would have to get rid of "this solution in search of a problem"
>>>>> attitude.
>>>>
>>>> What do you mean by the "this solution in search of a problem" attitude?
>>>> I can't follow you currently... :(
>>>
>>> People seem to first want to bash pthreads, and discuss own threadmanagers
>>> before actually defining what the problem of the current solution is, and
>>> why it is absolutely needed to create something new.
>>>
>>
>> One of the problem seems to be that we need to link an external library.
>> While FPC allows us to do so much things with Pascal code only, we need
>> to add a dependency when we want to use threading.
>
> Yes. So all your programs don't use any libraries except libc because
> pthreads? If so, what kind of programs do you make?
>

I didn't say that I personally have this problem; it's just the one 
problem that I'm really aware of. :)

But I even have an example where I don't even use libc, but use threads: 
a downloader (using lnet). Ok... I've recently switched from lnet and 
threading to calling wget with TProcess, because it was to unreliable, 
but I HAD such an application where I'd liked to not have any dependency. ^^

Another application in need of native threading might come in some time, 
because I have an arm-linux system with uclibc where I not yet know how 
good and reliable its NPTL implementation is.

>> Note: On Windows this feels different, as here kernel32.dll is THE
>> interface to the OS, while libc/pthread aren't on Unix systems (there
>> the OS interface consists of the syscalls to the kernel).
>
> I don't see it that way. I'm perfectly happy to take libc/pthread as system
> interface. If they actually went to the trouble to make it a decent API,
> somewhat stable in time, and decoupled from e.g. compiler version changes.
>
> THAT is the *nix problem, not the name of the library, be it user32 or libc
> or libroot.
>

That's somehow that what I meant, but with different words. libc/pthread 
aren't yet stable enough in my opinion to be seen as "OS interface". I'd 
have no problem with them as well if they'd remain stable...

> Still, while I agree with the problem, I don't think a native threadlib will
> change anything in this picture. The amount of pain won't lessen. One only
> adds yet another factor (libc threads vs pascal threads) to the chaos.
>

Yes, that's the biggest problem... :(

> The only possible thing is wait till Linux and the Unix world at large comes
> to its senses.
>

Which might take some time...

>>> Like, euh, why simply having a thread pool to workaround slow creation is
>>> not a better solution (and as done for decades by vast numbers of
>>> programmers)
>>
>> Maybe we should provide a generic thread pool system in the FCL? Then
>> some FPC users might realize that they don't really need a faster thread
>> creation.
>
> I think all sane contributions are welcomed. One can never fully exclude
> that somebody might actually come up with both a good Pascal only threading
> library and a good motivation to actually use it. ( I can imagine some
> limited uses to really quick initialize CGIs or so)
>
> But talking about making room in the FPC implementation before these two
> conditions are satisfied, just because some people think there is a problem,
> and a solution for that problem.
>

Who's talking about making room?

I thought the recent discussion was about having an alternative thread 
class that doesn't have the TThread heritage. And also having the 
possibility to switch between native threads or pthreads (on platforms 
of which the kernel provides such threading facilities).
I don't see why you'd need to make room in FPC's implementation for 
them. They'd simply be alternatives. :)

> It will just lead to orphaned attempts and a drain on FPC resources.  Just
> look at the discussion about the current TThread implementation, which afaik
> stems from the 2003-2004 timeframe (when what is now the 2.x series was
> finalized in the 1.9 beta series).  And that is just library usage.  How
> long do you think it will take to get a native implementation up to snuff
> for real use, beyond being some minimalists pet project ? Even if I forget
> for the moment that it is DOA out of principle (since it won't
> play nice with external libs probably), it would take years.
>

Regarding orphaned attempts: I know what you're talking about. I always 
think that I've started something without finishing it when I think 
about my two FPC pet projects "cppclass" and "nativent". But I still 
have the hope that I'll continue working on them :)

> And all the while the forces concentrate on that, the current implementation
> is on ice, waiting for the glorious new code that will fix everything and
> will never come. Better invest in dressing up
> the current implementation, synchronization features (syncobjs, TMonitor,
> locked queues etc), stuff that is really going to be used.
>

I really, really must do more serious work with threads...

Regards,
Sven