[fpc-devel] Compiler trheads

[Florian Klaempfl]

Am 12.08.2010 08:12, schrieb Hans-Peter Diettrich:
> Marco van de Voort schrieb:
>> In our previous episode, Hans-Peter Diettrich said:
>>> How would you count an level, when multiple concurrent parser threads
>>> create further threads concurrently?
>>
>> Hmm, I think a producer/consumer system with a queue for modules that
>> can be
>> compiled and N worker threads (specifiable by user)
>>  
>> So on start threads are created, and no new threads are created after?
> 
> That's not a solution :-(
> 
> *When* a parser becomes a thread, it can proceed only to the first
> "uses" clause - then it has to create further threads, one for each used
> unit, and has to wait for their completion. I.e. most threads are
> waiting for other threads to complete[1].
> 
> That behaviour makes it questionable, whether parsers should become
> threads at all. Furthermore it suggests (to me) a dynamic thread
> priorization, based on the number of other threads waiting for their
> completion[2]. At least we have to distinguish between processing the
> interface part (or loading the declarations from the ppu file), until
> which point all dependent threads (using that unit) are blocked. Once
> the interface information has become available, the dependent threads
> can continue parsing and generating object code (.o and .ppu files).
> 
> Brainstorming: What other models come into mind, for using threads in
> the compiler?

The "final" plan is to use a task queue with dependencies and all needed
information. Such tasks may be: parse module X, generated code for
module X, load module Y, search for module Y, assemble module X etc.
Each task has a list of dependencies which much be done first before the
task can be executed.

So not a task per unit or whatever is started but a couple of worker
threads depending on the number of cores available, given command line
switches etc. These worker threads look for task in the queue with all
dependencies solved and when such a task is found, it is executed. The
more cores we get during the next years, the finer we could split the tasks.