[fpc-devel] Parallel processing in the compiler

[Hans-Peter Diettrich]

Florian Klaempfl schrieb:

>> I have no real idea, how in such a model the initialization of the
>> threadvars has to be implemented. That's why I try to assign all
>> state-related variables to a definite object, whose reference can be
>> easily copied (or moved?) into any created thread.
> 
> In case of current_filepos, the initialization could be done by the
> specific worker thread class doing e.g. code generation.

I see the main problem in the identification of all the global 
variables, that have to become threadvars. Or do you know of an way, to 
make the compiler identify variables that have to become threadvars?


>> Then we also have to draw a border, between possible parallel and still
>> required sequential processing. Sequential processing must be used for
>> all output, be binary or log files. Also encountered errors must be
>> reported from the originating thread to all related (main and other)
>> threads, when they require to stop and shut down compilation in an
>> orderly manner.
> 
> Yes, this won't get easy, but we are far away from something like this.

IMO it won't take more time than any of my other experiments, to set 
apart some small part of the compiler for use in threads. The required 
threadvars have to be identified in that attempt, too, but possibly in a 
smaller scope.


>> IMO the most important decision is about the general direction of the
>> refactoring. Do we want more OO (encapsulation), more codegen
>> separation, or what else. IMO encapsulation is the most useful first
>> step, towards any other goal. 
> 
> I think we agreed on this already. And I also agree with you, that the
> front end needs some redesign and it won't be easy (it took me almost 2
> years to find the current back end design btw. ...)

It was one of my first experiments, to create a new target with no code 
generation. Unfortunately I forgot to backup the according 
documentation, so that only the code is still available.

Another more recent experiment was to convert the back-ends into 
classes. It turned out that much target-specific code resides in the 
general compiler units, whose separation into virtual back-end methods 
would involve much work. In another approach the $IFDEFs could be 
replaced by ordinary IFs, what would both simplify the redesign and 
preserve the current compiler speed.


>> The current compiler "structure" is
>> dictated by purely *formal* aspects (unit dependencies), and does not
>> reflect the *logical* dependencies between objects, variables,
>> procedures etc. 
> 
> The problem is (very simplified): in a compiler everything depends on
> everything, so getting some structure is only possible with some
> formalism, formalism is the last resort when logic does not work
> anymore. Formalism is also something absolute while logic might be
> rather subjective.

I dare to disagree. It's not so hard to encapsulate everything in 
dedicated (mostly existing) classes. Only the requirement, that 
*external* tools should be able to use part of the compiler, without 
importing too many dependencies, results in the implementation of 
otherwise unneeded hacks and workarounds. In a radical OO redesign the 
number of involved units (and global variables) could be reduced 
dramatically, so that finally the compiler could be decomposed into a 
small number of building blocks (packages...), with tiny interfaces to 
each other.


>> This lack of logical structure, next lack of up-to-date
>> documentation, 
> 
> Nobody volounteered yet to continue Carl's work of documenting the compiler.

Where can this work be found? Do you mean the 1.x PDF document, that is 
not helpful with newer versions?

As mentioned above, I've started an documentation attempt some time ago. 
But after many arising questions remained unanswered, I had to figure 
out everything myself. And after I found out that the current compiler 
implementation is based on formal requirements, not on logical ones, I 
see not much use in providing documentation for that unsystematic state. 
IOW the current implementation deserves more badly a redesign, in favor 
of documentation. IMO.

DoDi