[fpc-devel] External assemblers (also modular discussion about free pascal compiler)

[Sven Barth]

On 06.04.2011 22:46, Skybuck Flying wrote:
> Hello,
>
> Perhaps some last silly questions about "external assemblers".
>
> First I'd like to say that I saw some tutorial where the compiler itself
> produced some form of assembly output in text... probably real assembly...
>
> So that's kinda interesting to output text this makes it flexible for
> other tools to then compile that...
>
> It reduces complexity of the compiler because it doesn't actually need
> to fully assembly the assembly... this can be left to other tools.
>
>
> So far I understand free pascal has "internal assemblers" (probably
> mostly for i386/x86 target, does it have others as well ?)
>

Only i386 and x86_64 as far as I'm aware of.

>
> But there is also this "external assembler" thingy.... perhaps it's
> being used to implement internal assemblers as well...
>
>
> But my question is the following:
>
> What exactly is being "fed" towards the "external assembler" via the
> classes/api ?
>

Files containing assembler code for the selected target (because a 
(external) assembler just takes that file as if someone would have 
written it by hand and assembles that to the correct opcodes and 
operands (simply spoken).

>
> (I probably should take a closer look... )
>
> But from what I can remember I saw some kind of node structure... (maybe
> fp list/node and stuff like that ?)
>

This is bascially the "abstract syntax tree" and only used internally.

>
> To me it seemed like some kind of "free pascal classes/data structures"
> which might not be commonly supported by other languages like
> C/C++/Delphi/Etc...
>
>
> So just in case I am wrong I am asking the following question:
>
>
> Is there perhaps some kind of "standardized intermediate form (data
> structure like)" which all compilers could choose to output ? (Which I
> am not aware off ?)
>

No, there isn't.

>
> (Or is the external assembler thingy indeed something conceived by the
> pascal developers themselfes ;))
>

Every external assembler "understands" the assembly language that it can 
assemble code for. An i386 assembler understands i386 assembler 
instructions while it does not understand ARM ones and the other way 
round. So the compiler outputs the code that the assembler expects.

>
> Also one last question: Is there actually any other external tool which
> is actually invoked/call via this mechanism ?
>

No by the exact same mechanism, but there is also the resource compiler 
(fpcres) and the linker (seldom used on Windows targets as those have a 
internal linker (netware now as well as it seems...)).

> (If the mechanism was ment to function via text then I can imagine that
> ofcourse... but then the question would be: what kind of text would be
> fed ? (some early form of assembly which needs to be fixed up ? )
>

No. The optimized assembler code that should be assembled by the 
assembler into the final object code.

>
> Sending assembly towards an assembler seems kind weird... because first
> the compiler seems to use these nodes/data structures which are then
> apperently send towards some kind of cpu which seems to use assembly
> like data structures as well (but I could be wrong) which would then be
> turned back into text ????
>

Why is it weird when the external tool (which is not developed by the 
FPC developers but by others, e.g. the GNU devs) does only understand 
that text?

> (Why then not compile directly to binary ??? Perhaps because the
> binaries involve operating system specific structures and the CPU can
> actually be used for different operating systems ? (So I think that's
> probably the answer here...) The compiler compiles towards a certain CPU
> and makes sure the assembly is suited for the CPU... but it doesn't
> necessarily want to know about the operating system and thus sends it
> off to an operating system specific assembler ?)
>

The operating system isn't that important. But the compiler does not 
know by itself that "mov" is that or that opcode. So it needs an 
assembler to produce the final object code. And sometimes such an 
assembler is available in the compiler itself as well, so one doesn't 
need to rely on an external tool...

> (But then again this conflicts a bit with the RTL which does seem a bit
> operating system aware... but maybe that is not related to this and the
> RTL is ment for input towards the compiler and perhaps output as welll
> but in a more file like manner...)
>
> A guess what I described above is the roll of the linker.... The linker
> links the assemblies and adds some operating system specific binary
> headers/structures for towards/for the final binary.
>

No. The linker just puts the generated object files together and adds 
the needed headers for the final binary (PE or ELF header) so that the 
operating system can load and run the executable or binary. All the 
operating system specific functions are already resolved and added by 
FPC itself.

>
> Does free pascal actually have an internal linker as well ?
>
>

Yes. It contains a PE linker, an (unfinished?, experimental?) ELF 
linker, an unfinished linker for OS/2 and since some days also a linker 
for Netware libraries.

> Perhaps all these different aspects of a compiler are confusing things a
> little bit for me/noobies.... but perhaps also for the developers itself...
>

Not after you have wrapped your head around it :D

> Therefore I wonder if free pascal could benefit from a more modular
> approach where everything is nicely split up into modules... which
> somehow communicate with each other...
>
> This could be executables (though would get a bit difficult via files)
> or (dll's but that's windows specific) or (sockets... network
> communications seem odd for this...) and finally simply compiled units
> which would be placed in seperated module folders or so. (with the
> necessary headers... could also simply include full source as well).
>

That is all to slow... When the internal linker for Windows was added 
(and thus the need for the external one ceist to exist) there was a huge 
speed increase (on Linux that isn't as problematic as the linker there 
is already fast). So I personally prefer the "everything in one binary 
approach". Also you "just" need the compiler binary if everything is put 
in there although FPC might indeed profit once dynamic packages are 
supported (which are still in the far away future...)

>
> Currently I wonder how "modular" free pascal is... since I am not
> acquinted with the code (yet)... from the looks of it it seems to use
> classes here and there... so at least it's "class-modular".... but
> classes can also starts to "mingle" with each other a bit... and might
> not form a "clean cut". Probably tempting for developers to mingle stuff
> up a bit... None-the-less that's probably easily avoidable or fixeable...
>
>
> The problem remains that all source code is in pretty much one huge
> folder called: "compiler" with some "sub folders" for platforms.
>

At the end everything is in "one huge folder that contains some 
subfolders"...

> At least the platforms are seperated from it... but this seems also a
> bit out of necessity for the search paths to not produce
> naming-space-conflicts and such.
>
> The rest of the compiler code seems to be in one huge folder...
>
> Some of the unit names are pretty short and not very descriptive...
> fortunately most of the unit files have a little comment in the header
> explaining what each file is.
>
>
> Perhaps it would be interesting to try and split up the compiler folder
> into subfolders where each subfolder would represent a "module".
>
> For example tokenizer/lexer, parser, assemblers and what else you can
> think of which would/should be modular.
>

Give it a try and see whether the devs will apply your changes. When no 
one tries to do this we won't know ;)

> The platforms/cpu's can then be moved to their own modular sub folder
> which could be called:
>
> "platforms" or perhaps "cpu's" or perhaps "os" or perhaps "targets"
> which you think is best.
>
>
> With such a more modular approach via subfolders it could then be easier
> to spot where unwanted depedency might exist within the module itself...
>
> So suppose that a certain module is to be "self standing" and "self
> compileable" and this would immediatly show up if this was not to work
> because of
> search paths pointing towards unwanted folders.
>
> At first it might seem the compiler is to much linked towards all
> modules because ofcourse it goes from phase to phase... but that's where
> hooking into modules could come into play.
>
>
> It could work like this: each module gets it's own input interface
> (input from the higher phase/modules) and it's own output interface
> (towards the lower modules)
>
> Perhaps this could also be in both ways... However each module simply
> copies these interfaces so that they can be self standing...
>
> Only the final compiler main program/executable needs to link the
> modules up towards each other... perhaps test programs for the modules
> too. (These test programs should then be in seperate folders as well
> could be higher/sub folders to the module itself)
>
> One final module which seems to be in play in general theory is the
> "symbol table" this could also have it's own interface which would be
> used by all modules as well.
>
> So all modules would need to be hook-up to this symbol table as well.
>
>
> (Think of "hooking-up" as delphi's event properties/method
> pointers/function pointers stuff like that...I am pretty sure you know
> that already ! ;) =D)

I you would indeed try that, then do it in small steps (e.g. first 
moving units that form a "module" into a sub directory). Otherwise the 
chances for your changes to be applied will be low.

Good luck.

Regards,
Sven