[fpc-devel] z370 Cross Compilation, Pass 2 of ....

[Sven Barth]

Am 19.08.2013 13:00, schrieb Mr Paul Robinson:
> Based on information and belief, on Mon, 19 Aug 2013 10:37:43 +0200, 
> Sven Barth <pascaldragon at googlemail.com> wrote to the Free Pascal list:
>
> > I don't get what you're trying to say here.
>
> I mean I spent the better part of a year writing a cross-reference 
> tool for Free Pascal since the compiler doesn't include one as they 
> used to do (Mainframe and minicomputer Cobol, Fortran and Assembler 
> all have built-in cross-reference as part of the compiler/assembler), 
> then I had problems implementing a piece of it, so I realized that 
> I'll come back to it at some time later, and right now I want to go 
> back and try again with Free Pascal since I now know what I need to 
> tackle. When I wasn't sure how to ask for what I knew I needed, I set 
> this request aside to work on the cross-reference. I had to figure out 
> what was going on and what I had to do.
If you'd now only explain what a cross-reference tool is I might even 
understand what you're trying to tell me here...
> > Analyzing what a big-endian compiler generates won't help you there.
>
> Possibly, but I have to start somewhere.
>
> > You need to look into the compiler's code and learn how it
> > does generate code for a specific platform. That has nothing
> > to do with big endian or little endian.
>
> It does if the compiler generates object files directly (the way 
> Pascal 8000 did on the 370, UCSD did, and Delphi does). If the 
> compiler generates assembly language source files (the way OMSI Pascal 
> did on the PDP-11 and the National Bureau of Standards Pascal Compiler 
> did), then no, endianness probably won't matter. But I don't know from 
> direct examination which FPC does, it's a huge application and I have 
> no idea what all 100+ units and 264,000 lines of code do.
>
> I have to start somewhere and I have to take a guess. There are three 
> possibilities. (I don't care what FPC does when it scans the program; 
> that's all magic I probably don't have to worry about or deal with 
> right now.) (1) The compiler, when it generates code, generates an 
> intermediate output file processed by something else (Stanford Pascal, 
> P4, P5, Java Compilers). (2) The compiler, when it generates code, 
> generates an assembly language file (or possibly some other language) 
> for the target that is passed to the Assembler (OMSI Pascal for the 
> PDP-11). (3) The compiler, when it generates code, generates an actual 
> object file and/or an executable (Pascal 8000, Turbo Pascal, Microsoft 
> Pascal, Delphi).
>
> I know not which of these three Free Pascal does.
FPC uses 2 and 3 depending on the platform and OS. For example for i386 
and x86_64 we have an internal assembler, but for the other platforms we 
use an external assembler (mostly the GNU assembler, but sometimes also 
others are supported) [Note: the two x86 platforms also support external 
assemblers]. Also we internal linkers for some platforms. The first one 
was for PE based systems (Windows 32/64 Bit, Windows CE, Native NT) and 
some months ago we also gained a ELF linker which currently supports 
Linux, but still has a few restrictions (AFAIK). Otherwise we also rely 
on external tools (mostly the GNU linker) here. So as a first step you'd 
choose the approach of using an external assembler and linker, because 
simply calling a third party utility is easier than completely 
implementing an internal assembler and linker.
>
> > Platform independant stuff is normally located in
> > the ncg*.pas units and the platform specific stuff
> > is in the nXY*.pas unit of the corresponding CPU
> > directory where XY is a short form of the CPU, e.g.
> > n68kadd.pas for the m68k add/compare node. Best
> > is you look how other CPUs are implemented
> > (prefereably the simpler ones like > MIPS or m68k)
> > and copy that more or less for the new platform.
>
> My spell checker says you spelled "independent" and "preferably" worng. :)
And mine says that you spelled "wrong" wrong... can we continue now?
>
> Yeah, and I still have to figure out what compiler switches to set to 
> enable the existing compiler to create a cross-compiler for that 
> target that runs on Windows, and I'm asking for help on that point, 
> because I figure someone might be able to tell me in 10 words which 
> switches I need. The documentation is rather sparse on cross-compiling 
> and the code is, well, to say the least, not very forthcoming.
To create a cross compiler there are two possiblities (in both cases you 
need to have FPC 2.6.2 installed):

In a command window (cmd, PowerShell, whatever) go to the directory 
where the source code of FPC trunk is located and there execute:

cd compiler
make all OS_TARGET=linux CPU_TARGET=ppc

This will generate a compiler which can cross compile to PowerPC (Note: 
each compiler for a given platform (i386, x86_64, PowerPC, etc.) can 
generate code for each target that is supported there (e.g. an i386 
compiler can generate code for Linux, Win32, WinCE, Mac OS X, etc.))

Alternatively you can open the corresponding project file located inside 
the "compiler" directory in Lazarus (e.g. pp.lpi for the i386 compiler, 
ppcx64 for the x86_64 compiler, etc.) and just compile there.

To actually use the generated compiler you'll also need to compile the 
RTL. For this you should compile the utlity "dummyas" located in 
compiler/utils and copy the created executable to e.g. the main source 
directory and rename it to "as". Now in the directory "rtl" you should 
execute this:

make all OS_TARGET=linux CPU_TARGET=ppc BINUTILSPREFIX= 
CROSSBINDIR=c:\path\to\fpc\source 
FPC=c:\path\to\fpc\source\compiler\ppcppc.exe

Here "c:\path\to\fpc\source" is the path to the directory where your FPC 
trunk source is located. If you used Lazarus to compile the compiler the 
argument for FPC is different: 
"c:\path\to\fpc\source\compiler\powerpc\pp.exe"

Then you can compile programs using the following (also from the command 
line, assuming you are in the source main directory):

.\compiler\ppcppc.exe -n -Furtl\units\powerpc-linux -FEtestoutput 
.\tests\test\tgeneric1.pp

- assuming there exists a "testoutput" directory in the current 
directory which you'd need to create
- if you compiled the compiler using Lazarus the executable is called 
".\compiler\powerpc\pp.exe" instead.

>
> > Adding a new platform to FPC is not cheesecake
>
> I'm fully aware of this, I'm not stupid and this is not the first 
> compiler I've worked on, just the largest. I've written patches to six 
> different Pascal compilers and I've written two compilers from scratch 
> (not for Pascal), one was a Fortran IV to Visual Basic translator 
> written in 7,000 lines of VB5, the other was a compiler I needed for a 
> different purpose. I'm not someone who just fell off the turnip truck. 
> I know there's (a lot of) work involved.
>
> I have to take a guess on where to start. Given an existing 
> implementation, that I actually get it to work, I can then figure out 
> what is being done, and from that, then I can figure out what files 
> are being used and/or called, and what procedures/methods are invokes. 
> And that's what I'm asking for help. What do I do to get one of the 
> existing compilers that generates code for some other machine to run 
> on Windows and when fed itself, have it generate an output file of the 
> generated code of itself for that machine?
>
> Given that, I can then figure out what to look for.
No you can't. As you already said the compiler consists of 100+ units 
and around 262.000 lines of code plus the code of the RTL. You can't 
simply look at that output and transform that somehow for s370. You need 
to start small by having the compiler compiler a simple hello world 
program or maybe even an empty program (just in case: the parameters to 
get the generated assembler output is "-al" and maybe you'll also need 
to use "-s" so that the compiler does not try to call a non existant 
assembler/linker)
>
> > and you should know how the compiler's backend work. Just looking
> > at the output of a target won't help you!
>
> It will a little. Seeing what is generated means I can look in the 
> source code for where that is generated from, sort of the way the NSA, 
> if it thinks you're calling a terrorist, it can search the billions of 
> phone records it secretly seized to see who else you called!
>
> Given the generated output, I can (presumably) trace it back to 
> whatever program file generated it and thus figure out what files were 
> used. Going further, if necessary, I can insert procedure or unit 
> traces on those files and see what procedures were called in what 
> order and possibly by whom.
>
> Given that information, I can figure out what's going on. I'm just 
> having trouble figuring that out and I wanted any help someone could 
> point me in where to look.
It is easier to simple debug through the compiler. Open the pp.lpi in 
Lazarus, write a simple hello world program and set a breakpoint in 
compiler\pmodules.pas in the function proc_program at 
"main_procinfo.generate_code" (note: you should have no other 
procedures/functions in the example as a first step). You can then step 
through how the compiler converts the abstract syntax tree to abstract 
assembler output. Using the generated assembler to study the compiler's 
code generation WILL NOT GET YOU ANY WISER. I can guarantee you that, 
because with debugging it is already complicated enough the first few times.
>
> > First step is to get a cross compiler to s370/zSystem working so
> > that simple programs can be compiled which you can use to
> > implement the RTL. Only when enough of the RTL is implemented
> > and the code generator works good enough then you can try to
> > cross compile the compiler, but not earlier.
>
> Right. And I need to figure out what procedures in what files are 
> being invoked in order to change those so I take an existing compiler 
> which generates some other target than Windows, and while that 
> compiler is running under Windows, turn the switches and flags and 
> defines on, recompile that compiler so it generates code for the 
> machine it's targeting (MIPS, PPC, whatever) so that, even though it's 
> running on Windows, it creates code for the other target machine. 
> Then, taking that compiler that I know does work and generates PPC or 
> MIPS code or whatever target its for, change its flags so that while I 
> ask it to generate S370, it still
> generates PPC or MIPS code. Now, then, knowing that, I can then change 
> its code generator so that now, it will generate the output code for 
> S370 instead of MIPS or PPC.
>
> Now, I have a bootstrap compiler that will, on a Windows box, take a 
> program which is syntactically correct, and when it runs, it will 
> generate S370 code. Now, at that point I can recompile the run-time 
> libraries needed to implement the compiler. Take those generated files 
> over to a real or simulated 370, and (presumably) assemble them to 
> produce the run-time library. At that point, I can feed the compiler 
> into itself, produce its own assembler source, assemble it, put it on 
> the 370, link it against the previously compiled library, and if 
> that's right, I should then get a compiler that runs on the target 
> machine. Then, and at that point, I can then throw the compiler at 
> itself and it should generate the exact same output as it did on the 
> Windows box, and at that point the compiler now is no longer a 
> cross-compiler, it's a native compiler.
>
> As I said, I didn't just fall off the turnip truck, I do know what's 
> involved. But to do that, I first have to get a cross-compiler for 
> something running on Windows, and I'm trying to figure out what 
> switches and defines I need!
No, you are still thinking to simple. It will take much much time and 
running test cases again and again until you've reached the point that 
the code generator for s370 is stable enough that it can successfully 
compile the compiler's code. Just to name a few: you'll need to get 
parameter passing for functions correctly, you need to correctly 
retrieve addresses, you need to correctly handle compare operations, you 
need to correctly implement loops. The compiler is only at the end of a 
very long process which looks very roughly like this:
- copy an existing code generator (e.g. the aarch64 one as Florian 
mentioned) to a new directory s370, adjust the names and search in the 
remaining code (including .inc files) for "aarch64" and copy and adjust 
the corresponding ifdefs and add corresponding enum values
- test whether the compiler now works by adding a new ppcs370 project 
where you adjust the paths and the define (instead of -dAARCH64 it 
should be -dS370 then)
- start implementing the RTL platform specific functions inside rtl/s370 
based on an existing target (e.g. rtl/mips) again (e.g. setjump/longjump)
- best try to implement a s370-linux target at first so you don't need 
to implement a full RTL as the Linux RTL is already existant
- change the code generator step by step so that it generates correct 
code for s370 and test using simple programs maybe even using a dummy 
System unit
- try to compile and run testcases located in tests/test/cg which are 
especially for testing the code generator
- compile and run more testcases in tests
- only if these testcases run satisfactory you should try to compile the 
compiler itself
Please remember that this is only the rough process. It get's more 
complicated!
>
> >> So how would I get the constants and values added to
> >> include the s370 as a target for FPC?
> >
> > Search the compiler's source for e.g. m68k to see how other
> > platforms are implemented.
>
> Been there, done that. If you read my comments in the WIKI on how I 
> was implementing this, I got to the point where the flags were all 
> inserted, and they did nothing except neither did the compiler because 
> I didn't have the flags necessary to invoke what was needed to create 
> an actual working compiler that could do something, it basically 
> excluded the target generator!
>
> So, once I put all the flags in so that I can have something like 
> -DS370 in the command line recognized as a proposed target, how can I 
> get that symbol and the associated defines for the new processor 
> target added to the source base, even if it's not yet implemented? I 
> basically want to have a fixed target set so that if someone else 
> works on some other target there isn't a conflict.
The conflict can be resolved in the end. There aren't that many new 
platforms coming into existance anyway currently...
>
> I mean, if I didn't want to play nice and make this available to 
> everyone else, I could just say, "Okay, I need this done, I can't get 
> the FPC team to accept it for a branch for the 370, I'll fork the code 
> and make an incompatible version and now instead of a unified source 
> tree, there's now an independent branch somewhere else and people are 
> going to be confused. Not my problem." But I'd like to be nice and 
> avoid that.
You know that you could just ask Florian to give you a branch with write 
access? This way it will also be easier to comment on your changes.
>
> So, once I have picked the next set of flags to support a new 
> processor, that do not change anything for an existing compile that 
> doesn't invoke or use them, they're just there as hooks for when they 
> are placed into service, how do I get them added to the regular source 
> base so I don't end up creating a fork because what I've changed has 
> not been made available as part of the rest of the code base?
Just work on it in a branch. Even if someone should work at the same 
time on a different new platform than these problems can be resolved 
once your branch gets merged back to trunk.
>
> >> Second, how do I create a cross-compiler that runs on Windows,
> >> for any of the big-endian processors, so that I can see what it
> >> has to generate code for the target machine? Given that, I can
> >> change the code to either change the generated object file - if
> >> it does direct object files - or the assembly language so it
> >> generates S370 assembly.
> >
> > No. You should learn how the compiler generates code
>
> That's what I'm trying to figure out! One module calls another, which 
> calls another, and so on, and so on, "lather, rinse, repeat." In the 
> life of some program, the compiler front end calls one module, which 
> calls another, and on and on, and at some point it sees the "PROGRAM" 
> or "UNIT" symbol, or it sees a piece of code indicating that this is a 
> program that has no program statement, so it will then call the 
> backend telling it "You need to start compilation of a program, or a 
> unit" followed by "here's its name," or "it doesn't have a name."
>
> There's a place where the scanner hands off to the generator when it 
> gets a UNIT, VAR, or TYPE or PROGRAM or something else to indicate a 
> program or unit has started. Or an executable code for a BEGIN 
> statement. And I want to be there when it hands off, but I don't know 
> where.
>
The main part where code generation really kicks in is 
tcgprocinfo.generate_code in compiler/psub.pas. Please a breakpoint 
there and step through it looking at variables and such using the 
debugger. This helps very much though at first it might be confusing.
> The compiler as it stands is huge, and any hints on where to look is 
> what I'm asking for. And what I am asking for is, given the existing 
> code base, what flags and defines do I set, that, taking the compiler, 
> what flags do I set to change the default so that on this compiler, 
> running on Windows, with no flags set in the source, instead of 
> generating a Windows executable, it generates an executable for MIPS 
> or PPC?
>
> Now, then, once that is completed, taking the unmodified compiler and 
> running it through the new one will generate that new target even 
> though it's running on Windows. Then, I can go through the source tree 
> and see what's involved. But I have to start somewhere and I figure 
> one small question now could save me weeks of work trying to find it, 
> since, again, the documentation rather sparse on the question and is 
> none too open on how to do this.
>
> I figure someone's 30-second answer, well, you set this define, and 
> this switch, and point to this directory, and the compiler will now 
> generate MIPS or PPC code even though it's running on Windows. And so 
> I compile that version, once, now, every time I run that version of 
> the compiler it generates code for that machine instead of Windows. 
> Isolate the specific source files it uses, which could be anywhere 
> from 20 to 50 or more, and then I can go through that to change the 
> target machine. But first, I've got to figure out how to get FPC to 
> create a consistent different-machine target, and I figure someone 
> might be able to help.
>
> > and then implement a new code generator
> > backend for s370, maybe based on a copy
> > of another platform. Playing around with
> > the output won't help you.
>
> I don't want to play around with the output, I want to play around 
> with the files that generate that output! And I have to find out what 
> files are called, and by derivation, what objects, modules, procedures 
> and functions are being invoked.
>
The code generator is implemented inside the platform specific 
directories (e.g. compiler/i386, compiler/powerpc, etc.).

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freepascal.org/pipermail/fpc-devel/attachments/20130820/99c7590f/attachment.html>