[fpc-pascal] Progress indication while linking

[Mark Morgan Lloyd]

Mark Morgan Lloyd wrote:
> Jonas Maebe wrote:
> 
>> The ld manual page lists some options you can use to reduce its memory 
>> usage:
>> * --no-keep-memory (makes it re-read object symbol tables from time to 
>> time instead of keeping them in memory all the time)
>> * --reduce-memory-overheads (small hash tables and slower but less 
>> memory-intensive link map generation algorithm)
> 
> I scripted some builds over the afternoon with all combinations of 
> those, running on a Linux SPARC system with 192Mb which I was hoping 
> would be tight enough to be a good test. Unfortunately the timings on 
> all four were very similar: --no-keep-memory (without 
> --reduce-memory-overheads) gave about a minute's advantage out of about 
> 18 minutes link time.
> 
> I was hoping that --stats output might show something useful, but I 
> think it gets swallowed. Vincent's suggestion of -t shows some useful 
> info during the first part of the link, but in the sort of extreme case 
> that I've currently got with Solaris I don't know at what point that 
> would peter out.

--stats does work. Running on a tighter system (128Mb) I get times for 
make all  plus seconds in the linker:

time make OPT='-k-t -k--stats' all
real    138m30.494s
user    26m43.840s
sys     3m38.614s
/usr/bin/ld: total time in link: 103.262453	lazarus
/usr/bin/ld: total time in link: 32.394024	lazbuild
/usr/bin/ld: total time in link: 34.706169	startlazarus
Time to link all three binaries: 170 secs

time make OPT='-k-t -k--stats -k--no-keep-memory' all
real    130m47.770s
user    28m1.017s
sys     3m52.851s
/usr/bin/ld: total time in link: 150.777423	lazarus
/usr/bin/ld: total time in link: 44.086755	lazbuild
/usr/bin/ld: total time in link: 50.603162	startlazarus
Time to link all three binaries: 246 secs

time make OPT='-k-t -k--stats -k--reduce-memory-overheads' all
real    125m16.758s
user    26m57.849s
sys     3m28.657s
/usr/bin/ld: total time in link: 93.757858	lazarus
/usr/bin/ld: total time in link: 31.613976	lazbuild
/usr/bin/ld: total time in link: 33.146071	startlazarus
Time to link all three binaries: 159 secs

time make OPT='-k-t -k--stats -k--reduce-memory-overheads 
-k--no-keep-memory' all
real    114m6.972s
user    27m51.780s
sys     3m34.465s
/usr/bin/ld: total time in link: 142.376898	lazarus
/usr/bin/ld: total time in link: 43.362710	lazbuild
/usr/bin/ld: total time in link: 49.955122	startlazarus
Time to link all three binaries: 235 secs

That appears to show a correlation between "user time" and the time the 
linker thinks its linking the big binaries, suggesting that the latter 
is got from process accounting rather than simple wallclock.

The thing that really matters is the wallclock time, where performance 
on a tight system can be improved slightly by feeding options to ld. 
I've not tried this on a very tight system, i.e. 32Mb RAM since runtime 
for the four tests would be around a month.

In all cases ld reports that linking the lazarus binary requires >200Mb 
working store, while the others require 120-150Mb. I think that explains 
why differences in timing start to stand out once there's less than 
128Mb physical RAM.

I don't know whether there's a significant downside to using these 
options on a large system, there might be if disc storage was 
comparatively slow.

The thing that would probably make the most difference would be running 
in single user mode without any background daemons, but that's probably 
not possible unless logged in via a console connection.

-- 
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]