[fpc-devel] The future of fpmake

[Marco van de Voort]

In our previous episode, Hans-Peter Diettrich said:
> > functionality belongs in a different tool (although it could be in
> > fpmake), but let's not limit fpmake only to FPC please. Let us (end
> > users) also play with it ;)
> 
> My view on (traditional) Make:

(Note that if I say packages here, I mean a logical set of units (and maybe
tests, docs and examples though IMHO that is secondary).  Nothing more,
nothing less. IOW don't get BPL visions)

I agree with your view on the general Unix status.
 
> In a new approach I'd provide the interpretation of existing MakeFiles,
> and extend it to the specific needs and capabilities of the FPC/Lazarus
> project (package...) model.

Personally I don't think the core makefile principles are worth preserving
at all:

1. they are mostly generated anyway. The .fpc files are the info, not the
   makefiles themselves.
2. they totally pass-by certain core FPC principles (like being to able to
   compile multiple files in one go. In the packages this is worked around
   using buildunits)
3. For e.g. a multithreaded compiler, the number of files to be handed to
   the compiler should increase, not decrease (*)

(*) there are multiple issues here to fix:
  - compiler is not mutithreaded.
  - module system is scheduled for rewrite for about a decade now
  - packaging (also static, iow DCP) could decrease number of files and
    speed up compiler. But what about backend tools?

> WRT time consuming jobs, observed e.g. in building FPC itself, I'd
> integrate "clean"ing of the output directories, together with means to
> reduce the number of such directories, as far as possible. I'd also add
> FPC as an integrated module, with a reusable file cache to prevent
> excessive directory scans and loading of configs and modules.

This only works when integrated with FPC, as the compiler finds its own
files. But that is all far, far away. The point is more that the current
transition horse makeshift situation (fpmake) should be adapatable in the
future.

At least currently its data is fairly abstract (though I'm not entirely
happy having to specify fileinfo manually.  That should be based on compiler
feedback. But this is very difficult in practice since the exact
buildprocess is OS dependant)

One big step of fpmake is the killing of tools outside the FPC project like
make.
 
> The compiler module can be used in the Lazarus IDE as well, either as a
> built-in module or as a shared library, with beforementioned ability to
> share the module cache with the application. Unfortunately parallel
> compilation will never become available with FPC, so that other chances
> for parallel processing should be explored. As long as the build tasks
> are based on input/output files (disk I/O), a shared file cache looks
> like the most promising way for speeding up an build process.

Such plans are on the decade scale. If you are interested in it, I'd start
creating as many internal assemblers and linkers as possible and further the
module-rewrite, and the organizing of units into larger concepts (packages
or whatever) and reducing the proliferation of files.
 
> When fpmake consists of dedicated modules for various tasks, these
> modules can be used to build other applications, like test suite runners 
> or profile editors.

I think the core bit of fpmake/fppkg should be 

(1) killing off external tools, and the compromises to package metadata that they
    force upon up.
(2) start developing a logical package concept, both from a building and
   packaging concept. Maybe with a file based representation (.lib/.dcp like)
(3) provide some hooks for packages outside the current main build tree to
   integrate with release engineering.

Both 1 and 2 as a start to more involved systems. At future compiler devel
implementing some in-compiler package/library concept or unit cache will
more easily roll it out.

Point 3 is because there is an enormous rift between package in the tree,
and outside of it. By cutting down the administration for this, we might get
a more workable external package management.