[fpc-devel] Interface to compressed files and archives

[Marco van de Voort]

> I'm new to this list and want to introduce myself and my intended
> contributions to FreePascal.

You are not new to everybody:-)
 
> archive files (cpio, tar...).  I've already update or implemented some
> of these modules, now I want to define a common interface and API for
> compressed and archive streams, based on TStreams. The zstream unit is
> dedicated to a single compressor, but it has an handy name. How should I
> name a more general unit, would "zstreams" be acceptable?

Maybe just streams. I also have a encryption stream (Rijndael) based on a PD
unit that I could try to adapt. Of course that depends on how specific the
interface is for compression  (compres//decompress crypt/decrypt are similar)

> Then a general Open or Decompress procedure can determine which
> decompressor to use for an given stream, and can create the appropriate
> decompressor object. For compressors it may be better to create the
> according object directly, according to the desired compression format,
> in which case the according arguments also can be passed to the
> constructor of that class in the appropriate form.

Better have a separate way. Otherwise you can't set e.g. a compressionlevel
for that stream, _or_ you have to have lots of different constructors.

One other thing to keep in mind (iirc) is that some algo's require the
uncompressed size to unpack, and some the compressed size. So probably
your interface has to support both. 

And use a 64-bit size and an endianness indicator if possible.

> Archive files deserve a more elaborate API, so that the files in an
> archive can be extracted to individual files or streams. There was
> already a suggestion, to define something like a virtual file system
> interface for archive files. I suspect that something like this already
> exists for use in the GUI browsers of both Linux and Windows. This may
> deserve some research, before an accordingly compatible interface can be
> defined. Now I'm waiting for according contributions from the OS gurus
> before proceeding with this approach.

Search for "zfs" (zip filesystem). It was FPC compat for a while.
 
> Now you should have gotten the big picture of my intended activities.
> Many more questions will arise when I proceed with my work. I already
> decided to replace my own "stdc" unit by the FPC "libc" unit, with
> hopefully no changes to that unit. 

Then change them again to use BaseUnix/Unix :-)

The "libc" unit is not a base unit of FPC, but exists merely for Kylix
porting purposes, since it is pretty much only a direct x86 glibc
translation, and not a general POSIX abstraction. It is only supported for
Linux/x86.

As said, a portable subset is available in units baseunix/unix, with
threadsafe errno and most calls have "fp" prefixed to avoid nameclashes with
older FPC unix units (oldlinux) and Kylix' libc.

See http://www.freepascal.org/docs-html/rtl/  for some docs, and 
http://www.stack.nl/~marcov/unixrtl.pdf  

> For further compatibility it will be
> necessary to find compromises between my coding style, and the style
> used by the FPC community. E.g. I prefer to prefix all my units with an
> "u", so that the base names remain available for procedures or
> variables. I also use upper case characters in the unit names, what may
> not be appreciated by users from the Unix world. As a compromise it may
> be possible to use a "lib" prefix, but this may conflict with existing
> library names (libz...). Any ideas?

unit prefixes for units belonging together are not uncommon (see e.g. the
image and db routines), but I would avoid the uppercase unitnames and don't
use already taken prefixes like "lib", "db", "unix" and "fp"