It wasn't a word 5 minutes ago, but it is now. I spent all night practicing Buildery, or the act of building vast numbers of Solaris packages. Just as soon as Subversion 1.3 is released, I will unleash a raft of new packages via Blastwave. This will include Apache 2.2.0, Subversion 1.3.0, neon 0.25.4, and SVK 1.05. Most of the long night was going through the Blastwave build system and ensuring that configurations and package admin files for all of the 32 Perl modules that SVK depends on are up to date. Actually, SVK has 49 dependencies, if you count VCP and it's 16 dependencies, but I stopped counting VCP because I can't get it to work properly. VCP is an interesting module. It is used by SVK to allow the same command set to operate on Subversion, CVS, or Perforce repositories. I'm not certain how many people would actually use such a feature, but looking at the code, it appears that a great deal of effort has gone into it.

This work is part of an effort to get the build system back into shape after a long hiatus. In the beginning, I downloaded a version of GARNOME, one of the (finest) build systems for cutting edge GNOME sources. At the time, I was just trying to install a GNOME environment on my Solaris 8 desktop, as I can't really stand using CDE for any length of time. Also, as I was building GNOME, I was building a set of GNU utilities which I used for development and production support in Sun's Newark manufacturing facility. Each distribution was compiled and packaged manually, and was taking a long time, as the number of useful utilities I wanted to provide was increasing.

Eventually, I decided that it would be a great idea to put these package builds into a GARNOME style build sytem. GARNOME itself is based on the GAR ports system, which was developed by the Linux Bootable Business Card (Linux-BBC) project. GAR makes building software extremely easy -- just set a fiew fields in a template Makefile, and away you go. Over time, each time I received a request to add more Perl modules to our servers, or identified a new tool which was required for some purpose, I'd stick it in my GAR system. I then developed a GAR extension (GAR supports additional extensions as included make files) which would create Solaris packages. This system grew to approximately 300 software distributions, bundled into four packages (most of the 300 were CPAN modules).

In August 2004, I joined the Blastwave project. I had decided at that point, that it made no sense for me to maintain 100% of the packages that I used, when someone else was already doing it for me. I would contribute the packages that I build that nobody else had picked up, and reduce my workload. From day one, I used the GAR build system which I adapted for internal use. However, the Solaris packaging system, at this time based on make rules only -- phear, was error prone, and not as flexible as I'd like. For one thing, it could really only build a single package out of each build directory, and I needed it to build any number of packages from a single distribution install. In response, I split out the packaging logic into a utility called mkpackage.

The mkpackage utility reads a spec file, and performs some actions as a result. The idea is roughly like RPM, only it is far simpler, because, for better or worse, SVR4 packages are much more limited than RPM. It would solve so many problems if SVR4 packages adopted a provides/requires system, or at least supported minimum/maximum package version numbers rather than exact match. Maybe this functionality will appear once the pkg tools are eventually released to OpenSolaris. Anyway, the spec file is very simple. Here is the spec for apache2c, the core Apache 2.2.0 package:

%var            bitname apache2c
%var            pkgname CSWapache2c
%include        url file://%{GARDIR}/pkglib/csw_dyndepend.gspec
%var            desc Apache 2.2 web server (core)
%copyright      url file://%{WORKSRC}/LICENSE

Simple enough. The %var directive sets a variable for use later in the same file, or any included files. In the above case the bitname variable is used to construct the package filename. The %include directive does exactly what you think it does -- include another spec file. This directive takes two arguments: the first is a method, which in this case is url, indicating that the final argument is a URI which mkpackage must fetch. Any URI which is supported by LWP::UserAgent can be used here, including http://, ftp://, https://, and so on. The other supported method is exec, which will run some command and include the result.

The content of any %include is evaluated immediately. In the above case, the include resolves to this spec:

%include        url file://%{GARDIR}/pkglib/csw_vars.gspec
%include        url file://%{GARDIR}/pkglib/csw_prototype.gspec
%pkginfo        url file://%{GARDIR}/pkglib/csw/pkginfo
%depend         url file://%{GARDIR}/pkglib/csw/depend
%include        url file://%{GARDIR}/pkglib/std_depend.gspec

This is mostly includes, but shows the pkginfo and standard depend file for CSW (Community SoftWare -- Blastwave). There are several directives like %pkginfo which correspond to the different package admin files that SVR4 packages support, including %prototype, %preinstall, %postinstall, %space, and so on. These are treated a bit differently than an include. They create a file in the mkpackage work directory (configurable) called %{pkgname}.<admfile>, e.g. CSWapache2c.pkginfo. As the source file is written into the work directory, any %{variables} in that file are replaced with variables from the environment, those fabricated by mkpackage, and any set in prior %var statements. For instance, the standard CSW pkginfo file is:

PKG=%{MKP_PKGNAME}
NAME=%{MKP_BITNAME} - %{MKP_DESC}
ARCH=%{MKP_ARCH}
VERSION=%{SPKG_VERSION}%{SPKG_REVSTAMP}
CATEGORY=%{SPKG_CATEGORY}
VENDOR=%{SPKG_VENDOR}
EMAIL=%{SPKG_EMAIL}
PSTAMP=%{SPKG_PSTAMP}
CLASSES=%{SPKG_CLASSES}
HOTLINE=http://www.blastwave.org/bugtrack/

The MKP_ variables are supplied by mkpackage, and SPKG_ variables are exported to mkpackage by GAR. Everything is replaced when the pkginfo file is written, and any variables that cannot be replaced will cause mkpackage to complain.

As I mentioned earlier, there is a way to execute any arbitrary command and include the result. It is also possible to execute an arbitrary command to create one of the standard files. This is most often used to create depend and prototype files. Default outputs from pkgproto may be fine for most cases, but Blastwave has a set of standards for prototypes that need to be followed. That means that certain ownership needs to be assigned to files, and certain directories and filetypes need to be excluded. For this, I created a simple program called cswproto which takes arguments similar to pkgproto, and spits out a CSW-compliant prototype file. This utility is called as follows:

%prototype      exec cswproto -s %{TIMESTAMP} -v basedir=%{DESTDIR} %{DESTDIR}=/

This finds any files in the destination directory %{DESTDIR} which are newer than the timestamp file %{TIMESTAMP} (created and updated by an implicit make rule in GAR). As I don't build as root, I install software off to a temporary directory (usually /tmp/a). This means that the resultant prototype from cswproto always expects to pull the real file from /tmp/a/some/real/file. However, in the GAR system, it is possible (and useful in a number of circumstances) to change the DESTDIR at will. This makes life difficult for prototype files with a fixed DESTDIR path. The -v switch to cswproto replaces instances of %{DESTDIR} in each line of the prototype with the package build time variable $basedir. Before the transformation, a sample prototype line might be:

f none /opt/csw/lib/libfoo.so=/tmp/a/opt/csw/lib/libfoo.so 0755 root bin

After the transformation, the same line would be:

f none /opt/csw/lib/libfoo.so=$basedir/opt/csw/lib/libfoo.so 0755 root bin

The basedir variable is then supplied to pkgmk, so that it can find all files in the prototype, regardless of the current DESTDIR setting.

This system has proven to be very flexible -- I can now build any number of packages I require from a single software installation. For example, after building and installing PHP 5, I can now build 21 separate packages out of the installed files, separating out less commonly used modules that might have additional dependencies. This benefits users directly because less dependencies means less software to download for common tasks, and less disk space occupied by bits that will never be used.