pkg(5): Leaving the build system "out"

I've been busy the past weeks with school transitions and with getting the community defect tracking system requirements into a shape where we can start evaluating candidates.

Identifying the boundaries of a system during the design phase affects feasibility critically, perhaps more than any other choice. This choice reduces to "know what you're trying to create". As with my previous post, I'm going to describe something we're not doing—and explain why. I envision one more negative post, and then we'll get onto more positive expressions.

So: this packaging system does not contain a build or compilation component in its core architecture. There are pragmatic reasons for this choice, as well as technical ones.

One way to look at a packaging system is, similar to how we saw it as the connecting layer between the installer and lower-level OS services, as a way to collect and organize the set of components, binary and otherwise, into an always "bootable" flow of change. That is, various groups of people are, using their systems, emitting binaries, documents, images, and so on, that other groups of people combine into releases (or atoms of a release, like an updated package). The packaging system has to assist the latter group in that combination, by providing visibility into the completeness of the outputs of the former groups. It's less clear that the latter group needs to have the ability to construct the objects, as long as they can assess that the environment the objects need to execute can be realized.

Just as an aside, we're treating as critical design input exactly who touches and how they touch a software component as it proceeds through development, localization, release engineering, sustaining, and so forth. These touches might be reduced as the development becomes more open, but at present, they're a useful constraint on any tendency to make the system overly rigid (or overly monolithic).

The OpenSolaris consolidations—and related open source outputs like OpenJDK or OpenOffice, among others—don't presently share a common build system. In fact, one of the real impacts of Sun's progression to an open software development culture is that we're moving from being the originator or primary maintainer of 90% of the software we deliver, to a much more modest percentage—let's say 30 – 50%. Forcing a unified build system upon all of these disparate products is asserting the need for a long series of difficult conversations over many months.

Instead, let's defer those discussions, and see if we can get the same benefits while only managing the end outputs of each of the participating (or aggregated) publishers. (A byproduct is that injecting received binaries into the system is the common case, rather than a strange or special workaround.) As we noted in the earlier post, we have goals about safety, developer burden, and stopping "bad stuff" as early as possible.

Static analysis can get most of the dependency cases correct. Binaries, whether they are user applications, libraries, or kernel modules, contain a significant amount of dependency information. It turns out that many scripting languages can be roughly interpreted to determine their module requirements. Similarly, Java class files and JAR files—and even smf(5) manifests—contain a great deal of information that lets us determine the self-consistency of a system.

Of course, a program can evade these dependencies: beyond use of dlopen(3C), it can implement its own linking or overlay mechanism, or simply be implemented in a language unknown to the packaging system. The point is we can drive out most of the inconsistencies that a purely manual dependency statement allows. In fact, we can warn a developer about the incompleteness of their dependency statements, potentially correcting them: adjusting version requests, inserting omissions, even asking about possibly superfluous dependency correctness.

That said, it might be that an ideal build system is lurking out there to be layered atop this system; we'll leave room for expressions, like stating a build dependency on a certain tool, like lex(1) and yacc(1), say, so that a build system (or systems, if folks can't close those difficult conversations I mentioned) can benefit from the metadata discovered about each component. (If you're interested in constructing such a system, we thought a little about requirements for the SFW consolidation.)

[ T: OpenSolaris Solaris Indiana pkg smf ]

pkg(5): No more installer magic

I thought I would continue probing some of the problems that present themselves to any packaging system that might follow the System V packaging. For the next few topics, I'll phrase the problem in terms of an outcome I believe we want to avoid. Here we discuss aspects of eliminating special metadata from installers.

If you're familiar with the packaging and installation components involved with Solaris Express, Developer Edition, or any of the OpenSolaris distributions that Sun produces, then it won't surprise you that there's a large amount of upgrade specific knowledge in the installation layer. For instance, to upgrade a package installed with Solaris 10 to its currently delivered version in a recent build, there's the set of files associated with the package—and then there's the package history, which collects additional information, like files no longer delivered, or the responsibility of another package. The presence of this file, and the absence of a true package upgrade operation in System V packaging, mean that any kind of upgrade requires some form of installer: the operating system installer for upgrading the entire installation; an application-specific installer for upgrading a group of related application packages.

There are a number of other metadata files trapped in the installation layer, but the most important are the metacluster files, which group the package clusters into large installation profiles, and the package clusters themselves, which group sets of packages along feature boundaries, approximately. It seems evident that these groupings are merely another kind of dependency, much like how a System V package can, naïvely state a dependency on any other System V package.

The System V dependencies also show that another critical piece of metadata—the versioning vector, or "arrow of time"—is encoded in the installer.

All of this information, if we are to allow packages to upgrade from one version to another in some linear fashion, needs to be pulled out of the installers and moved into some aspect of the packaging system. This change in responsibility means that the role of the installer becomes more precise: it must prepare a location for software installation, optionally lay down some initial, and possibly stale image, and collect any required configuration information. Subsequent upgrade operations are driven primarily by the packaging system, which can utilize the version history and dependencies in a manner at least equivalent to what the historical metadata allows.

The other reason that we wish to push that historical metadata into the packaging system is so that it becomes accessible to a new class of application: the distro construction toolkit, which needs the dependency and versioning information to simplify the construction of self-consistent installable images. That leads us to an architectural diagram like

where we've suggested some internal structure to an updated packaging system. This separation shouldn't be that surprising: in the current system, lumake(1M) and most of LiveUpgrade as well as zoneadm(1M) for Zones are both performing image operations on either side of any packaging operations they might invoke. Designing a packaging system that makes the constructions of distributions and their installers substantially simpler will require such an API layer.

There are some open questions that come to mind:

• How much variation is there in the initial preparation an installer offers for the image location? Is space management a policy owned exclusively by the installer, or should the image operations layer have the ability to influence that allocation?
• Is there a marshallable image description, beyond the package dependencies? Is there a marshallable image format, or are we restricted to end-use formats, like ISO images or archived zones?
• What OS services should we consider modifying, beyond taking advantage of ZFS's current capabilities?

If you're interested in these questions, or the potential architecture of a packaging system, we'll be discussing these topics further in the Install community group.

[ T: OpenSolaris Solaris Indiana pkg ]