Fattening packages - supporting multiple variants in a single package

Dealing with parts of a package

Traditionally, packaging systems have placed optional components of a package in separate packages, and established conventions for naming such components, such as -localization, -locale, -devel, -doc, etc. This method of ad-hoc decomposition makes it more difficult for GUI programs to offer the appropriate choices when selecting components, makes the introduction of new optional components difficult and makes installing documentation after the fact a painful process.

Packaging options also exist which are mutually exclusive; the typical example is which architecture the package supports. One cannot select both sparc and x86, since the two architecture's files collide - /usr/bin/ls is either a sparc binary or a i386 binary. Other examples of such colliding options include debug vs non-debug kernel binaries, and global vs nonglobal zones.

Initially we started solving the problem of selecting parts of a package using more or less ad-hoc client-side filtering, but it became clear that the publisher of the package needed to be able to describe what components intersected or were optional, and that we needed to be able to clearly distinguish between the two cases. Thus, we refer to options that may be selected or not selected in any combination, such as various locales, documentation, etc., as facets. Options which must be mutually exclusive are called variants. Both variants and facet appear as tags on IPS actions, and result in the action being selected or de-selected for installation.

An example of variant tags on an action is:

dir group=sys mode=0755 opensolaris.zone=global owner=root path=kernel/drv/amd64
  variant.arch=i386 variant.opensolaris.zone=global

Here this directory action is tagged with variant.arch=i386, indicating that this directory should only be installed on i386 machines (Solaris doesn't distinguish between 32 and 64 bit architectures) and variant.opensolaris.zone=global, which shows that this directory should only be installed in the global zone and not in local zones as those don't contain kernel components.

Note that any action in the repository can be so tagged, including dependencies.

During installation planning, all actions not applicable to the current image are removed from consideration. If a user with a down-rev version of IPS that doesn't understand variants attempts to install a fat package, either an exception will occur indicating that duplicate actions have been found, or if a set action is found with an architecture tag an assertion will trigger. In either case the solution is either to stop attempting to install fat packages, or upgrade your version of IPS to the latest available for your build.

The initial implementation in build 106 supports only variants (facets are coming later on). To faciliate detection of attempts to install packages on the wrong architecture, our publication code now inserts set actions into packages to indicate what architectures the package supports:

set name=variant.arch value=sparc value=i386 

Publishing Fat packages

We produce fat packages by publishing separately on each architecture, and then merging the resulting manifests. Actions which are identical for both architectures are not tagged w/ variant tags; those that differ or exist only for one architecture are tagged.  Packages that exist only on one architecture are tagged to indicate that as well. This step can be repeated for different variants, and more than two variants types can be combined at once, so packages containing debug kernel binaries could be merged w/ non-debug versions, and then merged across several architectures. An example of a merged manifest can be seen here. Of course, those using pkgsend to create packages can simply insert the appropriate tags directly. For the curious, the code to merge packages can be seen here. Further work is needed to improve the performance of publishing the merged packages; allowing multiple repositories to share the filestore would help greatly here.

Posted at 03:56PM Feb 04, 2009 by barts in packaging and patching  |  Comments[9]

Rethinking patching

As Stephen mentioned recently, several of us have been thinking about revising the way we manage software change on Solaris.  I've been particularly focused on the difficulties Sun and it's customers have with the patching process, and the kinds of changes we need to make as a result in our technology and development processes.

 Today, most customers don't run OpenSolaris; they run a supported version of Solaris such as Solaris 8, 9 or 10.  A supported release means that someone will answer the phone, and that patches for problems are available.

Patches are a separate software change control mechanism distinct from package versions in Solaris.  Each patch may affect portions of several packages; patches are intended to include all the files necessary to fix one or more problems, either directly or by specifying dependencies.  If a patch affects packages which are not installed on this system (typically because it has been minimized), those portions of the patch are not installed.  If the administrator later adds the missing package, he must remember (good luck) to re-apply the patches since the packaging code knows nothing of patches.

Customers are today free to install which ever patches they feel are appropriate for their environment, consistent with the built-in dependency requirements.  This customization is a technique I refer to as Dim Sum patching, and is a major cause of patching difficulties.  Many customers pick and choose amongst the thousands of patches available for Solaris 10, for example; this means that customers are often pioneering new configurations.  Note that each Solaris release consists of a single source base; all Solaris 10 updates, for example, are but snapshots of the same Solaris patch gate at different times.  As a result, the developers are working on a cumulative  set of all previous changes; when a new patch is created, the files in the patch not only contain the desired fix, but all previous fixes as well.  Thus, the software change is constructed as a linear stream of change, but customers installs selected binaries from the various builds via patches.

When I've discussed the hazards of  Dim Sum patching with customers, the reasons given are typically characterizable as :

1. we don't need all those patches,  we don't have those drivers loaded
2. we're reducing downtime by not installing so many patches
3. the less change, the less risk.

To these, I reply:

1. If you don't need those drivers, then remove them them w/ pkgrm rather than leaving them in an unpatched state awaiting the introduction of new hardware or software to expose problems.  Minimization, not spotty patching, is the answer.  This is akin to disposing of an unused car, rather than simply leaving it unmaintained.
   
2. Today, you should be using Live Upgrade and patching the alternate boot image to reduce downtime.  This allows machines in production to be safely patched, and will not leave the system in an inconsistent or unbootable state in the case of power failure during patching operations.  In the future, the new packaging system will always patch a clone of the current system to avoid the potential for disaster in case of power failure.
3. Our experience has been that customers running all of the changes in an update are generally far less likely to experience problems than those who select only the fixes and features that appeal to them, and hope that our QA processes found all hidden dependencies on previous changes.

For our new packaging system, there is a powerful incentive to eliminate Dim Sum patching:  since we wish to use a single version numbering space for any package, attempting to support fine-grain Dim Sum patching would require very small packages - affecting the performance of packaging operations, and significantly increasing the workload of OpenSolaris developers.   Instead, we can set package boundaries according to what makes sense for minimization purposes. 

This implies that future (post Solaris 11) patches will be completely cumulative (aside from some exceptions for urgent security fixes), at least for the core OS.  Your system will be able to determine what is needed to bring the installed software up to the desired revision level automatically; needing to pick and choose patches will be a thing of the past. 

Posted at 03:56PM Jul 25, 2007 by barts in packaging and patching  |  Comments[10]