2008.05: More ways to get it

As I did for the preview releases, I'll collect links to mirror sites here. These will also get links on the various download pages out there.

Bart and I just finished updating the package repository with the new packages we'd received the past few weeks, and that means 2008.05 is out the door. (Thanks to everybody who tried the release candidate, filed a bug, shuffled a package, wrote or proofread a document, or just spent energy anticipating the bits.) You can get the ISO image, suitable for burning to a 700 MiB CD or immediate use in virtual environment, directly from the following locations:

• via HTTP from dlc.sun.com, and
• via HTTP from genunix.org,
• via HTTP from ftp.df.lth.se,
• via FTP from ftp.df.lth.se,
• via HTTP from mirrors.xservers.ro,
• via HTTP from gulus.usherbrooke.ca,
• via FTP from ftp.nluug.nl,
• via HTTP from ftp.nluug.nl,
• via rsync from ftp.nluug.nl,
• via FTP from ftp.surfnet.nl,
• via HTTP from ftp.surfnet.nl,
• via rsync from ftp.surfnet.nl, or
• via BitTorrent.

Reading the logs, and talking with some of our mirror sites, we know we all served out a lot of downloads for the previews; if you're interested in being a mirror, please let me know. (2008.05 remains completely redistributable.) We're using a bigger download complex this time, but every mirror helps.

If you already downloaded and installed Preview 2, it's more complete, easier, and probably faster to update directly using image packaging: see the update guidelines. These instructions involve a small script to safely update a clone of your installed system, and then switch to that on a system reboot. (If you were running Preview 1, you should update that to Preview 2, and then go on to 2008.05.)

Update 1: My thanks to Tobias Lundquist, who's once again mirrored via FTP and HTTP (Internet 2) in Sweden.

Update 2: My thanks to Luca, who's put up an HTTP mirror in Romania.

Update 3: Simon Valiquette has put up an HTTP mirror (Internet 2) at the Université de Sherbrooke in Québec, Canada. Merci, Simon.

Update 4: Bart Muijzer has arranged for mirrors by the Netherlands Unix Users Group (NLUUG) and surfnet.nl. Thanks, Bart.

These links are for the gzipped CD image, which contains the 12 "primary languages". It installs quite a bit faster, particularly on systems with slower CPUs. There is also an LZMA-compressed image, which has localization support for 42 languages, including those primary ones. It's available from dlc.sun.com, genunix.org, ftp.df.lth.se, mirrors.xservers.ro, and as a torrent. (Consult the language lists for specifics.)

[ T: OpenSolaris indiana ]

Live chat on #os200805

We're talking about OpenSolaris 2008.05 over on IRC, using the #os200805, from now until 2 p.m. Pacific Time. If you haven't got an IRC client, you can use freenode's Java applet, which worked great (until Danek kickbanned me). Enter a nickname, and then ask some questions.

Apparently the load from 60 000 users blew out the chat system, so we moved to IRC.

OpenSolaris: Bug dependencies and release management

Right now, if you're subscribed to any of the Installation and Packaging community group or project lists, you'll see a lot of commit notifications as the various teams attempt to fix various bugs noted since the second Developer Preview release. We've been using the trial Bugzilla instance—which is becoming the default defect tracker for opensolaris.org it appears—and trying out various features.

For tracking release completeness, we're using "blocker bugs" or "tracker bugs", which are synthetic bug entries that we mark various important bugs ("stoppers") as blocking. That means that we end up creating a little dependency graph that shows what unfixed bugs are stopping us from reaching some initial set of release criteria. We have two tracker bugs

• 571 Blocker bug for Indiana 3
• 1190 Packaging bugs to fix for May (tracker bug)

Bugzilla has two nice summaries for showing this information, in addition to the default bug status page. I'll use 571 as the example tracker bug, since we've made 1190 block it—which leads to a more nteresting graph. The tree view is a useful and succinct representation, where indentation shows dependencies. The graph view is a bit unwieldy for this bug, but might be useful if the tree view became too long.

A useful technique if you're trying to bring a release together.

[ T: OpenSolaris Bugzilla pkg ]

pkg(5): Reverse proxying your depot with Apache HTTPD

As part of the changes to get Developer Preview 2 ready, we decided to rejigger the HTTP handling on pkg.opensolaris.org so that we could have more options as more people attempt to use the early versions of image packaging. Previously, we ran pkg.depotd directly on port 80, in its read-only mode; now we use Apache HTTPD to listen on port 80, and use mod_proxy to proxy those incoming requests to a pkg.depotd instance listening on a separate port. With a couple of different approaches

Proxying and rewriting is one of those endlessly fun activities that somehow actually ends up being productive. Last time, proxying fun led Steve and I to fiddling around such that we ended up with proxy and rewrite patterns to enable the country portals for opensolaris.org.

If you want to share the top-level component of your URL space, you'll need to watch pkg(5) developments, as you have to map the list of operations one-by-one—and I know there are some new operations forthcoming. That would involve adding something like the following to a VirtualHost directive in your Apache configuration.

ProxyRequests On

Redirect /index.html http://pkg.opensolaris.org/status

ProxyPass /abandon http://pkg.opensolaris.org:10000/abandon
ProxyPass /add http://pkg.opensolaris.org:10000/add
ProxyPass /catalog http://pkg.opensolaris.org:10000/catalog
ProxyPass /close http://pkg.opensolaris.org:10000/close
ProxyPass /feed http://pkg.opensolaris.org:10000/feed
ProxyPass /file http://pkg.opensolaris.org:10000/file
ProxyPass /filelist http://pkg.opensolaris.org:10000/filelist
ProxyPass /manifest http://pkg.opensolaris.org:10000/manifest
ProxyPass /open http://pkg.opensolaris.org:10000/open
ProxyPass /search http://pkg.opensolaris.org:10000/search
ProxyPass /versions http://pkg.opensolaris.org:10000/versions

ProxyPass /css http://pkg.opensolaris.org:10000/css
ProxyPass /logo http://pkg.opensolaris.org:10000/logo
ProxyPass /icon http://pkg.opensolaris.org:10000/icon

ProxyPass /status http://pkg.opensolaris.org:10000/

Configuring your server in this fashion allows you to mix an image packaging server in with your other site content. You can easily deliver static content alongside your depot, for example.

If you don't mind pushing your package repository down one level in your URL space, then the above simplifies to

ProxyRequests On

ProxyPass /pkg/ http://pkg.opensolaris.org:10000/

(which should be a hint on how to create a repository farm under a single URL). To use the latter, you would use pkg(1)'s image-create subcommand

$ pkg image-create -F -a mypkgs.com=http://www.myserver.com/pkg /path/to/image

to connect your image to your reverse-proxied packaging depot.

In the two examples above, you should of course replace machine names like pkg.opensolaris.org and port numbers like 10000 with values appropriate to your own installation.

Happy proxied package serving!

Feel free to share your alternative configurations or approaches with other HTTP servers here, or on pkg-discuss@opensolaris.org.

[ T: OpenSolaris Solaris pkg httpd ]

Indiana: More ways to get to Preview 2

As I did for Preview 1, I'll collect links to mirror sites here, as well as on the project page.

So we just released the second Indiana preview ISO, which you can get from the following locations:

• via HTTP from dlc.sun.com, and
• via HTTP from genunix.org,
• via FTP from ftp.df.lth.se, or
• via HTTP from osol.acm.jhu.edu.
• via BitTorrent.

We had a lot of downloads for Preview 1; if you're interested in being a mirror, please let me know. (Indiana remains completely redistributable.) We're using a bigger download complex this time, but every mirror helps.

If you already downloaded and installed Preview 1, it's more complete, easier, and probably faster to update directly using image packaging: see the update guidelines on opensolaris.org. These instructions use ZFS and pkg(5) to safely update a clone of your installed system, and then switch to that on a system reboot.

Update 1: My thanks to Tobias Lundquist, who's once again mirrored via FTP (Internet 2) in Sweden.

Update 2: My thanks to trisk, who's once again put up an HTTP mirror (Internet 2) on the East Coast of the USA.

[ T: OpenSolaris Solaris indiana ]

OpenSolaris: Finding work(ers)

A relatively new forum on opensolaris.org is opensolaris-jobs, a community-moderated list where OpenSolaris-related positions can be posted and which should give access to the people most passionate, most expert, and most interested in making OpenSolaris work their job. In the past few weeks, there's been two postings you might find interesting:

• Networking summer interns, Sun Microsystems, and
• Senior kernel developers, Intel.

Have a look, or—if you're hiring for OpenSolaris knowledge—get in touch with great candidates.

[ T: OpenSolaris jobs ]

pkg(5): Fueling the next steps

Most Fridays, I spend the two hours before lunch assisting at our co-operative daycare. The hour and a half before are pretty good thinking hours, most of which lately have been spent on packaging. As the year passes, and we move into fall here in Northern California, the thinking's been best assisted by having a brief, and warming, snack.

The notes being written are about some of the points raised about the use of hashing, but then I buckled down and wrote an outline for our ARC inception materials, which will probably take more than one or two Friday sessions.

Location: Café Borrone, Menlo Park, California.

[ T: OpenSolaris pkg ]

Indiana: More ways to get it

So there are a few ways to get a copy of the Indiana preview ISO:

• via HTTP from dlc.sun.com,
• via HTTP from genunix.org,
• via HTTP from osol.acm.jhu.edu,
• via FTP from ftp.df.lth.se, or
• via BitTorrent.

Both our mirrors are in Northern California, so—since it's a completely redistributable set of software—if you're interested in being a mirror (even in California), please let me know. If you're a torrent fan, come join me and seed.

Update 1: My thanks to trisk, who's put up a fast mirror (Internet 2) on the East Coast of the USA.

Update 2: My thanks to Tobias Lundquist, who's put up an FTP mirror (Internet 2) in Sweden.

Icon Design by IconBuffet (from "Durango Office" and "Durango Research").

[ T: OpenSolaris Solaris indiana ]

Indiana: VESA if you need it

Over the past week, as we kept reassembling the distro constructor, image packaging, and slim install, we tested installs on a bunch of laptops. This manual operation let me rehearse how to get things working if the Preview LiveCD doesn't have a graphics driver that will work. So: here's the VESA workaround.

1. Boot text mode. One of the non-default lines in the GRUB menu should say "text"—pick that one.
2. Login as jack. User is "jack", password is "jack".
3. Generate a representative xorg.conf. Become root and have Xorg generate an initial configuration. The root password is "opensolaris". (Enjoy that `root`'s default shell is 'bash' for a bit.)

   $ su
   Password:
   # /usr/X11/bin/Xorg -configure
   ...
   

   This procedure should create /jack/xorg.conf.new.
4. Change the driver. As root, you'll continue by editing that xorg.conf.new file in vi(1). Search for the Device section, and modify the Driver line to

       Driver  "vesa"
   

5. Make it go. Stay root and hand-launch GNOME.

   # /usr/X11/bin/xinit /usr/bin/dbus-launch gnome-session -- \
           /usr/X11/bin/Xorg -config /jack/xorg.conf.new :0
   

6. Launch the installer. Bring up a terminal, and invoke the installer, by typing install-lan &.

If your install fails, please file a bug report at defect.opensolaris.org. If it succeeds, but graphics doesn't work, you can follow the same steps, but you'll have to edit the GRUB entry. (Roughly, type 'e', add "-s" to the end of the line with kernel/unix, and log in with the new root password you set during install. I haven't tested this portion—let me know if it fails.)

Last night, I had to do this for my trusty VAIO T370P, but tonight it's fine:

Please let us know how it goes.

[ T: OpenSolaris Solaris indiana pkg ]

pkg(5): Talking in the redwoods, talking on the beach, ...

Danek and I are at the 2007 OpenSolaris Developer Summit at UC Santa Cruz, talking about and, I hope, later demonstrating the image packaging prototype so far. I gave a brief overview of the project's goals and status, and mentioning some of the unmentionables we've encountered—some particularly undisciplined configuration files, some apparently important but encumbered drivers, and so on.

View the slides

[ T: OpenSolaris pkg UCSC opensolaris_summit_2007 ]

pkg(5): project opens, development continues

I asked my teammates to take a brief break from prototyping, and we jumped from a collection of systems inside Sun to the clean project hosting at opensolaris.org.

As the migration registers, you should be able to access

• the project web site at http://opensolaris.org/os/project/pkg/,
• the Mercurial repository via

      $ hg clone ssh://anon@hg.opensolaris.org/hg/pkg/gate pkg-gate
  

• the same source via OpenGrok (using the 'pkg' project as a filter) at http://src.opensolaris.org/, and
• have a discussion on pkg-discuss@opensolaris.org.

If you're interested, please come check it out.

[ T: OpenSolaris Solaris pkg ]

pkg(5): a no scripting zone

In my previous two posts, we examined two packaging system options—installer-specific knowledge and integrated build system—that I believe present costs that exceed their benefits. Here, we will again examine a design choice from a negative perspective: package-associated scripting.

System V packaging is rich with scripting hooks; scripts named checkinstall, preinstall, postinstall, preremove, request, and the class action scripts. Each of these scripts can do anything they like. Scripting, even in a relatively primitive shell, is an open-ended program—opaque to the invoking framework. It's difficult to catch an incorrect script prior to package publication time, which blocks our intent to prevent propagation of bad package versions. With a more limited set of actions—potentially with that limit enforced or marked—a class of incompletely known resource handling mechanisms can be kept off the most conservative systems.

One goal we have is to preserve or improve the hands-off behaviour associated with package operations. Legacy packaging allows hands-off by imposing a series of tasks on the deploying administrator. The pkgask(1M) tool can enable the deployer to develop a response to the request script; coming up with an appropriate admin(4) can restrict the framework's built-in interactive queries. (Interaction with signed packages also requires the deployer to modify their pkgadd invocation.) Removing the scripting degree of freedom means that obstacles to hands-off behaviour come solely from an interactive installer or from interactive services acting during system startup.

There's some amusingly egregious violations of the hands-off principle across the space of known packages. Less fun is that these set a bad example for later package developers.

A particularly error-prone aspect of the scripting interface in packaging comes from the variety of contexts the package developer must understand (and test within). It is legitimate to install packages on live systems, in alternate filesystem hierarchies of the same or different architecture, and in whole-root and filesystem inheriting zones; in fact, you have multiple choices about how your package should install in a zone.

We can expect the proliferation of virtualized systems, via the various mechanisms like LDOMs and xVM, to keep all of these contexts relevant as degrees of sharing make virtualization even more appealing. Making sure that the package system operates safely in these shared contexts is critical—another of our goals.

Returning to the zones case, the example pseudo-script in pkginfo(4)—a series of nested shell if ; then blocks to navigate some of these contexts—is helpful, but misleading. There is much more variable state a package developer needs to consider to reach correctness. In fact, if you aren't required to rediscover or reinvent a set of resource-handling cases for each components your package delivers, it becomes substantially simpler to make the package and return to improving the software it contains. Reducing the set of steps reduces developer burdens associated with packaging.

Two particular resources stand out: device drivers and smf(5) services. Although some limited amount of awareness—or at least easily duplicated code—makes these resources somewhat well-behaved during package operations, there are still problems that scripting presents: the addition of new contexts, the provision of multiple genealogies of copied code, and the failure to discover an associated best practice for any particular kind of resource.

There are other resources, of course; as a start, you could duplicate our survey of the ON postinstall and class action scripts.

I believe the key counterargument supporting scripting is that the set of configuration patterns on Unix-like systems is large, and that the easiest means of upgrading each of these potential patterns is to allow a complete programming environment to the package developer. Probably true, but if we look at service and application configuration with respect to when a correct configuration state is required, the update step appears to separate into three classes:

1. Correct at system startup, no runtime context needed. These are the configuration settings that the various low-level boot components, the kernel, and the drivers need to bring the system to its running state. This class of configuration is generally limited to a specific set of resources, potentially established by a packaging system via corresponding resource-handling actions—or by an installer.

2. Correct at system startup, requiring runtime context. These are settings where the manipulating agent might be influenced by policy or require some form of interprocess communication to effect configuration changes. smf(5) is an example of the latter, and handles its configuration evolution via the manifest-import service. Manipulation of the various local name service tables, like passwd or the RBAC configuration is another example, since data about potential principals must be correct for a group of affected services. Since such configuration can be required on the system as a result of package operations, these resources must also be handled via packaging, or require the use of an appropriate installer.

3. Correct prior to service startup. Most service and application configuration falls into this class. It's not necessary, for instance, to bring a web server's configuration up to date if the service has no enabled instances. There seem to be a number of avenues for handling this kind of configuration: leaving it to the service or application, providing assistance via a configuration mechanism, or giving a hook where such updates can be made as needed. But the packaging system needn't provide this hook—there are a number of possible facilities, of varying suitability.

I should point out that David is making the smf(5) configuration update scenarios much more capable and precise with the Enhanced Profiles project. So, at least, a "configuration mechanism with assistance" is likely to be present soon.

Since the first and second classes and how their configuration manipulations vary in the various operating contexts are generally known, elimination of the third class makes precise, no-scripting packages a viable design choice.

That's a long series of arguments in favour of a scripting-free package system. It would be reasonable to ask: "can you actually do it?" So, as a check on our prototype, we used the branded zone capability to let us create a pkg(5)-based whole root zone. Here's a transcript

# zonecfg -z pkg_test
pkg_test: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:pkg_test> create -t SUNWipkg
zonecfg:pkg_test> set zonepath=/export/pkg_test
zonecfg:pkg_test> commit
zonecfg:pkg_test> ^D
# zoneadm -z pkg_test install
Preparing image
Retrieving catalog
Installing SUNWcs SUNWesu SUNWadmr SUNWts SUNWipkg
Setting up SMF profile links
Copying SMF seed repository
Done (115s)

There's dependency following, but no constraint handling; there's no filtering or snapshotting, but also none of the obvious performance optimizations has been implemented (for our 211MB resultant image). But the main point is: it works—installs, boots, upgrades, and still boots—with no scripting. Time for a project proposal.

[ T: OpenSolaris Solaris zone Indiana pkg ]

pkg(5): Leaving the build system "out"

I've been busy the past weeks with school transitions and with getting the community defect tracking system requirements into a shape where we can start evaluating candidates.

Identifying the boundaries of a system during the design phase affects feasibility critically, perhaps more than any other choice. This choice reduces to "know what you're trying to create". As with my previous post, I'm going to describe something we're not doing—and explain why. I envision one more negative post, and then we'll get onto more positive expressions.

So: this packaging system does not contain a build or compilation component in its core architecture. There are pragmatic reasons for this choice, as well as technical ones.

One way to look at a packaging system is, similar to how we saw it as the connecting layer between the installer and lower-level OS services, as a way to collect and organize the set of components, binary and otherwise, into an always "bootable" flow of change. That is, various groups of people are, using their systems, emitting binaries, documents, images, and so on, that other groups of people combine into releases (or atoms of a release, like an updated package). The packaging system has to assist the latter group in that combination, by providing visibility into the completeness of the outputs of the former groups. It's less clear that the latter group needs to have the ability to construct the objects, as long as they can assess that the environment the objects need to execute can be realized.

Just as an aside, we're treating as critical design input exactly who touches and how they touch a software component as it proceeds through development, localization, release engineering, sustaining, and so forth. These touches might be reduced as the development becomes more open, but at present, they're a useful constraint on any tendency to make the system overly rigid (or overly monolithic).

The OpenSolaris consolidations—and related open source outputs like OpenJDK or OpenOffice, among others—don't presently share a common build system. In fact, one of the real impacts of Sun's progression to an open software development culture is that we're moving from being the originator or primary maintainer of 90% of the software we deliver, to a much more modest percentage—let's say 30 – 50%. Forcing a unified build system upon all of these disparate products is asserting the need for a long series of difficult conversations over many months.

Instead, let's defer those discussions, and see if we can get the same benefits while only managing the end outputs of each of the participating (or aggregated) publishers. (A byproduct is that injecting received binaries into the system is the common case, rather than a strange or special workaround.) As we noted in the earlier post, we have goals about safety, developer burden, and stopping "bad stuff" as early as possible.

Static analysis can get most of the dependency cases correct. Binaries, whether they are user applications, libraries, or kernel modules, contain a significant amount of dependency information. It turns out that many scripting languages can be roughly interpreted to determine their module requirements. Similarly, Java class files and JAR files—and even smf(5) manifests—contain a great deal of information that lets us determine the self-consistency of a system.

Of course, a program can evade these dependencies: beyond use of dlopen(3C), it can implement its own linking or overlay mechanism, or simply be implemented in a language unknown to the packaging system. The point is we can drive out most of the inconsistencies that a purely manual dependency statement allows. In fact, we can warn a developer about the incompleteness of their dependency statements, potentially correcting them: adjusting version requests, inserting omissions, even asking about possibly superfluous dependency correctness.

That said, it might be that an ideal build system is lurking out there to be layered atop this system; we'll leave room for expressions, like stating a build dependency on a certain tool, like lex(1) and yacc(1), say, so that a build system (or systems, if folks can't close those difficult conversations I mentioned) can benefit from the metadata discovered about each component. (If you're interested in constructing such a system, we thought a little about requirements for the SFW consolidation.)

[ T: OpenSolaris Solaris Indiana pkg smf ]

pkg(5): No more installer magic

I thought I would continue probing some of the problems that present themselves to any packaging system that might follow the System V packaging. For the next few topics, I'll phrase the problem in terms of an outcome I believe we want to avoid. Here we discuss aspects of eliminating special metadata from installers.

If you're familiar with the packaging and installation components involved with Solaris Express, Developer Edition, or any of the OpenSolaris distributions that Sun produces, then it won't surprise you that there's a large amount of upgrade specific knowledge in the installation layer. For instance, to upgrade a package installed with Solaris 10 to its currently delivered version in a recent build, there's the set of files associated with the package—and then there's the package history, which collects additional information, like files no longer delivered, or the responsibility of another package. The presence of this file, and the absence of a true package upgrade operation in System V packaging, mean that any kind of upgrade requires some form of installer: the operating system installer for upgrading the entire installation; an application-specific installer for upgrading a group of related application packages.

There are a number of other metadata files trapped in the installation layer, but the most important are the metacluster files, which group the package clusters into large installation profiles, and the package clusters themselves, which group sets of packages along feature boundaries, approximately. It seems evident that these groupings are merely another kind of dependency, much like how a System V package can, naïvely state a dependency on any other System V package.

The System V dependencies also show that another critical piece of metadata—the versioning vector, or "arrow of time"—is encoded in the installer.

All of this information, if we are to allow packages to upgrade from one version to another in some linear fashion, needs to be pulled out of the installers and moved into some aspect of the packaging system. This change in responsibility means that the role of the installer becomes more precise: it must prepare a location for software installation, optionally lay down some initial, and possibly stale image, and collect any required configuration information. Subsequent upgrade operations are driven primarily by the packaging system, which can utilize the version history and dependencies in a manner at least equivalent to what the historical metadata allows.

The other reason that we wish to push that historical metadata into the packaging system is so that it becomes accessible to a new class of application: the distro construction toolkit, which needs the dependency and versioning information to simplify the construction of self-consistent installable images. That leads us to an architectural diagram like

where we've suggested some internal structure to an updated packaging system. This separation shouldn't be that surprising: in the current system, lumake(1M) and most of LiveUpgrade as well as zoneadm(1M) for Zones are both performing image operations on either side of any packaging operations they might invoke. Designing a packaging system that makes the constructions of distributions and their installers substantially simpler will require such an API layer.

There are some open questions that come to mind:

• How much variation is there in the initial preparation an installer offers for the image location? Is space management a policy owned exclusively by the installer, or should the image operations layer have the ability to influence that allocation?
• Is there a marshallable image description, beyond the package dependencies? Is there a marshallable image format, or are we restricted to end-use formats, like ISO images or archived zones?
• What OS services should we consider modifying, beyond taking advantage of ZFS's current capabilities?

If you're interested in these questions, or the potential architecture of a packaging system, we'll be discussing these topics further in the Install community group.

[ T: OpenSolaris Solaris Indiana pkg ]

Observations on packaging

Over the past few months, a bunch of us have been exploring various options for packaging. (Actually, I suppose I've been pursuing this on a part-time basis for a year now, but it's only recently that it's made it to the top of the stack.) I've looked at a bunch of packaging systems, ported a few to OpenSolaris, run a bunch on their native platforms, and read a slew of manual pages, FAQs, blogs, and the like. In parallel with those efforts, Bart, Sanjay, and others have been analyzing the complexity around patching in the Solaris 10 patch stream, and improving the toolset to address the forms of risk that the larger features in the update releases presented.

In the course of those investigations, we've come up with a number of different approaches to understanding requirements and possibilities; I'll probably write those out more fully in a proper design document, but I thought it would be helpful to outline some of those constraints here. For instance, one way to look at how we might improve packaging is to separate the list of "design inputs" for a packaging system into "old" and "new".

When I make a list of this kind, I know it's bound to offend. (I already know it's more scattershot than the argument we'll present in a design document.) Feel free to send me the important inputs I've omitted, as I have a few follow-up posts on requirements—lower level, more specific intentions—and architectural thoughts where I can cover any issues not mentioned here.

"Old" inputs

The "old" inputs are those that are derived from current parts of the feature set around software packaging and installation. Many of these inputs will really be satisfied by the new installer work ("Caiman") Dave's leading, but the capabilities of the packaging system will change some from difficult and fragile to straightforward and robust. With effort, some might even achieve some elegance.

1. Hands off. For a long time, via the JumpStart facility, it's been easy to provision systems via scripted installed. This technology is still sound, particularly when we consider that the Flash archives allow a mix of image-based provisioning with the customizations JumpStart offered.
2. Virtualized systems. As long or longer, we've supported the notion of diskless systems, where the installed image is shared out, in portions, between multiple running environments. Zones was a direct evolutionary successor of the diskless environments, and shows that this approach can lead to some very economical deployment models. Lower-level virtualization methods can also benefit from the administrative precision that comes out of sharing known portions of the installation image.
3. Availability and liveness. With Live Upgrade, it's been possible for a while now to update a second image on a running system while the first image is active—a command to switch images and a reboot is all that's required to be running the new software. This approach requires too much feature-specific knowledge at present, but provides a very safe approach to installing an upgraded kernel or larger software stack, as reverting to the previous version is just a matter of switching back to the previous image. So, a package installation option that doesn't ruin a working system is a must.
4. Change control. In principle, with the current set of package and patch operations, it is possible to create a very specific configuration—that may never have been tested at any other site, mind you—from the issued stream of updates. From a service perspective, the variety of possible configuration combinations is too broad, but the ability to precisely know your configuration and make it a fixed point remains important.
5. Migration and compatibility. There's a very large library of software packaged in the System V format that will run on most OpenSolaris-based systems. Providing well-defined compatibility and migration paths is an obvious constraint, given the goals of the development process.
6. Networked behaviour. Believe it or not, the current installer and packaging support a large set of network-based operations—you can install specific packages and systems over the network, if you know where everything is. That is, the current components need to be assembled into some locally relevant architecture by each site to be useful—any replacement needs to make this assembly unnecessary, potentially via a number of mechanisms, but definitely via a list of known (trusted?) public package repositories.

"New" inputs

But, as you might expect, the efforts around Approachability, Modernization, and Indiana have brought to light some new qualities a packaging system must possess.

1. Safety. One of the real complications from virtualized systems, at least in current packaging, is that a developer has to understand each of the image types his or her package might reach, and make sure that the set of pre- and post-install scripts, class actions scripts, and the like are correct for each of these cases. When that doesn't happen, the package is at a minimum incorrectly installed; in certain real cases, this class of failures compromises other data on the system. Restrictions in this space, particularly during package operations on an inert image, seem like a promising trade-off to achieve greater safety.
2. Developer burden. Current packaging requires the developer provide a description of the package, its dependencies, and contents across a minimum of three files, in addition to knowing a collection of rules and guidelines around files, directories, and package boundaries. Most of these conventions should be encoded and enforced by the package creation and installation tools themselves, and it should be possible to construct a package with only a single file—or from an existing package or archive.
3. smf(5) aware. OpenSolaris-based systems have an understanding of services and service interdependencies, derived from the Service Management Facility's service descriptions. smf(5) service management during packaging operations is awkward under current packaging and very much needs to be improved but, more importantly, the service graph provides a rich set of relationships among larger components that should lead to a better understanding of the rules around consistent system assembly.
4. Minimization. For a while, the current system has had some very coarse package "blocks", from which one could construct a system—the various metaclusters. These, with the exception of the minimally required metacluster and the entire metacluster, split the system along boundaries of interest only to workstation installs. Any suitable packaging system must provide query and image management tools to make image construction more flexible and less error-prone (and eliminate the need for things like a "developer" metacluster, for that matter).

   It's also pretty clear that the package boundaries aren't optimized in any fashion, as evidenced by the differing rates of change of the binaries they currently enclose—in the form of the issued patches against Solaris 10.

5. Multiple streams of change, of a single kind. Although we noted the continued need to control change above, it's also important to be able to subscribe to a stream of change consistent with one's expectations around system stability. The package system needs to allow one to bind to one or more streams of change, and limit how the interfaces the aggregate binaries from those streams evolve. That is, it should be possible to subscribe to a stream of only the most important updates, like security and data corruption fixes, or to the development stream, so that one's system changes in a way consistent with one's expectations.

   Conversely, the tradeoff between complexity and space optimization in current patches—which introduce a separate and difficult-to-calculate dependency graph and distinct namespace entries for each platform, among other issues—has slid much too far, given the increase in system richness and the increases in disk space and bandwidth. There seems to be little long-term benefit in preserving the current patch mechanism, particularly since Sun never offered it in a form useful outside of Sun's own software.

6. ZFS aligned. ZFS offers the administrator access to so many sophisticated options around deployment and data management that it would be foolish for a packaging system to not explore how to take advantage of them—zfs snapshot and zfs clone are the most obvious capabilities that allow the packaging system to limit the risk of its modifications to an image, without excessive consumption of storage space or system time.
7. Prevent bad stuff early. Another classic OpenSolaris-style requirement is that the set of available packages be known to be self-consistent and correct at all times, to spare the propagation of incomplete configurations. In a packaging system, this input expands on our intent to reduce the developer burden to assist the developer in writing as correct a package as possible, and to enable the repository operator to block incomplete packages from being made available to clients. There's a rich set of data here, as we noted for smf(5) service descriptions above.
8. Friendly user deployment. Direct from Indiana, but sensible and apparent to all is that packaging systems have advanced to a point where the usability of the client is expected, and not an innovation. I haven't got the complete timeline of packaging developments—the literature survey continues—but it's clear that Debian's apt system marks the current expectations about the client-side capability and ease-of-use.

In the course of the Indiana discussions, Bryan raised one point, which I'll paraphrase as "it's not an idea until there's code". That's a sound position, which I also happen to hold—Danek and I (and Bart and Daniel, I hope) and have been quietly prototyping for a little while now, to see if we had a handle on this collection of inputs. I'd like to give a bit more background, in the form of requirements and assertions, in an upcoming post or two. Then we're hoping to start a project on opensolaris.org to take the idea all the way from notional fancy to functioning code.

[ T: OpenSolaris Solaris smf Indiana pkg ]