2008.05: More ways to get it

As I did for the preview releases, I'll collect links to mirror sites here. These will also get links on the various download pages out there.

Bart and I just finished updating the package repository with the new packages we'd received the past few weeks, and that means 2008.05 is out the door. (Thanks to everybody who tried the release candidate, filed a bug, shuffled a package, wrote or proofread a document, or just spent energy anticipating the bits.) You can get the ISO image, suitable for burning to a 700 MiB CD or immediate use in virtual environment, directly from the following locations:

• via HTTP from dlc.sun.com, and
• via HTTP from genunix.org,
• via HTTP from ftp.df.lth.se,
• via FTP from ftp.df.lth.se,
• via HTTP from mirrors.xservers.ro, or
• via BitTorrent.

Reading the logs, and talking with some of our mirror sites, we know we all served out a lot of downloads for the previews; if you're interested in being a mirror, please let me know. (2008.05 remains completely redistributable.) We're using a bigger download complex this time, but every mirror helps.

If you already downloaded and installed Preview 2, it's more complete, easier, and probably faster to update directly using image packaging: see the update guidelines. These instructions involve a small script to safely update a clone of your installed system, and then switch to that on a system reboot. (If you were running Preview 1, you should update that to Preview 2, and then go on to 2008.05.)

Update 1: My thanks to Tobias Lundquist, who's once again mirrored via FTP and HTTP (Internet 2) in Sweden.

Update 2: My thanks to Luca, who's put up an HTTP mirror in Romania.

These links are for the gzipped CD image, which contains the 12 "primary languages". It installs quite a bit faster, particularly on systems with slower CPUs. There is also an LZMA-compressed image, which has localization support for 42 languages, including those primary ones. It's available from dlc.sun.com, genunix.org, ftp.df.lth.se, mirrors.xservers.ro, and as a torrent. (Consult the language lists for specifics.)

[ T: OpenSolaris indiana ]

Live chat on #os200805

We're talking about OpenSolaris 2008.05 over on IRC, using the #os200805, from now until 2 p.m. Pacific Time. If you haven't got an IRC client, you can use freenode's Java applet, which worked great (until Danek kickbanned me). Enter a nickname, and then ask some questions.

Apparently the load from 60 000 users blew out the chat system, so we moved to IRC.

pkg(5): Reverse proxying your depot with Apache HTTPD

As part of the changes to get Developer Preview 2 ready, we decided to rejigger the HTTP handling on pkg.opensolaris.org so that we could have more options as more people attempt to use the early versions of image packaging. Previously, we ran pkg.depotd directly on port 80, in its read-only mode; now we use Apache HTTPD to listen on port 80, and use mod_proxy to proxy those incoming requests to a pkg.depotd instance listening on a separate port. With a couple of different approaches

Proxying and rewriting is one of those endlessly fun activities that somehow actually ends up being productive. Last time, proxying fun led Steve and I to fiddling around such that we ended up with proxy and rewrite patterns to enable the country portals for opensolaris.org.

If you want to share the top-level component of your URL space, you'll need to watch pkg(5) developments, as you have to map the list of operations one-by-one—and I know there are some new operations forthcoming. That would involve adding something like the following to a VirtualHost directive in your Apache configuration.

ProxyRequests On

Redirect /index.html http://pkg.opensolaris.org/status

ProxyPass /abandon http://pkg.opensolaris.org:10000/abandon
ProxyPass /add http://pkg.opensolaris.org:10000/add
ProxyPass /catalog http://pkg.opensolaris.org:10000/catalog
ProxyPass /close http://pkg.opensolaris.org:10000/close
ProxyPass /feed http://pkg.opensolaris.org:10000/feed
ProxyPass /file http://pkg.opensolaris.org:10000/file
ProxyPass /filelist http://pkg.opensolaris.org:10000/filelist
ProxyPass /manifest http://pkg.opensolaris.org:10000/manifest
ProxyPass /open http://pkg.opensolaris.org:10000/open
ProxyPass /search http://pkg.opensolaris.org:10000/search
ProxyPass /versions http://pkg.opensolaris.org:10000/versions

ProxyPass /css http://pkg.opensolaris.org:10000/css
ProxyPass /logo http://pkg.opensolaris.org:10000/logo
ProxyPass /icon http://pkg.opensolaris.org:10000/icon

ProxyPass /status http://pkg.opensolaris.org:10000/

Configuring your server in this fashion allows you to mix an image packaging server in with your other site content. You can easily deliver static content alongside your depot, for example.

If you don't mind pushing your package repository down one level in your URL space, then the above simplifies to

ProxyRequests On

ProxyPass /pkg/ http://pkg.opensolaris.org:10000/

(which should be a hint on how to create a repository farm under a single URL). To use the latter, you would use pkg(1)'s image-create subcommand

$ pkg image-create -F -a mypkgs.com=http://www.myserver.com/pkg /path/to/image

to connect your image to your reverse-proxied packaging depot.

In the two examples above, you should of course replace machine names like pkg.opensolaris.org and port numbers like 10000 with values appropriate to your own installation.

Happy proxied package serving!

Feel free to share your alternative configurations or approaches with other HTTP servers here, or on pkg-discuss@opensolaris.org.

[ T: OpenSolaris Solaris pkg httpd ]

Indiana: More ways to get to Preview 2

As I did for Preview 1, I'll collect links to mirror sites here, as well as on the project page.

So we just released the second Indiana preview ISO, which you can get from the following locations:

• via HTTP from dlc.sun.com, and
• via HTTP from genunix.org,
• via FTP from ftp.df.lth.se, or
• via HTTP from osol.acm.jhu.edu.
• via BitTorrent.

We had a lot of downloads for Preview 1; if you're interested in being a mirror, please let me know. (Indiana remains completely redistributable.) We're using a bigger download complex this time, but every mirror helps.

If you already downloaded and installed Preview 1, it's more complete, easier, and probably faster to update directly using image packaging: see the update guidelines on opensolaris.org. These instructions use ZFS and pkg(5) to safely update a clone of your installed system, and then switch to that on a system reboot.

Update 1: My thanks to Tobias Lundquist, who's once again mirrored via FTP (Internet 2) in Sweden.

Update 2: My thanks to trisk, who's once again put up an HTTP mirror (Internet 2) on the East Coast of the USA.

[ T: OpenSolaris Solaris indiana ]

Indiana: More ways to get it

So there are a few ways to get a copy of the Indiana preview ISO:

• via HTTP from dlc.sun.com,
• via HTTP from genunix.org,
• via HTTP from osol.acm.jhu.edu,
• via FTP from ftp.df.lth.se, or
• via BitTorrent.

Both our mirrors are in Northern California, so—since it's a completely redistributable set of software—if you're interested in being a mirror (even in California), please let me know. If you're a torrent fan, come join me and seed.

Update 1: My thanks to trisk, who's put up a fast mirror (Internet 2) on the East Coast of the USA.

Update 2: My thanks to Tobias Lundquist, who's put up an FTP mirror (Internet 2) in Sweden.

Icon Design by IconBuffet (from "Durango Office" and "Durango Research").

[ T: OpenSolaris Solaris indiana ]

pkg(5): a no scripting zone

In my previous two posts, we examined two packaging system options—installer-specific knowledge and integrated build system—that I believe present costs that exceed their benefits. Here, we will again examine a design choice from a negative perspective: package-associated scripting.

System V packaging is rich with scripting hooks; scripts named checkinstall, preinstall, postinstall, preremove, request, and the class action scripts. Each of these scripts can do anything they like. Scripting, even in a relatively primitive shell, is an open-ended program—opaque to the invoking framework. It's difficult to catch an incorrect script prior to package publication time, which blocks our intent to prevent propagation of bad package versions. With a more limited set of actions—potentially with that limit enforced or marked—a class of incompletely known resource handling mechanisms can be kept off the most conservative systems.

One goal we have is to preserve or improve the hands-off behaviour associated with package operations. Legacy packaging allows hands-off by imposing a series of tasks on the deploying administrator. The pkgask(1M) tool can enable the deployer to develop a response to the request script; coming up with an appropriate admin(4) can restrict the framework's built-in interactive queries. (Interaction with signed packages also requires the deployer to modify their pkgadd invocation.) Removing the scripting degree of freedom means that obstacles to hands-off behaviour come solely from an interactive installer or from interactive services acting during system startup.

There's some amusingly egregious violations of the hands-off principle across the space of known packages. Less fun is that these set a bad example for later package developers.

A particularly error-prone aspect of the scripting interface in packaging comes from the variety of contexts the package developer must understand (and test within). It is legitimate to install packages on live systems, in alternate filesystem hierarchies of the same or different architecture, and in whole-root and filesystem inheriting zones; in fact, you have multiple choices about how your package should install in a zone.

We can expect the proliferation of virtualized systems, via the various mechanisms like LDOMs and xVM, to keep all of these contexts relevant as degrees of sharing make virtualization even more appealing. Making sure that the package system operates safely in these shared contexts is critical—another of our goals.

Returning to the zones case, the example pseudo-script in pkginfo(4)—a series of nested shell if ; then blocks to navigate some of these contexts—is helpful, but misleading. There is much more variable state a package developer needs to consider to reach correctness. In fact, if you aren't required to rediscover or reinvent a set of resource-handling cases for each components your package delivers, it becomes substantially simpler to make the package and return to improving the software it contains. Reducing the set of steps reduces developer burdens associated with packaging.

Two particular resources stand out: device drivers and smf(5) services. Although some limited amount of awareness—or at least easily duplicated code—makes these resources somewhat well-behaved during package operations, there are still problems that scripting presents: the addition of new contexts, the provision of multiple genealogies of copied code, and the failure to discover an associated best practice for any particular kind of resource.

There are other resources, of course; as a start, you could duplicate our survey of the ON postinstall and class action scripts.

I believe the key counterargument supporting scripting is that the set of configuration patterns on Unix-like systems is large, and that the easiest means of upgrading each of these potential patterns is to allow a complete programming environment to the package developer. Probably true, but if we look at service and application configuration with respect to when a correct configuration state is required, the update step appears to separate into three classes:

1. Correct at system startup, no runtime context needed. These are the configuration settings that the various low-level boot components, the kernel, and the drivers need to bring the system to its running state. This class of configuration is generally limited to a specific set of resources, potentially established by a packaging system via corresponding resource-handling actions—or by an installer.

2. Correct at system startup, requiring runtime context. These are settings where the manipulating agent might be influenced by policy or require some form of interprocess communication to effect configuration changes. smf(5) is an example of the latter, and handles its configuration evolution via the manifest-import service. Manipulation of the various local name service tables, like passwd or the RBAC configuration is another example, since data about potential principals must be correct for a group of affected services. Since such configuration can be required on the system as a result of package operations, these resources must also be handled via packaging, or require the use of an appropriate installer.

3. Correct prior to service startup. Most service and application configuration falls into this class. It's not necessary, for instance, to bring a web server's configuration up to date if the service has no enabled instances. There seem to be a number of avenues for handling this kind of configuration: leaving it to the service or application, providing assistance via a configuration mechanism, or giving a hook where such updates can be made as needed. But the packaging system needn't provide this hook—there are a number of possible facilities, of varying suitability.

I should point out that David is making the smf(5) configuration update scenarios much more capable and precise with the Enhanced Profiles project. So, at least, a "configuration mechanism with assistance" is likely to be present soon.

Since the first and second classes and how their configuration manipulations vary in the various operating contexts are generally known, elimination of the third class makes precise, no-scripting packages a viable design choice.

That's a long series of arguments in favour of a scripting-free package system. It would be reasonable to ask: "can you actually do it?" So, as a check on our prototype, we used the branded zone capability to let us create a pkg(5)-based whole root zone. Here's a transcript

# zonecfg -z pkg_test
pkg_test: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:pkg_test> create -t SUNWipkg
zonecfg:pkg_test> set zonepath=/export/pkg_test
zonecfg:pkg_test> commit
zonecfg:pkg_test> ^D
# zoneadm -z pkg_test install
Preparing image
Retrieving catalog
Installing SUNWcs SUNWesu SUNWadmr SUNWts SUNWipkg
Setting up SMF profile links
Copying SMF seed repository
Done (115s)

There's dependency following, but no constraint handling; there's no filtering or snapshotting, but also none of the obvious performance optimizations has been implemented (for our 211MB resultant image). But the main point is: it works—installs, boots, upgrades, and still boots—with no scripting. Time for a project proposal.

[ T: OpenSolaris Solaris zone Indiana pkg ]

pkg(5): Leaving the build system "out"

I've been busy the past weeks with school transitions and with getting the community defect tracking system requirements into a shape where we can start evaluating candidates.

Identifying the boundaries of a system during the design phase affects feasibility critically, perhaps more than any other choice. This choice reduces to "know what you're trying to create". As with my previous post, I'm going to describe something we're not doing—and explain why. I envision one more negative post, and then we'll get onto more positive expressions.

So: this packaging system does not contain a build or compilation component in its core architecture. There are pragmatic reasons for this choice, as well as technical ones.

One way to look at a packaging system is, similar to how we saw it as the connecting layer between the installer and lower-level OS services, as a way to collect and organize the set of components, binary and otherwise, into an always "bootable" flow of change. That is, various groups of people are, using their systems, emitting binaries, documents, images, and so on, that other groups of people combine into releases (or atoms of a release, like an updated package). The packaging system has to assist the latter group in that combination, by providing visibility into the completeness of the outputs of the former groups. It's less clear that the latter group needs to have the ability to construct the objects, as long as they can assess that the environment the objects need to execute can be realized.

Just as an aside, we're treating as critical design input exactly who touches and how they touch a software component as it proceeds through development, localization, release engineering, sustaining, and so forth. These touches might be reduced as the development becomes more open, but at present, they're a useful constraint on any tendency to make the system overly rigid (or overly monolithic).

The OpenSolaris consolidations—and related open source outputs like OpenJDK or OpenOffice, among others—don't presently share a common build system. In fact, one of the real impacts of Sun's progression to an open software development culture is that we're moving from being the originator or primary maintainer of 90% of the software we deliver, to a much more modest percentage—let's say 30 – 50%. Forcing a unified build system upon all of these disparate products is asserting the need for a long series of difficult conversations over many months.

Instead, let's defer those discussions, and see if we can get the same benefits while only managing the end outputs of each of the participating (or aggregated) publishers. (A byproduct is that injecting received binaries into the system is the common case, rather than a strange or special workaround.) As we noted in the earlier post, we have goals about safety, developer burden, and stopping "bad stuff" as early as possible.

Static analysis can get most of the dependency cases correct. Binaries, whether they are user applications, libraries, or kernel modules, contain a significant amount of dependency information. It turns out that many scripting languages can be roughly interpreted to determine their module requirements. Similarly, Java class files and JAR files—and even smf(5) manifests—contain a great deal of information that lets us determine the self-consistency of a system.

Of course, a program can evade these dependencies: beyond use of dlopen(3C), it can implement its own linking or overlay mechanism, or simply be implemented in a language unknown to the packaging system. The point is we can drive out most of the inconsistencies that a purely manual dependency statement allows. In fact, we can warn a developer about the incompleteness of their dependency statements, potentially correcting them: adjusting version requests, inserting omissions, even asking about possibly superfluous dependency correctness.

That said, it might be that an ideal build system is lurking out there to be layered atop this system; we'll leave room for expressions, like stating a build dependency on a certain tool, like lex(1) and yacc(1), say, so that a build system (or systems, if folks can't close those difficult conversations I mentioned) can benefit from the metadata discovered about each component. (If you're interested in constructing such a system, we thought a little about requirements for the SFW consolidation.)

[ T: OpenSolaris Solaris Indiana pkg smf ]

pkg(5): No more installer magic

I thought I would continue probing some of the problems that present themselves to any packaging system that might follow the System V packaging. For the next few topics, I'll phrase the problem in terms of an outcome I believe we want to avoid. Here we discuss aspects of eliminating special metadata from installers.

If you're familiar with the packaging and installation components involved with Solaris Express, Developer Edition, or any of the OpenSolaris distributions that Sun produces, then it won't surprise you that there's a large amount of upgrade specific knowledge in the installation layer. For instance, to upgrade a package installed with Solaris 10 to its currently delivered version in a recent build, there's the set of files associated with the package—and then there's the package history, which collects additional information, like files no longer delivered, or the responsibility of another package. The presence of this file, and the absence of a true package upgrade operation in System V packaging, mean that any kind of upgrade requires some form of installer: the operating system installer for upgrading the entire installation; an application-specific installer for upgrading a group of related application packages.

There are a number of other metadata files trapped in the installation layer, but the most important are the metacluster files, which group the package clusters into large installation profiles, and the package clusters themselves, which group sets of packages along feature boundaries, approximately. It seems evident that these groupings are merely another kind of dependency, much like how a System V package can, naïvely state a dependency on any other System V package.

The System V dependencies also show that another critical piece of metadata—the versioning vector, or "arrow of time"—is encoded in the installer.

All of this information, if we are to allow packages to upgrade from one version to another in some linear fashion, needs to be pulled out of the installers and moved into some aspect of the packaging system. This change in responsibility means that the role of the installer becomes more precise: it must prepare a location for software installation, optionally lay down some initial, and possibly stale image, and collect any required configuration information. Subsequent upgrade operations are driven primarily by the packaging system, which can utilize the version history and dependencies in a manner at least equivalent to what the historical metadata allows.

The other reason that we wish to push that historical metadata into the packaging system is so that it becomes accessible to a new class of application: the distro construction toolkit, which needs the dependency and versioning information to simplify the construction of self-consistent installable images. That leads us to an architectural diagram like

where we've suggested some internal structure to an updated packaging system. This separation shouldn't be that surprising: in the current system, lumake(1M) and most of LiveUpgrade as well as zoneadm(1M) for Zones are both performing image operations on either side of any packaging operations they might invoke. Designing a packaging system that makes the constructions of distributions and their installers substantially simpler will require such an API layer.

There are some open questions that come to mind:

• How much variation is there in the initial preparation an installer offers for the image location? Is space management a policy owned exclusively by the installer, or should the image operations layer have the ability to influence that allocation?
• Is there a marshallable image description, beyond the package dependencies? Is there a marshallable image format, or are we restricted to end-use formats, like ISO images or archived zones?
• What OS services should we consider modifying, beyond taking advantage of ZFS's current capabilities?

If you're interested in these questions, or the potential architecture of a packaging system, we'll be discussing these topics further in the Install community group.

[ T: OpenSolaris Solaris Indiana pkg ]

Observations on packaging

Over the past few months, a bunch of us have been exploring various options for packaging. (Actually, I suppose I've been pursuing this on a part-time basis for a year now, but it's only recently that it's made it to the top of the stack.) I've looked at a bunch of packaging systems, ported a few to OpenSolaris, run a bunch on their native platforms, and read a slew of manual pages, FAQs, blogs, and the like. In parallel with those efforts, Bart, Sanjay, and others have been analyzing the complexity around patching in the Solaris 10 patch stream, and improving the toolset to address the forms of risk that the larger features in the update releases presented.

In the course of those investigations, we've come up with a number of different approaches to understanding requirements and possibilities; I'll probably write those out more fully in a proper design document, but I thought it would be helpful to outline some of those constraints here. For instance, one way to look at how we might improve packaging is to separate the list of "design inputs" for a packaging system into "old" and "new".

When I make a list of this kind, I know it's bound to offend. (I already know it's more scattershot than the argument we'll present in a design document.) Feel free to send me the important inputs I've omitted, as I have a few follow-up posts on requirements—lower level, more specific intentions—and architectural thoughts where I can cover any issues not mentioned here.

"Old" inputs

The "old" inputs are those that are derived from current parts of the feature set around software packaging and installation. Many of these inputs will really be satisfied by the new installer work ("Caiman") Dave's leading, but the capabilities of the packaging system will change some from difficult and fragile to straightforward and robust. With effort, some might even achieve some elegance.

1. Hands off. For a long time, via the JumpStart facility, it's been easy to provision systems via scripted installed. This technology is still sound, particularly when we consider that the Flash archives allow a mix of image-based provisioning with the customizations JumpStart offered.
2. Virtualized systems. As long or longer, we've supported the notion of diskless systems, where the installed image is shared out, in portions, between multiple running environments. Zones was a direct evolutionary successor of the diskless environments, and shows that this approach can lead to some very economical deployment models. Lower-level virtualization methods can also benefit from the administrative precision that comes out of sharing known portions of the installation image.
3. Availability and liveness. With Live Upgrade, it's been possible for a while now to update a second image on a running system while the first image is active—a command to switch images and a reboot is all that's required to be running the new software. This approach requires too much feature-specific knowledge at present, but provides a very safe approach to installing an upgraded kernel or larger software stack, as reverting to the previous version is just a matter of switching back to the previous image. So, a package installation option that doesn't ruin a working system is a must.
4. Change control. In principle, with the current set of package and patch operations, it is possible to create a very specific configuration—that may never have been tested at any other site, mind you—from the issued stream of updates. From a service perspective, the variety of possible configuration combinations is too broad, but the ability to precisely know your configuration and make it a fixed point remains important.
5. Migration and compatibility. There's a very large library of software packaged in the System V format that will run on most OpenSolaris-based systems. Providing well-defined compatibility and migration paths is an obvious constraint, given the goals of the development process.
6. Networked behaviour. Believe it or not, the current installer and packaging support a large set of network-based operations—you can install specific packages and systems over the network, if you know where everything is. That is, the current components need to be assembled into some locally relevant architecture by each site to be useful—any replacement needs to make this assembly unnecessary, potentially via a number of mechanisms, but definitely via a list of known (trusted?) public package repositories.

"New" inputs

But, as you might expect, the efforts around Approachability, Modernization, and Indiana have brought to light some new qualities a packaging system must possess.

1. Safety. One of the real complications from virtualized systems, at least in current packaging, is that a developer has to understand each of the image types his or her package might reach, and make sure that the set of pre- and post-install scripts, class actions scripts, and the like are correct for each of these cases. When that doesn't happen, the package is at a minimum incorrectly installed; in certain real cases, this class of failures compromises other data on the system. Restrictions in this space, particularly during package operations on an inert image, seem like a promising trade-off to achieve greater safety.
2. Developer burden. Current packaging requires the developer provide a description of the package, its dependencies, and contents across a minimum of three files, in addition to knowing a collection of rules and guidelines around files, directories, and package boundaries. Most of these conventions should be encoded and enforced by the package creation and installation tools themselves, and it should be possible to construct a package with only a single file—or from an existing package or archive.
3. smf(5) aware. OpenSolaris-based systems have an understanding of services and service interdependencies, derived from the Service Management Facility's service descriptions. smf(5) service management during packaging operations is awkward under current packaging and very much needs to be improved but, more importantly, the service graph provides a rich set of relationships among larger components that should lead to a better understanding of the rules around consistent system assembly.
4. Minimization. For a while, the current system has had some very coarse package "blocks", from which one could construct a system—the various metaclusters. These, with the exception of the minimally required metacluster and the entire metacluster, split the system along boundaries of interest only to workstation installs. Any suitable packaging system must provide query and image management tools to make image construction more flexible and less error-prone (and eliminate the need for things like a "developer" metacluster, for that matter).

   It's also pretty clear that the package boundaries aren't optimized in any fashion, as evidenced by the differing rates of change of the binaries they currently enclose—in the form of the issued patches against Solaris 10.

5. Multiple streams of change, of a single kind. Although we noted the continued need to control change above, it's also important to be able to subscribe to a stream of change consistent with one's expectations around system stability. The package system needs to allow one to bind to one or more streams of change, and limit how the interfaces the aggregate binaries from those streams evolve. That is, it should be possible to subscribe to a stream of only the most important updates, like security and data corruption fixes, or to the development stream, so that one's system changes in a way consistent with one's expectations.

   Conversely, the tradeoff between complexity and space optimization in current patches—which introduce a separate and difficult-to-calculate dependency graph and distinct namespace entries for each platform, among other issues—has slid much too far, given the increase in system richness and the increases in disk space and bandwidth. There seems to be little long-term benefit in preserving the current patch mechanism, particularly since Sun never offered it in a form useful outside of Sun's own software.

6. ZFS aligned. ZFS offers the administrator access to so many sophisticated options around deployment and data management that it would be foolish for a packaging system to not explore how to take advantage of them—zfs snapshot and zfs clone are the most obvious capabilities that allow the packaging system to limit the risk of its modifications to an image, without excessive consumption of storage space or system time.
7. Prevent bad stuff early. Another classic OpenSolaris-style requirement is that the set of available packages be known to be self-consistent and correct at all times, to spare the propagation of incomplete configurations. In a packaging system, this input expands on our intent to reduce the developer burden to assist the developer in writing as correct a package as possible, and to enable the repository operator to block incomplete packages from being made available to clients. There's a rich set of data here, as we noted for smf(5) service descriptions above.
8. Friendly user deployment. Direct from Indiana, but sensible and apparent to all is that packaging systems have advanced to a point where the usability of the client is expected, and not an innovation. I haven't got the complete timeline of packaging developments—the literature survey continues—but it's clear that Debian's apt system marks the current expectations about the client-side capability and ease-of-use.

In the course of the Indiana discussions, Bryan raised one point, which I'll paraphrase as "it's not an idea until there's code". That's a sound position, which I also happen to hold—Danek and I (and Bart and Daniel, I hope) and have been quietly prototyping for a little while now, to see if we had a handle on this collection of inputs. I'd like to give a bit more background, in the form of requirements and assertions, in an upcoming post or two. Then we're hoping to start a project on opensolaris.org to take the idea all the way from notional fancy to functioning code.

[ T: OpenSolaris Solaris smf Indiana pkg ]

OpenSolaris: Five updates conservative developers should make

It's been almost two-and-a-half years since Solaris 10 was released, and if we look at Nevada (via Developer Edition or one of the other distributions), we can see that many of the technologies introduced in S10 are becoming still more capable. At this point, even the most conservative software developer can assume that certain features are always present. So, for the conservative OpenSolaris application developer, here are the five low-risk, high-reward updates you should make to your application:

1. Provide x86 and SPARC versions. OpenSolaris has two primary instruction set targets, i386 and sparc. Each of these has both a 32- and a 64-bit variant. The metrics on Solaris 10 and SX:CE/DE downloads tell us that the Solaris volume is substantial on both targets so, for maximum uptake, you should attempt to offer software on both.

   On x86, you should consider delivering both 32- and 64-bit versions, if your application can take advantage of a 64-bit address space. But there is a large contingent of 32-bit only users, so don't stop delivering appropriate binaries prematurely.

   Of course, if you're writing at a hardware-independent level, like on a Java language platform, then you get x86 and SPARC (and presumably others) for free.

2. Make packages that deliver into sparse zones. The primary software delivery mechanism is still System V packages—but your software's already packaged properly, so that's not an issue. (Right?) With Solaris 10, the Zones feature offers a sparse variant that requires package support. Roughly, this support means that the package author shouldn't deliver into /usr and should add the three properties needed to the pkginfo file.

   There are some fairly serious Zones deployers out there; Joyent is probably the most public example, but there are plenty of corporate datacentres using Zones to their advantage. If you want your software run by them or their customers, providing a Zones-compatible package seems like the easiest way to get it into their hands and onto their Zones.

3. Replace your init.d scripts with smf(5) manifests. The Service Management Facility (smf(5)) provides a collection of capabilities that make service administration easier, while also reducing the development burden on application authors. Converting your service from the rc*.d script to a service description and methods means that administrators get automatic restart (and higher service availability), an easy on/off switch, and a place to make site-specific annotations (using the various template properties). There's a free comptetive advantage here, if your service runs under smf(5) and a rival's doesn't.

   Of course, you can do more: placing key configuration values in the service repository means that various administrative utilities can be taught to make manipulating your application's feature set easy to the deploying administrator. But that won't happen without an initial service conversion.

   (Once you write a manifest for your service, you'll also probably want to write a rights profile, so that administrative authority for your service and its instances can be easily delegated.)

4. Understand needed privileges. One of the more interesting features in Solaris 10 and later is the work Casper did to split out the absolute privilege owned by root into a specific collection of privileges. That means that you can take away a process's ability to fork or exec, change file ownership, or manipulate or utilize various subsystems of the operating system. If your application runs with the minimal set of privileges it needs to function, then the set of actions a hypothetical exploit against your application can invoke becomes limited, which reduces the impact of an intrusion. You can reduce your privileges via the smf(5) manifest you wrote for #3, via the role-based access control (RBAC) configuration, or via the privileges API.

5. Don't unnecessarily ship duplicate components. The various OpenSolaris distributions include a lot of software; most of these offer one or more update mechanisms for the components they include. Whether or not you prefer minimal patches to wholesale package replacements, if you ship a duplicate component, it's your responsiblity to update it if a defect or security hole is found. Sometimes you have to ship a component—the distros don't update it often enough—but private libraries (or private copies of the Java runtime) have a collection of costs, many of which are imposed on your customer.

For specific kinds of software, there's more to investigate. Language interpreters and byte-code virtual machines (and probably complex daemons) should have DTrace providers. Network device drivers should write to the latest version of the generic LAN device (gld) interface. Archival programs should be ZFS-compatible—there's going to be a lot of data on ZFS. Daemons should investigate using libumem for memory allocation (and event ports in place of poll(2) or select(3C)). And so on.

There are OpenSolaris communities for each of these topics but, if you're having trouble getting started, I would suggest an email to opensolaris-code, that reads something like: "I have a [daemon/driver/app] that does [practical purpose/amazing feat/curious entertainment]. Are there any OpenSolaris features I can use to make it better?"

Looking forward to your mail.

Thanks to Dave for #5. Dave also confesses to being keen on #3.

[ T: OpenSolaris Solaris smf privilege RBAC zones ]

SFW: Integrating coreutils and which variants

Last night, I finished up another task, in an attempt to reduce my current multitasking factor: I integrated initial versions of coreutils and which from the GNU Project into the Freeware consolidation. As a lower priority task, it took longer than a more dedicated developer might have managed, but it's reasonably pleasing to look back:

• initial idea, in case form, proposed and repeatedly revised on companion-discuss,

• draft fast track circulated on sfwnv-discuss,

• fast tracks for /usr/gnu, coreutils, and which submitted as open PSARC fast track cases,

• preliminary, intermediate, and final code reviews on sfwnv-discuss, and,

• finally, an integration message to sfwnv-discuss.

There's still a bunch of process associated with SFW that requires redesign—legal review and Section 508 compliance, in particular—but I think, barring the latent intervals, this sequence was a reasonable consensus-driven open development experience.

If you look at other opensolaris.org mailing lists during June – November and February – April, you'll be able to verify that I was indeed working—just on other things, and not just surfing the Web...

Of course, now that I know that these commands will start to show up more widely when Build 67 is released, I can update my dotfiles, so I get the versions I prefer:

$ svn diff
Index: sh-functions
===================================================================
--- sh-functions        (revision 91)
+++ sh-functions        (working copy)
@@ -36,6 +36,10 @@
                        PATH=$HOME/bin:$HOME/bin/$(/usr/bin/uname -p):$PATH
                        MANPATH=$HOME/man:$MANPATH
                        ;;
+               gnu)    # PREPEND: Bundled GNU command variants
+                       PATH=/usr/gnu/bin:$PATH
+                       MANPATH=/usr/gnu/share/man:$PATH
+                       ;;
Index: bashrc
===================================================================
--- bashrc      (revision 91)
+++ bashrc      (working copy)
@@ -49,8 +49,12 @@
        path clear home sfw csw
 fi

-if hash gls > /dev/null 2>&1; then
-       alias ls="gls --color -CF"
+if [ -x /usr/gnu/bin/ls ]; then
+       alias ls="/usr/gnu/bin/ls --color -CF"
 fi

+if [ -x /usr/gnu/bin/which ]; then
+       alias which="/usr/gnu/bin/which"
+fi
+
 export CDPATH=$MACHINE_CDPATH_PRE:$HOME/projects:$HOME:$MACHINE_CDPATH_POST

If you're using a distribution that offers SUNWgnu-coreutils, SUNWgnu-which, and the other /usr/gnu packages, do share your feedback with the maintainers on sfwnv-discuss—or become one and pick your favourite package.

[ T: OpenSolaris Solaris SFW coreutils which ]

Bespoke services: network/rmi/registry

Gary and I were recently prototyping an application that uses Java RMI, and so I ended up searching around to see if anyone has done a service conversion for rmiregistry(1). (rmiregistry(1) is the daemon that lets RMI clients find the available remote objects being served by various virtual machines on a given system.) Turns out no one has (or no one's published it), which means it's time to rev up the convert-o-tron.

Since we're still developing our application and it's likely we'll change a definition or two, and since we need to restart the registry to cause the remote objects, we're going to make our prototype service restart automatically if we restart the registry. That means our prototype service has a dependency on network/rmi/registry with specific restart_on behaviour, meaning that its service description has a fragment like the following:

<!--
    As an RMI server application, we expect to be able to
    register our RMI classes with the registry server.
-->
<dependency
    name='rmi-registry'
    grouping='require_all'
    restart_on='restart'
    type='service'>
        <service_fmri value='svc:/network/rmi/registry' />
</dependency>

Inject that fragment into your various RMI servers' descriptions (or the equivalent property group into the repository) and you'll save a bit of time on application reinitializations.

So, if you're interested, please feel free to take a copy of network/rmi/registry; comments and corrections welcome.

[ T: OpenSolaris Solaris smf RMI ]

store.sun.com to Niagara/Solaris 10

James Dickens spied a new BluePrint on the planned redeployment of store.sun.com [PDF] to SunFire T2000 systems running Solaris 10. Beyond the tremendous reduction in occupied space and the around 90 percent estimated reduction in input power and output heat, the document describes the use of Solaris Containers to consolidate the middle tier servers, complete with invocations of zonecfg(1M) (for the zones the applications run inside of) and poolcfg(1M) (for the resource pools the zones sit upon).

Business applications are complex—maybe some smf(5) service conversions will appear in the next version to make the dependency and failure handling more precise.

[ T: Solaris zones pools ]

LISA05 Tuesday: device errors, iostat, and logging

One of the questions raised at Tuesday night's BoF was "why are some of the statistics that iostat -E displays result in a console message and some do not?" I was sitting in the back with a copy of Mike Kupfer's split ON source tree, and decided to have a look. iostat(1M) is a kstat reader, with some simple processing and formatted output. The output function is show_disk_errors() but requires understanding how iostat groups the disks and statistics in its implementation; the code that acquires the device statistics is located in cmd/stat/common/acquire_iodevs.c. Searching for error show that the critical function is acquire_iodev_errors(), and that two classes of kstats contribute to the error output: device_error and iopath_error.

The most direct way to see these statistics is to invoke kstat(1M) with these classes. On my laptop, the result for device_error is

$ kstat -p -c device_error
sderr:0:sd0,err:Device Not Ready        0
sderr:0:sd0,err:Hard Errors     0
sderr:0:sd0,err:Illegal Request 1
sderr:0:sd0,err:Media Error     0
sderr:0:sd0,err:No Device       0
sderr:0:sd0,err:Predictive Failure Analysis     0
sderr:0:sd0,err:Product UJ-832D         Revision
sderr:0:sd0,err:Recoverable     0
sderr:0:sd0,err:Revision        1.50
sderr:0:sd0,err:Serial No       
sderr:0:sd0,err:Size    0
sderr:0:sd0,err:Soft Errors     1
sderr:0:sd0,err:Transport Errors        0
sderr:0:sd0,err:Vendor  MATSHITA
sderr:0:sd0,err:class   device_error
sderr:0:sd0,err:crtime  76.139658104
sderr:0:sd0,err:snaptime        1857.960128997

(The laptop has no kstats of class iopath_error.)

We then look for the creation of the named kstat for each of these strings—invocations of kstat_create() to identify the structure member names associated with each. And then we can look for statements that involve those member names; this leads us to the various SDUPDATEERRSTATS() invocations throughout uts/common/io/scsi/targets/sd.c.

The discrepancy between updates and logging arises because the macro, SD_UPDATE_ERRSTATS(), which bumps the counters and the function which displays the error, sd_print_sense_msg(), are sometimes both invoked, and sometimes are not. I don't know the details of the SCSI error categories, but the decision to make some of these errors silent and some not appears arbitrary. So, unfortunately, the only answer today to determine "why is this messaged" is to look at the code. (Perhaps an 'sd' expert can offer an enlightening comment.)

If you're trying to anticipate and avoid potential failures, having random messages emitted arbitrarily isn't very helpful: that's why Mike and the FMA team developed the fault management architecture to have a framework in which errors are processed in a predictable fashion, resulting in the proper diagnosis of faults. Eric described one possible scenario involving disk errors, FMA, and ZFS a couple of weeks ago, but there's a smaller step that seems useful involving only sd and FMA: the error increments could be converted to error events, and the decision to issue a notice deferred until a series of errors is diagnosed into a fault except, I suppose, if the error can be immediately diagnosed as a critical fault. Taking this step would result in consistent reporting for all disk consumers, including less sophisticated consumers than ZFS, like older filesystems and raw disk accessors.

In a software engineering sense, the FMA approach, where error issuance and diagnosis are separated, is much more sound: at the initial driver software composition, the field experience with the hardware device is typically limited. Over time, the actual impact of errors on system practice becomes better known, and the diagnosis and the actions associated with it can be refined. fmd(1M) can handle on-the-fly module updates gracefully, and also deal with overlapping event flows so that both primitive and ZFS-specific fault handling policies can be implemented, depending on the use of a particular device.

Now, the community that's discussing technical issues and directions for fault management is aptly called the Fault Management community; if you are interested in how this work is going to proceed, and ways to contribute, I suggest joining it.

[ T: Solaris OpenSolaris LISA05 FMA zfs iostat kstat ]

smf(5): Stepping through an rc.d conversion

Over on mediacast.sun.com, I see Bob Netherton has posted a nice tutorial from Solaris Boot Camp, entitled "Migrating a legacy RC service". The presentation covers the hiccoughs you might run into during your first conversion, and its step-by-step approach is very soothing.

[ T: OpenSolaris Solaris smf ]