Non-Root Builds

In ancient times,¹ it was necessary to be root in order to build ON (the "operating system and networking" part of Solaris). Every once in a while, this would have tragic consequences, such as vaporizing an important build server with one stroke of "rm -rf /" invoked by setting $ROOT to "/". But mostly it served as an annoyance factor: you couldn't delete your own workspaces (because they'd have UID 0 files in them), you had to give out the keys to the farm to all your build server users, you couldn't necessarily figure out who was using the machine, and so on.

As part of Solaris 10, I removed this restriction. Much remains to be done, as some still needlessly cling to the root-to-build lifestyle, but many use this new feature. This is the story of how it was done.

The old build system was centered around the concept of the "proto area." This is pointed to by $ROOT, and is typically set to "proto/root_$MACH/" in the current workspace.

The proto area is intended to be almost an exact image of the system being constructed. Each "make install" rule copies its resulting binaries and configuration files to the final location they'd have on the installed system, treating "$ROOT" as the root directory. There's thus "$ROOT/etc", "$ROOT/lib", "$ROOT/usr/bin", and so on.

The "almost" part is that we also deliver a few internal test objects, internal libraries, and other detritus to the proto area, so it's more like a superset of the final system ... minus the things that come from other consolidations.

This proto area is used for multiple purposes:

• It is used during the build itself, with "-I" (include path) and "-L" (library path) set to point into this area. This way, we can build our libraries and executables using the bits in the workspace rather than whatever (usually older) bits happen to be on the build server.

• It is used to simplify the construction of System V packages, which are used to deliver the software. The pathnames used in the prototype(4) files can be set to the actual installed location, rather than having to use "=" everywhere.

• It is used to construct cpio archives (called "BFU images") that can be quickly installed on top of test systems. See the bfu shell script for details.

• Some developers copy things directly out of $ROOT onto a running system during development, and having the objects in the right part of the tree makes that simpler. (Caution: don't try this at home, kids. We're what you'd call "experts.")

• Finally, many developers expect to be able to do "cd $ROOT" and see the files that would be installed on the system before actually doing an install, in order to verify that their changes worked as intended.

In order to do all that, though, the developer needs to be root when he² invokes "make install" in order to create the usual root-owned files, set-uid binaries, and other things that will appear on the system.

Various proposals were considered when figuring out how to eliminate the use of root during the build process. A few of those were:

• Use RBAC so that ordinary developers can invoke "make," and then the installer itself (blessed by the RBAC profile) automatically runs as root. This fails as a complete solution because there's still a UID root process running, and the user who invoked it can still torch the build system with an errant directory path. It's also complicated because the build server administrator will need to add explicit permissions for these users

• Create a new filesystem that ordinary developer can mount over $ROOT, and that has no restrictions on setting UIDs and the like. This solves the RBAC problem, but creates two new ones: ordinary users could possibly use this to side-step build system security, and it's an awfully complicated solution to a simple problem.

• Create special install tools that all Makefiles must use, and that record file owner/group information in a separate hidden file. Then run a new tool (as root) over that saved information and fix up the proto area to have the desired owners and groups. This is potentially troublesome for build performance, and still doesn't entirely get rid of root.

An additional, hidden design flaw exists in the old build system and all of the above answers. When the developer installs a file in the proto area, he must make sure that it gets installed with the right UID, GID, and mode bits, usually by manipulating Makefile variables. But he also must add that very same information into the packaging prototype files. If the two are out of sync, then needless (and sometimes hard to understand) build failures result. As a basic rule, copying the same information to two different places is just bad design.

Instead, I chose to go a different route:

1. When the build process is run as root, we will install into the proto area as we always have in the past, with all the right user IDs and other bits. This is for those "legacy" developers who insist on running the build as root and getting the same results.

2. When it's run as a non-root UID, things would "mostly" still be installed with the right modes (permissions), but with the user and group set to the developer's default.

3. To create BFU (cpio) archives, I wrote a new utility called cpiotranslate. This takes a cpio data stream on standard input and copies it to standard output after modifying the owner, group, and mode in the archive headers according to the data read from the packaging database.

4. Because a few developers like to use the new tools to build very old workspaces, I included a new option "o" in NIGHTLY_OPTIONS that restores the old build behavior by disabling cpiotranslate. This shouldn't be used on workspaces that are based on Solaris 10 or newer.

Packaging, of course, never cared about the owner, group, or mode of the files in the proto area, because it has its own internal information for that.

This solution means that the files in the proto area no longer have the right owner/group set on them. Developers who insist on seeing this will have to take an additional step after building. For this purpose, I modified protocmp, which is ordinarily used by the build process to validate the file modes in the proto area, so that it can restore the correct owner/group/mode to proto area files as recorded in the packaging database. To invoke this functionality (you must be root), run protocmp -GU -d $SRC/pkgdefs $ROOT. This uses essentially the same logic as cpiotranslate, just applied to files rather than a cpio stream.

The cpiotranslate utility plugs into the existing mkbfu and makebfu pair. Neither of these is really intended for ordinary use. Instead, nightly normally invokes them when needed during the build process. The upper-half makebfu script sets up some environment variables and then invokes mkbfu like this:

        mkbfu   -f "cpiotranslate -e $pkg/$exc $bpkgargs $pkg" \
                $zflag $ROOT $CPIODIR
else
        mkbfu $zflag $ROOT $CPIODIR

The first case is for non-root builds, and the second is for traditional root builds. The "-f" argument specifies a "filter" that is inserted into the usual "find | cpio | gzip" string when building the archives.

An interesting side-effect of the non-root build process is that several directories and files had to be made owner-readable and owner-writable. We'd previously relied on root's ability to access directories and files regardless of mode in order to build the proto area; e.g., creating a directory with mode 0555 and then copying in a file. This doesn't work for ordinary users. Fortunately, setting mode bits 0300 is essentially a no-op for root. The ARC documentation to describe these changes included:

Name				Old	New	Note	Stability
----------------------------	----	----	----	------------------
usr/bin/ct			4111	4511	A	Stable
usr/bin/cu			4111	4511	A	Stable
usr/bin/uucp			4111	4511	A	Standard (SCD-2.0)
usr/bin/uuglist			4111	4511	A	Stable
usr/bin/uuname			4111	4511	A	Stable
usr/bin/uustat			4111	4511	A	Standard (SCD-2.0)
usr/bin/uux			4111	4511	A	Standard (SCD-2.0)
usr/lib/pt_chmod		4111	4511	A	Project Private
usr/lib/spell			555	755	B	Stable
usr/lib/uucp/bnuconvert		110	510	A	Project Private
usr/lib/uucp/remote.unknown	4111	4511	A	Project Private
usr/lib/uucp/uucheck		110	510	A	Stable
usr/lib/uucp/uucico		4111	4511	A	Stable
usr/lib/uucp/uucleanup		110	510	A	Stable
usr/lib/uucp/uusched		4111	4511	A	Stable
usr/lib/uucp/uuxqt		4111	4511	A	Stable
usr/sadm/install/scripts	555	755	B	Sun Private
var/spool/cron/crontabs/adm	400	600	C	Evolving
var/spool/cron/crontabs/root	400	600	C	Evolving
var/spool/cron/crontabs/sys	400	600	C	Evolving

The changes break down into three categories, indicated by the "Note"
field above.  These are:

  A - executables to which the "read by owner" 0400 permission has
      been added.  The reason this is done is so that the package
      builds, which read these files from the proto area, will work
      when the build process isn't running as root.  The current
      package build process assumes that the invoker is root, so that
      even 'unreadable' files are readable.  Since the contents of the
      executables are already visible in packages and patches, it's
      assumed that allowing the file owner to read the file contents
      is not a security risk.

  B - directories to which the "write by owner" 0200 permission has
      been added.  The build process assumes that the invoker is root,
      and thus installing files into unwritable directories is
      possible.  In both cases above, the actual directory owner is
      root, so adding write permission (which root already enjoys)
      changes nothing about the overall access model.

  C - files to which the "write by owner" 0200 permission has been
      added.  In most cases, the build process does not require that
      files in the proto area be writable.  However, in the case of
      these three files, the build process incrementally manipulates
      these files (">> file") during the build.  It thus relies on
      root's ability to write to unwritable files, and won't work when
      the invoker is not root.  Since these files are always owned by
      root, this adds no new permissions, and no bypass of
      cron.{allow,deny} is possible.

The one slightly controversial part of this change is 'C'.  This
change means that a foolhardy administrator running as 'root' who runs
'vi' on one of the /var/spool/cron/crontabs files (instead of doing
"crontab -e" as documented) will see no error when doing ":w", where
he once would get a "read only file" warning.  Of course, that same
administrator running as root could always have typed ":w!" to
overwrite the file, so the additional exposure seems slight.

One drawback to the whole non-root plan is that, as many people start using the non-root build system, root builds (which still rely on the duplicated information for file ownership) will start rotting on the vine. They won't be routinely tested, and the (now unnecessary) Makefile goop that makes them work will likely be broken by accident by people who test their changes with only a non-root build. This effect is minimized by continuing to set the mode in the proto area, even though it's not needed, but can't easily be eliminated.

The potential for silent discrepancies means that the option to build as root does need to be removed from the system. Someday.

No good deed should go unpunished. As expected, there was a fair amount of fallout from these changes. In order to test, I did root and non-root builds with various settings of "nightly" flags to produce likely build scenarios. I caught many potential problems and fixed them, but I didn't catch all of them.

The changes broke the obscure "realmode" build, as documented in 4801654. The "realmode" builds were used to build special x86 boot bits.

The problem reported in 4864878 was due to the stderr handling in the BFU build scripts. It was at least in part an existing problem, but non-root build made it worse.

And 4858563, which broke old-style root builds, was a foolish typo on my part.

As a possible warning to others, once you touch one of these sorts of things, other folks seem to assume you "own" it and can debug their problems.

1. This refers to Solaris 9 and before.

2. I use the standard English masculine gender pronoun to refer to persons of unknown gender because all of the alternatives are just plain awkward. Much like this footnote. Unless otherwise noted, no actual chauvinism is intended. (Thanks, Mr. Knuth!)

Posted at 11:26AM Jun 14, 2005 by carlson in Solaris  |  Comments[3]