The Random Bit Bucket: Moinak Ghosh's Weblog

I have been working quite a bit with the OpenSolaris LiveCD project and have now reached a milestone where I have a proper x86 LiveCD that boots up the 32 bit kernel. I am now fine-tuning nd ironing out the rough edges before I can put up the ISO image and tools for download. It had been quite challenging to reach this point and I intend to document all the details fully.

Before I get around to doing that, here is a rough sketch of what happens behind the scenes when my OpenSolaris LiveCD boots:

1. After the PC is powered on and the BIOS has completed it's work it loads the code located in the first sector of the boot device and passes control to it. In case of the hard disk this is known as the Master Boot Record and in case of a CD it is the Boot Sector. The x86 Bios uses a predefined order when searching for bootable devices with removable media typically being the first. So in my case the Bios loads the first sector from the CD and passes control to the code.

2. The boot sector simply passes control to another larger program called the stage2 bootloader. Since OpenSolaris uses GRUB, I am using Grub's stage2_eltorito bootloader that is used to boot off cdroms and knows how to handle the High Sierra/iso9660 filesystem. This stage2 bootloader then loads and parses the /boot/grub/menu.lst file and displays the boot choices to the user.

3. The menu entries in Grub when booting off a CD just specify the kernel to load and the initrd or the boot archive that contains the given kernel and basic necessary kernel modules. This kernel is expected to comply with Grub's multiboot specification. This is true of the Linux kernel. But in Solaris the multiboot implementation has been kept in a separate standalone program called /platform/i86pc/multiboot. This is essentially the secondary bootloader for Solaris. Once a Solaris menu entry is selected Grub loads the multiboot program and the boot archive into memory. The boot archive is a gzip-compressed file that contains a UFS filesystem image containing the barebones kernel/modules/drivers binaries. Grub then passes control to multiboot.

4. The multiboot program is Solaris's secondary bootloader. Among other things it queries the boot args passed to Grub and initializes the memory allocator. The boot archive that was uncompressed and copied into RAM by Grub is a minimal Ramdisk having a UFS filesystem. Multiboot mounts this as the root filesystem and reads the kernel and krtld and performs a jump to the kernel's main. The multiboot source code can be viewed at:     http://cvs.opensolaris.org/source/xref/usr/src/psm/stand/boot/i386/common/multiboot.c  Before loading the kernel multiboot also reads the /boot/solaris/bootenv.rc file. On SPARC  Solaris depends on the OBP (Openboot Prom) for a lot of configuration and device infomation. Since there is no OBP on x86, an OBP emulation is used and the configuration parameters are stored in /boot/solaris/bootenv.rc. One of the parameters "bootpath" holds the physical device node that contains the root filesystem. This is ultimately mounted by the kernel as "/". If the bootpath parameter is absent then the kernel assumes that the ramdisk created from the boot archive is the rootfs and mounts it under "/". This is the crucial bit of detail that enables us to ultimately have a LiveCD.

5. Booting the kernel from the ramdisk is used in Solaris's install framework to load a basic filesystem into RAM and boot the OS off that. This is called the install miniroot that also  contains all the binaries and utilities required by the installer. This enables the install  process to eject CDROMS as and when their contents have been installed. This approach is also used to boot off the Failsafe miniroot and perform system recovery if needed. This approach also enables us to prepare a LiveCD. However the setup required for a LiveCD is  slightly different than the Install/Failsafe miniroot. On a LiveCD we try to minimize the space used by the ramdisk and try to use as many of the binaries directly from the CD as possible. The approach is to mount the cdrom onto some directory and loopback mount other directories in the cd onto appropriate points in the minimal rootfs. This enables all the utilities and apps to run directly off the CD.

6. In order to achieve the objective mentioned above the CD should have a proper rootfs layout and should be mounted as early as possible in the startup process. In OpenSolaris the CD has to be mounted in one of the initial services that SMF loads. In order to identify the appropriate service I needed to get a complete chart of service startup order in SMF. While discussing this with a co-worker I was pointed to this excellent resource on Dan Price's Blog: More on Bootchart for Solaris Looking at the bootchart one can clearly see that net-physical runs first and then fs-root runs that checks the rootfs and remounts it since the rootfs would have been mounted read-only initialy by the kernel. It also mounts an optimized platform-specific libc over /usr/lib/libc.so.1. So fs-root should be the appropriate place to mount the CDROM and perform the other lofs mounts. But the problem here is that though /devices is available at this point, devfsadm is only executed  by devices-local that runs after net-loopback: fs-root->net-loopback->devices-local. Without devfsadm being run, we do not have the /dev symbolic links in order to identify the /dev/dsk/*/usr/sbin/fstyp entries. After pondering this for sometime I decided for now to explicitly remount root as rw at the beginning of fs-root and execute devfsadm. Of course I added check to ensure that I am running in the LiveCD environment before doing this. Subsequently I added code to loop through the /dev/dsk/* entries and run on each entry to identify the fs type. Known fs types in Solaris include ufs, hsfs, udfs, pcfs (fat32) etc. if it is of type hsfs then it mounts the CD under /mnt/.cdrom and checks if this is the LiveCD. If so then it performs the other lofs mounts of /usr, /sbin and /var/svc/manifest. In future I'd also need to add an lofs mount for /opt. When devices-local is executed, it runs devfsadm again to link devices that may have been missed. I realize that this is an inefficient solution and am working on another approach to directly identify CD devices in /devices. As a side-effect the code automatically mounts all the recognizable partitions on the hard disk as well. Something that happens when you boot one of the many Linux LiveCDs.

7. After the initial jugglery it is pretty much the standard service startup process that follows. Apart from the bootenv.rc entries another crucial element of booting from a CD involves reconfiguration. The "/reconfigure" file must be present in the boot archive to enable reconfiguration every time the CD is booted to ensure that all devices are setup properly. I was faced with some trouble in the service startup due to a missing service manifest called "sysidtool.xml". There are a bunch of network services that depend n this. Sysidtool is part of the Solaris installer and is not included in OpenSolaris as the install framework is still closed. So I ended up writing a new services manifest called "sysconfigure" having two instances called "sysconfigure:net" and "sysconfigure:system" as is the case for sysidtool. I also changed the dependent services to depend on sysconfigure. Presently "sysconfigure:system" does nothing and "sysconfigure:net" brings up all network interfaces using "ifconfig -a plumb". I will be adding code here try DHCP on all the interfaces whose link is up (except loopback).

8. Apart from the above SMF also does a manifest-import to refresh the repository if it detects a reconfiguration reboot. However this delays the boot and is unnecessary since I am already providing a prebuilt /etc/svc/repository.db in the ramdisk. The simple solution is to add a check for the livecd environment at the beginning of the manifest-import service and do "exit 0" if this is the case.

9. After I was able to achieve full startup of all services, I got an issue with inetd. At first I  noticed that all the inetd services were failing to start with the error    "Property exec for method inetd_start of instance svc:/application/.../...:default is invalid" It took me a while before I could figure out what was wrong. All the inetd services have a method definition in their manifest that is named "inetd_start". Here is an example:

   <exec_method type="method"
            name="inetd_start"
            exec="/usr/lib/inet/in.chargend -s"
            timeout_seconds="0"
            delete="false">
         ...
          
   

   It appears that the command string given to the parameter exec="..." can contain variables and meta-characters to be expanded by the shell. In order to do the shell expansion inetd calls the libc function  "wordexp(3C)". A quick scan through the wordexp implementation shows that it either calls /usr/bin/ksh or /usr/xpg4/bin/sh with the parameters "-^EuN" and passes the string passed to it by the caller. The    shell is supposed to treat these parameters specially and just echo the expanded string. Ordinarily /usr/lib/libc.so.1 will call /usr/bin/ksh to perform the expansion unless it was compiled for xpg4 compliance in which case /usr/xpg4/bin/sh will be called. Now the problem is /usr/bin/ksh or ksh88 as  we all know is closed source and not even redistributable in binary form. So it is absent from OpenSolaris. The open-source ksh93 does not support these options/feature though it can be coaxed to perform word expansion only using a combination of other options. Initially I tried to do that using a wrapper shell script that checks for the "-^EuN" options and execs ksh93 with the appropriate options. However inetd started exiting without any error message when I did that. Finally I noted that /usr/xpg4/bin/sh has most of the features of ksh including, of course, word expansion. Thus the easiest solution is to copy it into /usr/bin as ksh.

    

10. Finally a note about swap is not out of place here. Since OpenSolaris is booting from a CD with only "/" mounted read/write on the ramdisk, there is no separate swap device configured. However the service fs-usr still adds physical swap and "/tmp" and "/var/run" are mounted on swap. In actuality anonymous pages are alocated dynamically for swap in the absence of a swap device.

There is still a bunch of work remaining to do. Some of the things I am doing at present include:

1. Optimising space usage on the ramdisk. The ramdisk needs to occupy as little space as possible containing only the stuff required to boot and perform startup process till it reaches the fs-user service at which point /usr is mounted.
2. Add some of the free drivers, incuding Casper's acpi stuff available for Solaris.
3. Optimize space usage on the CD as well. Whatever is present in the ramdisk/boot archive and is not mounted from the CD (/kernel, /platform, /etc, /lib) need not be duplicated on the CD as well. This will allow more space to add other software.
4. Fix some issues preventing a clean shutdown.
5. Add a dummy /etc/defaultrouter entry to prevent in.routed from starting. It will also be overridden if we are able to get a DHCP config.
6. Modify sysconfigure:net to attempt autoconfiguration via DHCP.
7. Fix a few more minor issues.

Some of the major work I have planned for the longer term is to have a larger boot archive for systems having 512MB ram or more to enable faster bootup and to add support for bringing up X11 and Gnome desktop. On systems with less memory (minimun 256MB ram required) it may not be possible to bring up Gnome or KDE. In such a scenario an alternative lightweight desktop like Windowmaker or Xfce can be used. I'd be nice to be able to detect the amount of memory on startup and tune the behavior automatically.