Thursday January 12, 2006
All
|
General
Thursday January 12, 2006 Background
One of the reasons we released BrandZ to the community before integrating with Solaris, was the hope that people would use the framework to create their own brands.
Less complex than creating a completely new brand is extending the lx brand to support other Linux distros. The simplest case is supporting a distro based on the same kernel/glibc version as our original RHEL/CentOS brand. To support such a distro, in theory we just need to create an installer for the new distro.
There are two distros that I have been asked about multiple times: Debian and Ubuntu. Both are based on the Debian package format, instead of RPM like the distros we currently support. I wanted to install Ubuntu, since its emphasis on getting wide distribution means it has a much smaller learning curve than Debian. Unfortunately (in this case), Ubuntu is also much more agressive about upgrading their core components. Debian still has an actively used version ('sarge') based on the kernel/glibc revs currently supported by BrandZ, while Ubuntu does not. So, Debian it was.
What follows is a description of the steps I went through to get Debian 'sarge' installed and running in a zone on Solaris. This isn't meant as a tutorial on how to perform a similar feat on your own system, since (as will become clear), it's really a pain in the neck. Rather, this is offered as a starting point (or possibly a cautionary tale) for anybody interested in writing a new install script for Debian or any other distro.
Zone configuration and layout
The first step, as always, is to configure the new Linux zone:
crunchyfrog# zonecfg -z debian
debian: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:debian> create -B lx
zonecfg:debian> set zonepath=/export/zones/debian
zonecfg:debian> add net
zonecfg:debian:net> set physical=bge0
zonecfg:debian:net> set address=10.8.28.84
zonecfg:debian:net> end
zonecfg:debian> commit
zonecfg:debian> exit
Next we do the install. Normally this process actually installs the Red Hat or CentOS bits into your zone. In this case, I wanted a blank slate on which to install the Debian software, so I used the 'tarball' install option to unpack an essentially empty tarball:
crunchyfrog# cd tmp/
crunchyfrog# mkdir usr
crunchyfrog# tar cf dummy.tar usr
crunchyfrog# zoneadm -z debian install -d /tmp/dummy.tar
With this done, I now had a Linux zone that was nominally in the 'installed' state, but it didn't actually have any software in it.
Installing Debian
When installing Red Hat or CentOS, we use the RPM software in the SFWrpm package. Debian uses a different packaging format, which we don't have any tool for manipulating. Fortunately, there is another OpenSolaris project which is working on creating a new Solaris distro built around this packaging format: Nexenta. I grabbed the version of dpkg they had built for Solaris, and I was ready to start installing.
Since I already had a sarge 'netinstall' CD handy, that's the medium I decided to install from. It can downloaded from the debian site here: debian-31r1a-i386-netinst.iso.
Getting to single-user mode
Through a little trial and error, I settled on these as the minimal set of packages needed to boot a zone that is capable of further self-installation:
libc6, libacl1, libattr1, dpkg, dselect, tar, grep, perl, sed,
find, mawk, gzip, zlib1g, libuuid1, debianutils, coreutils, procps,
passwd, login, sysvinit, initscripts, sysv, util-linux, bsdutils,
mount, base-passwd, base-files, bash, hostname, libncurses5,
libpam0g, libpam-modules, libpam-runtime
It turns out that dpkg is much more insistent on running pre/post-install scripts than RPM. So much so that even an 'unpack' operation requires running a pre-install script. So, I decided to break the process into two parts: 'dpkg-deb --extract' and then do a real install once the zone was booted.
crunchyfrog# for i in `cat pkgs`; do dpkg-deb --extract $i /export/zones/debian/root/; done
Since this method ensures that the install scripts aren't run, we have to do some of their work by hand before the zone will boot. Again, this was done largely by trial and error. The RHEL / CentOS installer we ship as part of lx does this work in the setup_lx_env script invoked by the installer.
1. crunchyfrog# cd /export/zones/debian/root
crunchyfrog# cp usr/share/sysvinit/inittab etc/
crunchyfrog# cp usr/share/base-passwd/passwd.master etc/passwd
crunchyfrog# cp usr/share/base-passwd/group.master etc/group
crunchyfrog# cp usr/share/passwd/shells etc/shells
crunchyfrog# cat > etc/rcS.d/S20proc
#!/bin/bash
echo "Mounting /proc"
mount -n -t proc /proc /proc
^C
crunchyfrog# chmod +x etc/rcS.d/S20proc
crunchyfrog# cat /etc/fstab
none / ufs defaults 1 1
none /proc proc defaults 0 0
2. I also modified inittab to eliminate the gettys running on the virtual consoles (which Solaris doesn't support) and add one on the zone console.
[...]
1:2345:respawn:/sbin/getty 38400 console
#2:23:respawn:/sbin/getty 38400 tty2
#3:23:respawn:/sbin/getty 38400 tty3
#4:23:respawn:/sbin/getty 38400 tty4
#5:23:respawn:/sbin/getty 38400 tty5
#6:23:respawn:/sbin/getty 38400 tty6
[...]
3. Finally, there are a few links that need to be created because the zones framework expects the files to be in one place while Linux puts them somewhere else:
ln -s /bin/sh sbin/sh
ln -s /bin/su usr/bin/su
ln -s /bin/login usr/bin/login
We now have a zone that can boot to single user mode:
crunchyfrog# zoneadm -z debian boot -s
On the debian console we see:
[NOTICE: Zone booting up]
INIT: version 2.86 booting
/etc/init.d/rcS: line 27: /etc/default/rcS: No such file or directory
Press enter for maintenance
(or type Control-D to continue):
root@debian:~# uname -a
Linux debian 2.4.21 BrandX fake linux i686 GNU/Linux
Housekeeping
Now to install the rest of the software. I started by copying the list of already-installed packages into the zone, and by making the mounted CD image visible as well:
crunchyfrog# cp /tmp/pkgs /export/zones/debian/root/tmp/
crunchyfrog# mount -F lofs /mnt/ /export/zones/debian/root/mnt/
The first thing I did was ensure sanity in the dpkg database by reinstalling all of the packages I had extracted before. Most of the following dpkg operations spit out a number of warnings, basically reflecting the fact that the environment is still insane.
To make dpkg happy I had to create a few files.
debian# touch /var/lib/dpkg/status
debian# touch /var/lib/dpkg/available
I then installed two packages 'by hand':
debian# dpkg -i --force-depends /mnt/pool/main/d/dpkg/dpkg_1.10.28_i386.deb
debian# dpkg -i --force-depends /mnt/pool/main/g/glibc/libc6_2.3.2.ds1-22_i386.deb
As part of its install process, libc interactively sets the timezone. Thus the need to install the package this way. Presumably this could be automated, but that's a problem for another day.
Next I did a bulk (re)install of all the other packages we added to the zone prior to booting.
debian# for i in `cat /tmp/pkgs`; do dpkg -i --force-depends $i; done
We now have a working debian zone with a consistent dpkg database. dpkg -l should list all of the installed software and dpkg --audit shouldn't report any problems.
One final bit of cleanup is needed. In the process of properly installing all those packages, /etc/rcS.d was populated with a bunch of startup scripts that we don't want to run. The easiest way to deal with this is:
debian# cd /etc/rcS.d
debian# mv S20proc /tmp
debian# rm *
debian# mv /tmp/S20proc .
This is a pretty heavy-handed operation, but it works because we don't have any necessary services installed yet. All we're wiping out are the low-level, hardware-config kind of scripts that we don't need inside a zone.
Preparing to apt-get the universe
The next step was to start loading the bulk of the software into the zone. My goal was to be able to use apt-get to get as much as possible off of the net, but obviously I needed to install at least enough stuff to get networking working first.
libgcc1
gcc-3.3-base
libstdc++5
apt
net-tools
debconf [1]
debconf-english
ifupdown
netkit-inetd
netkit-ping
libwrap
tcpd
netbase
With these packages installed and my /etc/resolv.conf configured properly, I could now run apt-get to install any additional software into the zone.
First up: satisfy any lingering dependencies for the packages I already manhandled into place:
root@debian:~# apt-get -f install
Reading Package Lists... Done
Building Dependency Tree... Done
Correcting dependencies... Done
The following extra packages will be installed:
e2fslibs e2fsprogs initscripts libblkid1 libcap1 libcomerr2
libdb1-compat libdb3 libdb4.2 libss2 slang1a-utf8
Suggested packages:
gpart parted e2fsck-static
The following NEW packages will be installed:
e2fslibs e2fsprogs initscripts libblkid1 libcap1 libcomerr2
libdb1-compat libdb3 libdb4.2 libss2 slang1a-utf8
0 upgraded, 11 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 1685kB of archives.
After unpacking 4497kB of additional disk space will be used.
Do you want to continue? [Y/n]
Now we can install whatever applications we intend to run, and rely on apt-get to resolve any dependencies for us. For example, I want to run the IRC client 'xchat', so I do:
root@debian:/# apt-get install xchat
The tool identifies the 41 packages I need to install before this will work, downloads all 25MB of .deb files, and installs them for me.
Lather, rinse, repeat until you have a zone that does everything you need it to.
Conclusion and Caveats
After all the work I went through to get the zone installed, I haven't actually used it very much. Our focus is primarily on Red Hat, since that's the distro we expect to formally support, so the Debian zone isn't getting a lot of use.
The only problems I found with the zone which are not resolved in our Build 31 release revolve around threading. Due to the completely different approaches Linux and Solaris take to threading, this has been a particularly tricky area for us to get right. It appears that Debian is still using the old linuxthreads model, while Red Hat has moved ahead to NPTL. In the unlikely event that I have some free time, I might investigate the linuxthreads issues. Otherwise, if you're trying to run a Debian zone and run into problems, this is probably the first place you should look.
As I said at the top, this really isn't a process that most people are going to want to go though manually. Bootstrapping any operating system is ugly, and trying to do it on top of a different operating system is doubly so.
My original goal was just to work through the install process to help light the way for other people, but I ended up uncovering a handful of bugs as well. The libraries and apps available on Debian are different enough from the CentOS code we've been using that they poke our emulation support in new and different ways. Since it revealed bugs that would have bitten us at some point, this has turned out to be a more generally useful exercise than I had originally expected.
[1] Something funny was going on with perl's flock(). When trying to configure debconf, it kept complaining that the configuration database was locked by another user. At first glance it appears that this is a bug in either perl or debconf. The file is being opened read-only, but we are trying to get a write-lock on it. That's not allowed in Solaris, but if it is allowed in Linux, then this is a bug in our emulation. Temporary workaround: comment out the flock() calls in
/usr/share/perl5/Debconf/DbDriver/File.pm.
Technorati Tags: BrandZ Solaris OpenSolaris Debian