Having been a customer of Sun, as well as an employee, when I've had a performance issue, I knew enough to make comparisons on like systems to remove any doubt about the issue.

So the other day, a customer starts complaining about performance on a 16
core box running Oracle.  When it all comes out in the wash, it turns out
they put a 16-core system behind a 4 disk, HW Raid 5 config and thought
that it should run like a comparable system (another OS, Fibre channel storage).

Huh? Are you kidding me? That's like running just on rims on an Indy Car.

Then, just to up the ante, I get another arbitrary comparison to
a different system, running a different OS, with double the cache
on the controller, and the worst possible unit of test. dd(1)

Off to get smarter about raid caching controllers.....

Off debugging my Visiontek x1550 PCI board for the radeonhd developers and I need to run something called rhd_conntest. That requires pciutils, which provides tools like lspci and friends, including a library of the function calls. The README says pciutils compiles on Solaris, but there's some assumptions about the tool chain, which are not well stated.

Getting it working

1. Download the pciutils tarball and unroll it in your source tree

2. The configure script uses sed to mash up some variables for the the lib/Makefile include (config.mk), but Solaris sed falls down on the job. Fix the configure script to use gsed, or have a functional sed in front of /usr/bin in the PATH variable.

3. The install program has problems with the constructs in the pciutils Makefile for targets install and install-lib. Just replace install with ginstall, or make sure a GNU install is ahead of /usr/bin and /usr/ucb (since /usr/ucb/install is much more aligned with open source "install programs", than /usr/bin/install) in your PATH variable.

4. The libpci.pc.in prototype in pciutils-2.2.10/lib doesn't have Libs correctly set, so add "-R${libdir}" to the line starting with Libs:

5. gmake is required for pciutils, and you'll need to set CC=gcc to make it compile on Solaris.

Once it gets installed, then I had to figure out how to get pkg-config to pick it up, which turned out to be a simple:

PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/usr/lib/pkgconfig:/usr/share/pkgconfig

Since I'm working from a pristine environment. This went pretty smooth given all things.

One of the things I get used to on my systems is the tool chain I get in my system, once I have it setup the way I like. However, I get frustrated when someone says "oh, go download it, run configure, you're done". Tried that this weekend with the radeonhd driver on a new Solaris Express install, and it's not quite that simple. I got a Visiontek x1550 PCI recently since it's supposedly one of the fastest 32-bit PCI video cards, however Solaris Express B85 didn't recognize it. It appears to have a PCI-PCI_Express bridge in it and that just flat confused Solaris, reporting that all my onboard peripherals were now PCI-Express (USB, Via Rhine Ethernet, Floppy, IDE, Audio) on my Soyo Dragon Plus V1.0. I finally got the machine booting by putting back in an AGP card, and set to work at getting the driver working. I interpreted what I read on the Opensolaris.org xwindows-discuss list that it should be a straightforward compile. IFF, you've got GIT and some other things setup, and I've got a bog-stock install.

The first pain is getting the code out of the radeonhd GIT. I haven't seen a Blastwave download for GIT, and have since got the hang of spec-files-extra , so I fired up my ubuntu 7.10 QEMU image, which I was using to download the Wine git, and pulled down the latest radeonhd source. I won't go into what happened next. Needless to say, lots of gnashing of teeth and FLW's(tm), and even when I got the driver compiled, it just spit out something that it wasn't recognized. It probably didn't help that Solaris reported all my onboard peripherals into PCI-Express devices (with nary a PCI-E bus on this AGP/PCI system) once the Visiontek PCI x1550 goes in the system, or that I have to have an AGP board in the system just to get it to boot. And for some reason, Solaris also doesn't see the second video port. So I thought it wasn't working. Eventually, after some debugging I figured out my DVI port wasn't recognized, and I got a xorg.conf file that eventually got me dual screens with my Nvidia FX5700.

I think I've got the punch list to make this work starting from a fresh Solaris Express install.

1) get xorg utils-macros-1.1.6, configure and install. It ends up in /usr/local/share/aclocal if you don't give ./configure a --prefix={someplace}.

2) get xorg-server.m4 from freedesktop.org and install it in /usr/local/share/aclocal or whereever you set --prefix={someplace}

3) download the radeonhd GIT or a V1.2.0 April 12th, 2008 tarball.bz2 from here and unroll it someplace.
If you have a git client, then it's a simple

git clone git://anongit.freedesktop.org/git/xorg/driver/xf86-video-radeonhd

4)Download and build pciutils as listed in that blog entry.

5) setup environment variables to help autogen.sh (since the radeonhd code from the git HEAD doesn't have configure or friends built, though the released xf86-vidoe-radeonhd-1.1.0.tar.bz2 does.

For sh/ksh/bash

  ACLOCAL="/usr/bin/aclocal-1.10 -I/usr/local/share/aclocal"
  AUTOMAKE="/usr/bin/automake-1.10"
  PKG_CONFIG_PATH="/usr/local/lib/pkgconfig:${PKG_CONFIG_PATH}"
  export ACLOCAL AUTOMAKE PKG_CONFIG_PATH

For csh/tcsh 

  setenv ACLOCAL "/usr/bin/aclocal-1.10 -I/usr/local/share/aclocal"
  setenv AUTOMAKE "/usr/bin/automake-1.10"
  setenv PKG_CONFIG_PATH "/usr/local/lib/pkgconfig:${PKG_CONFIG_PATH}"

Initially, I just symlinked aclocal-1.10 to aclocal and did the same for automake, but decided for the purposes of blogging, that I should do it the Right Way(tm), so I found out what I should be doing.

5) Generate the radeonhd source build toolchain (configure, ltmain.sh, etc) and build

  ./autogen.sh  (default install is to /usr/local/lib/xorg/modules/drivers)

  ./configure

  make

6) Install the driver and get Solaris to use it

  make install 
  cd /usr/X11/lib/xorg/modules/drivers
  mv radeon_drv.so radeon_drv.so.ORIG
  ln -s /usr/local/lib/xorg/modules/drivers/radeon_drv.so .

This will allow you to switch back to the original driver should something go badly.

7) do some testing. According to the release notes for Solaris, by default radeonhd is not used so you have to create a xorg.conf file for it.

   /usr/X11/bin/Xorg -configure

   edit ~/xorg.conf.new and make sure variables are right, and push up to 
   /etc/X11/xorg.conf

Once you've got the driver installed, it's probably best to disable cde-login or gdm-login through smf and remotely login via ssh to start the driver.

   svcadm disable cde-login

   /usr/X11/bin/Xorg -logverbose 7

   svcadm enable cde-login

This generates a bucket of debug in /var/log/Xorg.0.log, and is one of the reasons that I figured out that my driver was actually working, though one of my ports just isn't discovered... The good news is that it's working, at least on the VGA port.

Life happens the way it happens. Seems rather bizarre I haven't blogged in such a long time, but it's time to get back to it. I'm still mucking with QEMU on Solaris, but not much these days as I've been working on extending the open solaris zyd driver (USB 2.0 WIFI). Funny how a $15 device can make you spend weeks of time trying to make a driver work for it, especially when the openbsd folks basically have all the bits. Oh well. it's moving forward. I just wished I'd known what I was getting into when trying to expand a driver whose hardware encompasses something like 70 different vendors, 2 chipsets, and at least 5 known working radios. Other fun stuff has been sticking a USB 2.0 card in the 64-bit, 66Mhz slot of the LX50 I just loaded up with Solaris Express B84 and Ubuntu 7.10 so I can do some performance testing on the zyd driver. Now if I could only get the Radeon 9100 actually displaying in the 66Mhz slot, since Ubuntu saw it in the /var/log/Xorg.0.log file.

Was over at a friends house last night and came in to him watching his collection of Penn & Teller's syndicated Showtime series called "BullShip!" (well, that's not quite the title, but work with me here). Actually, the collection is just a temporary loaner from NetFlix, which he makes heavy use of.

Anyway, Penn and Teller have been around for quite a while. My first memories of them were doing magic tricks on television specials, and then they debunk how they did it. I guess that's how they got their start into their Showtime series. Penn Jillette has also done a fair amount of writing, and I'd forgotten where I'd seen it until I had gone further into their website where I realized he had written for PC-Computing magazine for quite a few years. More recently, I was out at the Server Consolidation and Storage Symposium last summer in Las Vegas, and Penn was the special guest speaker near the end of the conference. If there's anyone who is dedicated to making people open their eyes, I don't know who it is, or I haven't heard or seen them. It was a real pleasure to see his monologue in person.

The funniest of the programs we watched last night was on Water. They'd actually setup a water steward at a restaurant and printed up a water list, and were pushing these $3-7 bottles of water and listening to these folks go on about the water. Meanwhile, they keep cutting back to the water steward filling up the P&T created bottles with a garden hose from the patio of the restaurant with a manical smile on his face. At the end, they informed their paying customers that all the different water they had drunk that night had come out of the same garden hose out back. Maybe they ought to do the same with wine and see how some of the wanna-be wine snobs do. I doubt they could trick the real wine experts though.

Me, I drink club soda with lime. I get free refills that way, and mostly what I taste is the lime. But occasionally I will opt for a bottle of Peligrino if someone else is paying. I think I quit buying bottled water in a restaurant when I realized one night that I spent more on bottled water than I did on the $6 desert.

Mr Bill always seems to have the most interesting web tidbits on his blagh. This South Park characature editor is actually the most interesting of the self tests and such. It's not based on some web persona's interpretation of how you answer a bunch of questions, but how you decide to create the character. Now if doing things like manipulating the image to have a transparent background was as easy as creating the South Park character persona... (at least until I found a more pleasing background).

Ben

Surely you jest. Windows 98? Yep. It's kind of Microsoft's bane. I didn't believe it when I heard it, but one of the target markets of Sun's Java Desktop System was the Windows 98 Desktop replacement market. I guess when you're looking at a site license for Win2K, WinXP or Windows 2003, or just the incrmental upgrades, $50 a pop for JDS looks like a pretty good deal, especially with Star Office thrown in the mix. And that also means it should run decently on on PC's that are a couple of years old.

Why Windows 98 under Solaris?

Most of the applications that I find myself lacking for Solaris X86 include the normal kinds of things that day-to-day desktop users need. A browser, a couple of of plugins, a couple of applications that probably everyone needs, like media players (other than the M$ one). And then there are the ubiquitous Windows upgrades, since it is Windows. In about 2 hours of part time work, I managed to load all the windows 98 updates, Mozilla, Firefox, Thunderbird, Adobe Acroread 6.0,Quicktime, the Macromedia Flash and Shockwave plugins for both Internet Exploder and the Mozilla family, Real Player, and TightVNC. Shock. The Solaris X86 community has been complaining about this for *years*. Adobe didn't do Acrobat Reader 7 for Windows 98, but Acroread 6 is better than the ancient Acroread 4.05 which was the last supported version for Solaris x86. Xpdf and Gpdf are allright but there always seems to be some document which doesn't render correctly. Macromedia finally got on board and there's a Flash plugin for Mozilla 1.4 for Solaris X86, though no Shockwave, and RealPlayer is still a noshow. If you really want RealPlayer for Solaris X86, Juergen Kiel in Germany figured out how to write a shim so we could use the UnixWare (This still has me baffled why UnixWare had a port and Solaris X86 did not) RealPlayer 8 plugin. You can find the code to make it work at The Solaris X86/Real Player Wedge web site. Now that I have all those tools, the other reason is Windows 98 is lightweight enoughto run decently under QEMU's emulation. As I pointed, it's pretty ubiquitous, so finding a copy should be dead easy, and it doesn't requires that NTFS which Solaris doesn't have any support for, unless you use my hack from yesterday to use Knoppix to get at the NTFS file system.

Installing Windows 98 under QEMU on a Solaris/X86 Host

So you need a couple of things. 1) Windows 98 (SE Preferably) media and license. An ISO image will do just fine to. 2) Disk space for the ISO and IMG files 3) a couple of hours or more depending on the speed of your machine, and the amount of memory you can give the QEMU guest VM. If you don't want to be flipping CDROM's more than once, I prefer to use unix "dd" to turn them into ISO images. I have finally gotten used to the Solaris vold (though I know people who still hate it, I have my days too) so here's how we do this:

    # Put the Win98 CD in your CD/DVD Drive
    $ eject -q cdrom       # or cdrom0 or cdrom1. Whatever your solaris system uses

    $ cd /export/src/iso          # and have at least 450MB of free disk space for the Win98SE ISO

    $ dd if=/vol/dev/rdsk/c1t0d0/win98 of=win98se.iso bs=10240k

    # If someone has a way of doing this with cdrw, let me know and I'll modify this. I don't see
    # a way around this other than stopping a copy halfway through, so that seems more of a hack
    # than using "dd" 

Next, we need to make an image for Windows 98 to reside in. We whip out qemu-like

    $ cd /export/src/images
    $  mkdir Win98SE; cd Win98SE
    $ /tmp/qemu/bin/qemu-img create -f qcow win98se.img 2G    # We can always grown this later

Next, I find it easier to prepare the disk by hand, than to let Windows 98 do it's thing. So I cheated to bootdisk.com and got a Windows 98 Boot disk, and started qemu with that like:

    # from /export/src/iso/Win98SE
    $ /usr/sfw/bin/wget http://www.mirrors.org/archived_software/www.bootdisk.com/winfiles1/boot98se.exe
    $ unzip boot98se.exe    # Since it is a self-un-archiving exe, unzip works fine on it.
    #
    # now start a basic qemu just to get the disk made
    #
    $ /tmp/qemu/bin/qemu -fda winb98se.IMA -boot a -hda win98se.img
    #
    # the normal windows 98 boot disk comes up, run fdisk, allocate the 2G of space to
    # primary DOS partition, then run format/s c:.  Once the format is complete, you can
    # reboot to verify you can see all 2G of the C: drive, or take it for granted that it
    # worked, and exit via "[CTRL-ALT-2], then "quit".

Next we setup the QEMU environment for loading Windows 98/SE. Decide how much memory you're going to use, etc, and use a variation of the script from the first day, to start qemu:

    $ /tmp/qemu/bin/qemu -cdrom win98se.iso -user-net -enable-audio \
                         -pci -m 256 -k en-us -boot d -hda win98se.img
    # and go through the normal Windows 98 install.
    # when complete, shutdown and if you're paranoid, make a backup copy of win98se.img.

Using the Solaris Samba server to provide writable space to the Windows 98 Image

Using ftp in a QEMU Guest OS can be kind of a pain, especially since the firewalling done by QEMU and it's slirp network stack. I've found that under a Solaris X86 guest, a ftp client called "lftp" actually works pretty good, and is available from blastwave.org using pkg-get. But trying to fight the Windows 98 Command.exe and FTP.exe is hopeless. So, it turns out to be very easy to enable Samba so you can use native Windows 98 File sharing back to the host.

    $ mkdir /export/src/samba; chmod 755 /export/src/samba  # or whatever permissions float your boat.
                                                              # only the QEMU guest instance will have access
    $ cd /export/src/samba
    # first thing to do is get windows 98 to properly use the HLT command so when
    # idle, it's not sucking the CPU like crazy. It's on the samba share so
    # we can just run the install once we have it mounted.
    #
    $ /usr/sfw/bin/wget http://www.user.cityline.ru/~maxamn/amnhltm.zip
    $ mkdir win98hlt; cd win98hlt; unzip ../amnhltm.zip
    #
    # start Windows 98 for the first time since the install, note the change in
    # the "-boot" parameter from "d" to "c"
    #
    $ /tmp/qemu/bin/qemu -cdrom win98se.iso -user-net -enable-audio -pci -m 256 \
                -smb /export/src/samba -k en-us -boot c -hda win98se.img
    #
    # This boots windows, and login normally.  Open a COMMAND prompt and  enter
    #
    C:\WINDOWS> echo "10.0.2.4 smbserver" > lmhosts
    #
    # or use notepad.exe, but be sure to make sure it saves it as "lmhosts", 
    # or you rename it from lmhosts.txt to lmhosts. This cost me about an hour 
    # today debugging this since notepad.exe added the ".txt" extension to the file.
    # when done, you should have a file called "C:\WINDOWS\LMHOSTS.
    #
    % Right Click on "Network Neighborhood", and select "Map Network Drive"
    # QEMU setup an internal IP address in the user-net stack that connects 10.0.2.4
    # as a proxy to the Solaris Samba server, that QEMU starts up when you request to
    # map a drive.  
    #
    % Map "\\SMBSERVER\QEMU" to drive "E:"
    #
    # If this happened succesfully, Windows opens an explorer window on "E:"
    #
    # In the command window you have open, type
    $ E:
    $ cd win98hlt
    $ AMNHLT.BAT       # This installs the sequence to use the CPU HLT instruction.
    # close the command prompt
    #
    % Open DISPLAY under the Control Panel and go to Effects and turn off most of the
    % visual effects. Things like annimation and such go a long way to making Windows 98
    % under QEMU not feel so sluggish.  While you're in the display panel, change
    % the resolution and color depth to something more pleasing.  The Cirrus Logic
    % GDM 5446 chipset can do 1280x1024 @ 16-bit color with little issue. There is
    % probably not much to be gained by going to 24-bit color, and it will be 
    % slower.  If you need 24-bit color, you should be asking yourself if you 
    % don't need a specific PC for that application.
    #
    # At this point, you can run windows update, and get as much stuff updated as
    # possible.  The updates don't take too long on a 2Ghz Athlon.  Once you have   
    # the updates done, it probably a good time to stop the system, and make a backup
    # of the current image.  Just in case something goes wrong.  My image turned out
    # to be about 250MB at the time of the great IE6 Update.

So at this point, you should have a working Windows 98 SE install, all patched up, with visual effects turned down to minimum and the HLT cpu instruction enabled, and you can start loading your favorite batch of applications. And making backups is as easy as copying one big file.

On my laptop, I have a hibernation partition on P1, WinXP [ntfs] on P2, and Solaris Next on P3. Since Solaris cannot natively read an NTFS partition, I'd like to be able to read my files off of my WinXP partition without having to use something like a FAT32 partition to transfer data around. Since Knoppix can read an NTFS partition, I figured I should try and see if I can get to the data. It's a bit convoluted to do it this way, but it prevents having to reboot Solaris to get back to Windows XP so I can transfer some files. So I start qemu like this, still booting from the cdrom image, and assigning -hdd to be the primary Solaris fdisk partition /dev/rdsk/c0d0p0 on my laptop. Of course, unless you're running qemu as root, you won't have access to the physical partition as a user. We will setup access to the raw device for read only access by running the following command, as root, against the physical device's acl, for the user running qemu to access the NTFS partition.

    $ su root
    # getfacl `ls -l /dev/rdsk/c0d0p0 | sed 's,.* -> ../..,,'`

        # file: /devices/pci@0,0/pci-ide@11,1/ide@0/cmdk@0,0:q,raw
        # owner: root
        # group: sys
        user::rw-
        group::r--            #effective:r--
        mask:r--
        other:---
    # setfacl -m user:bent:r-- `ls -l /dev/rdsk/c0d0p0 | sed 's,.* -> ../..,,'`
    # getfacl `ls -l /dev/rdsk/c0d0p0 | sed 's,.* -> ../..,,'`

        # file: /devices/pci@0,0/pci-ide@11,1/ide@0/cmdk@0,0:q,raw
        # owner: root
        # group: sys
        user::rw-
        user:bent:r--
        group::r--            #effective:r--
        mask:r--
        other:---

I use the -snapshot to prevent any writes from happening to the disk, just as a matter of caution, even though I'm defining the boot device as -boot d. I do the following as the user which was given the acl read rights to the device:

    /tmp/qemu/bin/qemu -cdrom /vol/dev/rdsk/c1t0d0/knoppix -user-net -enable-audio \
                         -pci -m 256 -k en-us -boot d -snapshot -hdd /dev/rdsk/c0d0p0

At the Knoppix boot prompt, I type:

    boot: knoppix dma screen=1152x864

and wait for Knoppix to boot. If all goes well, you should see a couple of "Hard Disk" icons on the desktop, and since Knoppix sets up the mount entries, the knoppix user (non-root) is allowed to mount the drive. Verify that the partition that has the NTFS file system on it is listed in /etc/fstab in your Knoppix file system. Then just use the mount command with the mount point that corresponds to the mountpoint in the /etc/fstab. In my case, /dev/hdd2 is the location of the NTFS file system.

    $ mount /dev/hdd2

If there are no errors in the mount, then you should be able to cd /mnt/hdd2 and see the NTFS file system under Knoppix. There should be no problems because by default, Knoppix will mount the NTFS file system read only, which is all you need to access the files so that you can copy them from NTFS to the Knoppix image, and then transfer them out of the Knoppix virtual machine using ftp, scp, or samba (which I haven't figured out how to setup yet).

Using this configuration, is a perfect example of using a quick start configuration. It turns out that leaving the NTFS file system mounted in the QEMU Knoppix session caused the quick start to be something more like 60 seconds, rather than the 10 seconds I was seeing off a standard config. It also might have something to do with using the "-snapshot" option. I plan on doing some more testing.

At this point, we should have a working set of qemu binaries loaded in /tmp/qemu, a ISO image or CDROM copy of the Knoppix Live CD and a little experience starting up a Knoppix instances under QEMU in a Solaris 10 Host.

Next, we need to create a disk image file that we can use to load an OS image on. The command used to create these images file is called qemu-img. The qemu-img binary supports multiple formats including raw, cow, qcow and vdmk to name a few. Raw format appears to be the most platform neutral and independent, while the most versitile format appears to be qcow, which supports compression and encryption, as well as copy-on-write (COW). To create our image file to load Knoppix onto, run the commands:

     cd /export/src/images/Knoppix_3.8
     /tmp/qemu/bin/qemu create -f qcow knoppix.img 4G

You can verify that it built out the image the way you expected to by invoking the command:

     $ /tmp/qemu7/bin/qemu-img info knoppix.img                                                                   
     image: knoppix.img
     file format: qcow
     virtual size: 4.0G (4294967296 bytes)
     disk size: 17K

Starting Knoppix LiveCD to install on a QEMU Disk image

As it turns out, if you don't have your hard disk partitions setup and the /dev/hda1 partition doesn't have an ext2fs on it, running "tohd=/dev/hda1" at the Knoppix prompt doesn't work. So the first thing to do is to boot the Knoppix image off of CDROM or ISO image, yet add in the virtual disk image defined by "-hda knoppix.img". Because we're stil specifying "-boot d", QEMU will boot off the KNOPPIX CD or ISO image first.

     /tmp/qemu/bin/qemu -cdrom /vol/dev/rdsk/c1t0d0/knoppix -user-net \
              -enable-audio -pci -m 512 -k en-us -boot d -hda knoppix.img

Once Knoppix is booted, open a Konsole window from the KDE toolbar at the bottom of the Knoppix screen (It looks like a terminal), and type:

     $ su   # no password required for Knoppix
     # fdisk /dev/hda

From the fdisk partition menu, create a primary Linux partition on partition 1 of 3600MB and a Linux Swap partition on (0x82) on partition 2 of 500MB (remainder of the disk). Make partition 1 the active partition, but I'm not certain that because of how QEMU boots a system on whether or not this make any difference. But that's how you do it with a normal hard disk so let's do it that way here. Write out the changes to the disk and then do a mkfs /dev/hda1. Once that is complete, Halt the Knoppix image and restart it with the above command line. Once you see the Knoppix prompt again like this type:

     boot: knoppix screen=1152x864 dma tohd=/dev/hda1   # 1152x900 works under JDS/Linux Host, but 1152x864 is all Solaris can do

and then watch the disk spin for a bit, with the screen looking somewhat like this. When it's all done, Knoppix will boot normally. [On my 2Ghz Athlon system that I did this test on, the transfer rate racked up an impressive 30Mb/second for transferring the image from the ISO to the virtual disk on /dev/hda1.] What this actually does is just copy over the directory structure on the CDROM onto the Hard disk. (Gee, I thought you could install Knoppix to the Hard disk from the boot menu). Now, when you boot from your cdrom, you can add "fromhd=/dev/hda1" which will make it boot off the hard disk image. Given all things, this is probably not a big win, given that running from the CDROM or ISO image isn't going to be the limiting factor. However, it's a very quick test to make sure your virtual hard disk is writable, and you can boot from the image using the CDROM or ISO image. The start line for qemu doesn't change from the previous run, but how we invoke Knoppix is different. At the boot prompt, type:

     boot knoppix screen=1152x864 dma fromhd=/dev/hda1

Making Knoppix boot very fast while using a HD install

This morning, we looked at how we can make QEMU boot a system much faster using the savevm/loadvm commands from the monitor. Once the above system has booted, and you've got all the applications setup the way you want them, then type CTRL-ALT-2 to go into the QEMU monitor. Once inside the QEMU monitor, type "savevm knoppix-save.vm" and then "quit". The file knoppix-save.vm should be in directory /export/src/images/Knoppix_3.8

Restarting a QEMU savevm'd image

Once again, we add another parameter to the startup line for the qemu instance. To get the very quick restart of a running image, we run:

     /tmp/qemu/bin/qemu -cdrom /vol/dev/rdsk/c1t0d0/knoppix -user-net -enable-audio \
           -pci -m 512 -k en-us -boot d -hda knoppix.img -loadvm knoppix-save.vm

and in about 10 seconds on my PIII/1Ghz with 1.5G of Ram, my Knoppix instance is back where it was when I ran the savevm command from the QEMU monitor.

After you restore an image like this, I found out that you should do a [CTRL-ALT-1] to reset the virtual machine's system state, which explains the weirdness I saw with the keyboard and restored image

I wish I had found this sooner. Waiting 10-15 minutes for something to bootup and enumerate all the devices is pretty hideous. Makes me really want to see a Solaris kqemu module for QEMU get written so we can have the near machine speed that the Linux folks have (I run JDS3/Build 32 on my Toshiba M2, and with kqemu it's really really nice, but this blarg is about making QEMU work on Solaris).

Yesterday, I talked about QEMU and what it can do. As I've been working with it, there are a couple of reasons I want to run an OS under QEMU with Solaris 10 as my Host.
1) I need/want to run some Windows applications that I don't have under Solaris (say acrobat 7 or real player).
2) I want to test an OS out without having to frag my laptop again (as I've been doing since last year with Solaris 10 in beta, and now Solaris 10.1 [aka 11] in beta)

Getting started with QEMU

Assuming you can follow the download and patching of qemu-0.7.0 from yesterday, then running of

     ./configure --prefix=/tmp/qemu --with-oss=yes --oss-inc=/opt/oss/include; gmake install

the next thing you need to do is setup a working area. For the purposes of the demostration, I use /export/src/images as my top level directory where I keep the QEMU images. I also use /export/src/iso as the location for any CDROM image iso files I use to boot/install a guest OS under QEMU.

     $ mkdir -p /export/src/iso /export/src/images
     $ cd /export/src/images
     $ mkdir Knoppix_3.8
     $ cd Knoppix_3.8

The great thing about Knoppix is the ability to boot a complete OS on a PC without having to touch the hard disk, not to mention it's free and they have several yearly updates to the code base. These disks have become great recovery tools and have a lot of different uses. My favorite use of Knoppix is a test of how QEMU is running. Since it seems to take about 4 hours to install Solaris Express from a DVD image in a QEMU session, I'm really not interested in finding out it doesn't work after 4 hours. Plus, since it doesn't really require any disk space requirements at all, starting Knoppix under qemu is dead simple. Not everyone has Windows 98, 2000, XP, etc, and the time requirements to actually get comfortable with it are minimal compared to the amount of time it's taken me to get things like a 5 CD iso build of JD3 Linux/Build 32 installed, or Solaris Express on DVD. So I think is a pretty good example, plus I think any tech ought to be carrying a Knoppix disk, or something like it.

Getting Knoppix 3.8

Assuming you've gotten a Knoppix 3.8 iso image from your local bittorrent, and burned it to a CDROM, this is all you need. Blastwave has a version of bittorrent for Solaris, so getting a copy is just a matter of doing a "pkg-get -i bittorrent", assuming you loaded pkg-get from www.blastwave.org. (It seems that Knoppix V 3.7 was the last version available via FTP)

Booting Knoppix 3.8 from a CDROM under QEMU

So either you have an ISO image sitting in a directory like /export/src/iso/KNOPPIX_V3.8.1-2005-04-08-EN.iso or you actually burned the image onto media and it's in a CDROM drive in the Solaris system. If you are running vold (volume managment), we need make sure we can see the physical device.

$ eject -q cdrom       # to get the volume name of the knoppix disk
                       # it should be something like /vol/dev/dsk/c1t0d0/knoppix
                       # but we really need to give it the rdsk name so it can boot it.
#
#  To start Knoppix under QEMU if using a real CDROM under Solaris with vold enabled, try
#
$ /tmp/qemu/bin/qemu -cdrom /vol/dev/rdsk/c1t0d0/knoppix -user-net -enable-audio -pci -m 512 -k en-us
#
#  To start Knoppix under QEMU if using an ISO image file under Solaris, try
#
$ /tmp/qemu/bin/qemu -cdrom /export/src/iso/KNOPPIX_V3.8.1-2005-04-08-EN.iso -user-net -enable-audio -pci -m 512 -k en-us
#
#
# -user-net      : enables the slirp network stack that QEMU presents to the guest OS. This is default
#                  if you don't have tunnels (The tunnel code in QEMU for solaris is not quite there yet)
# -enable-audio  : have QEMU present a Soundblaster 16 to the virtual machine
# -pci           : have QEMU present a PCI bus.  The opposite is ISA, but at this point is only useful for
#                  things like DOS. Solaris Newboot as a guest chokes on "-isa"
# -m 512         : the ammount of physical memory to be given to QEMU for this guest OS.
#                  my system has 1536MB so giving it 512 is not too bad. This variable you
#                  will need to be careful with.
# -k en-us       : Sets up the keymapping for QEMU. Seems to be important if you're doing a savevm/loadvm,
#                  which I'll talk about later.
#

What you see immediately is a screen that looks like this. At this prompt type:

     boot knoppix screen=1152x864 dma # or screen=1024x768, something workable with your display window
                                         # 1152x900 works in JDS but for some reason doesn't work on Solaris. Go figure.

This takes about 15 minutes to boot on my PIII/1Ghz, and about 7 minutes to boot on my 2GHz Athlon XP. What this gives you is a working Knoppix instance running in QEMU under a Solaris host. Assuming you don't have any compile problems, and your Solaris host has a valid network/DNS mappings, your Knoppix guest OS should have an eth0 interface with an address of 10.0.2.15. Since QEMU provides a virtual DHCP server to the client (How else did it get that 10.0.2.15 address), and proxies DNS from a virtual host using 10.0.2.3 and having a defaultroute of 10.0.2.2, networking for the most part should just work. One big caveat - You cannot ping out of a guest OS to the host or any other address because SLIRP does not support ICMP. Telnet, FTP, SSH/SCP, X11, VNC and Samba all work without issue.

To verify, open a kconsole window (In your Knoppix QEMU Guest) , and run:

     $ ifconfig eth0

and see

     eth0      Link encap:Ethernet  HWaddr 52:54:00:12:34:56  
               inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
               inet6 addr: fe80::5054:ff:fe12:3456/64 Scope:Link
               UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
               RX packets:634 errors:0 dropped:0 overruns:0 frame:0
               TX packets:571 errors:0 dropped:0 overruns:0 carrier:0
               collisions:0 txqueuelen:1000 
               RX bytes:267069 (260.8 KiB)  TX bytes:53495 (52.2 KiB)
               Interrupt:11 Base address:0xc100 

(YES, your networking will look *exactly* like that, because the "virtual machine" presented by QEMU to the guest OS looks the same.)

One thing that occurs when you click your mouse into the QEMU guest OS window, is that QEMU grabs it, and doesn't let go of it. If you feel like you've lost your mouse, hit CTRL_ALT and see if it releases your mouse so you can go outside the QEMU window.

Start firefox and surf the web (though if your Solaris host uses a web proxy, so should the Guest OS's brower). You can ssh to your Solaris host (make sure to use the real network address of the host, not 127.0.0.1). FTP also works, but I find that often you may have to use passive mode ("quote pasv") in the ftp client to pass data back and forth. I've found ssh/scp much easier to work with when dealing with a QEMU guest OS and the user-net stack.

Booting Knoppix Really Fast

QEMU has the ability to save the state of the local VM, and restore back to it from the command line. How we do this is by going into the QEMU monitor using [CTRL-ALT-2]. The monitor is a command line tool which provides a set of of tools and features, as displayed by this help menu

     QEMU 0.7.0 monitor - type 'help' for more information
     (qemu) help
     help|? [cmd] -- show the help
     commit  -- commit changes to the disk images (if -snapshot is used)
     info subcommand -- show various information about the system state
     q|quit -- quit the emulator
     eject [-f] device -- eject a removable media (use -f to force it)
     change device filename -- change a removable media
     screendump filename -- save screen into PPM image 'filename'
     log item1[,...] -- activate logging of the specified items to '/tmp/qemu.log'
     savevm filename -- save the whole virtual machine state to 'filename'
     loadvm filename -- restore the whole virtual machine state from 'filename'
     stop  -- stop emulation
     c|cont  -- resume emulation
     gdbserver [port] -- start gdbserver session (default port=1234)
     x /fmt addr -- virtual memory dump starting at 'addr'
     xp /fmt addr -- physical memory dump starting at 'addr'
     p|print /fmt expr -- print expression value (use $reg for CPU register access)
     i /fmt addr -- I/O port read
     sendkey keys -- send keys to the VM (e.g. 'sendkey ctrl-alt-f1')
     system_reset  -- reset the system

Assuming you started your session as I described, you can hot-key between the QEMU guest, the QEMU monitor, and the QEMU serial port. CTRL-ALT-1 from the MONITOR or QEMU serial port will take you to the Guest OS. CTRL-ALT-2 from the Guest OS or the QEMU serial port will take you to the MONITOR. CTRL-ALT-3 from the GUEST or the MONITOR will take you to the QEMU serial port.

Saving a running QEMU Guest OS session

Use CTRL-ALT-2 to go to the MONITOR and type

     savevm knoppix-save.vm

then

     quit

You will find a file created in the local directory (still in /export/src/images/Knoppix_3.8, right?) called knoppix-save.vm which contains the memory contents of the virtual machine. Since we're running off of a CDROM iso image, it doesn't matter about what's in the "file system". We're not using one yet. As you can see, the more you do with QEMU, the more command line parameters you need, which is why I wrote the script to do this. There are some java GUI's, but I'm a command line kind of guy, so I write my own stuff. Restart the Knoppix session using the command from above plus a new parameter -loadvm knoppix-save.vm and make sure that the amount of memory you originally specified at the time you saved this instance is the same that you start the -loadvm knoppix-save.vm: like:

     /tmp/qemu/bin/qemu -cdrom /vol/dev/rdsk/c1t0d0/knoppix -user-net -enable-audio \
                        -pci -m 512 -k en-us -boot d -loadvm knoppix-save.vm

This will have the Knoppix system back up in about 10 seconds. Once the system is back up, switch to the QEMU console with [CTRL-ALT-2], then switch back to the VM using [CTRL-ALT-1]. This appears to prevent the weirdness I'm seeing with a Konsole window and the keyboard interaction, where the keyboard doesn't seem to work correctly. But still, it's much faster than the 15 minutes it took to actually start the virtual machine from a dead stop.

This is all fine and good in testing. However, the savevm/loadvm feature is more likely to be used on Operating Systems that have a read/write disk, and not a read-only disk such as a CDROM or ISO image. This is where it gets tricky. In order to prevent corruption of the Hard disk image file containing the Operating System, if you use the savevm/loadvm feature, it is imperative that you not boot the disk image without the -loadvm and it's statefile. Otherwise, some pretty serious corruption could take place.

QEMU is a generic and free open source process emulator, and has the ability to run practically any operating system across platforms.

Well, almost.

I found this code about a month ago, and with some help from other Solaris type developers, google and the qemu-developers mailing list, I've been able to get the code working under Solaris 10, 32-bit X86. So far, I've been able to install Win98SE, Win2KPro, Win2K Svr SP4, WinXP Home upgrade and Win 2003 Server under QEMU on a Solaris host. In addition, JDS3, Solaris 10, and Solaris Express work, as well as things like the Knoppix live CD.

Windows products are most finicky. Win98SE actually runs pretty decently on my Toshiba M2, now that I added configure support for OpenSounds' OSS Audio driver. I've got an image that has all the latest patches, Real Player, Adobe Acroread 6.0, Quicktime, Flash/Multimedia plugins and the latest Mozilla product suite (Firefox, Thunderbird and Mozilla). Why Windows 98SE? It's very lightweight, and there are two patches for it. One which implements the halt instruction properly so when I leave the qemu instance idle, it's not sucking the CPU down. And second, the power management actually turns itself "off" so I don't actually have to quit from the qemu monitor session (Which you get into by typing "ctrl-alt-2")

However, getting Win2K, XP Home and Win 2003 installed can be somewhat traumatic. The emulation is not quite perfect, so the virtual machine tends to hiccup, especially during installs. There is a well-known bug when installing Win2K under QEMU, and manifests itself by filling up the virtual disk. This can be worked around by going into the QEMU monitor (ctrl-alt-2) and stopping the instance and restarting. I've done this a few times, and mostly it never worked. I eventually figured out from a QEMU user's mailing list posting that you can work around this problem by adding -hdachs cylinders,heads,sectors to the the start line for qemu, such that the virtual disk is has 16 heads and 63 sectors/per cylinder, and just do the math to divide the size of the virtual disk by (16*63*512) to get the number of cylinders.

However, I've found out that often times when I've gotten a Windows install to work, I can't patch it. Some technology called Slipstream which allows you to take a Windows install CD and add patches to it becomes very attractive, especially since the patching isn't quite 100%. I'll eventually figure out if this works and post how it goes.

The most current release of QEMU (0.7.0) available at http://fabrice.bellard.free.fr/qemu/download.html and the patch against qemu-0.7.0 to make it compile under Solaris 10/X86 is found at qemu-0.7.0-solpatch-050504. To configure and build QEMU-0.7.0 for Solaris, I used gcc-3.4.3, gmake, gnu fileutils and libSDL-1.2.8 from www.blastwave.org. If you've never used Blastwave's tools, checkout the website. It has a script called pkg-get which is Solaris derivitive of the Debian apt-get tools. The Blastwave Solaris packages all have dependencies, which pkg-get will automagically download and install for any package you download. In addition, the sound drivers at www.opensound.com can be downloaded and used for several months if you don't want to buy a license. They definitely seem to work better than using the stock Solaris audio driver, but that could just be a feature of the SDL library.

Once you have the the code downloaded, cd to the directory holding qemu-0.7.0/ directory and run

gpatch -p0 < qemu-0.7.0-solpatch-050504 

to patch the current code. The configure script can be run like this depending on whether you have OSS sound drivers or not.

./configure --prefix=/tmp/qemu --with-oss=yes --oss-inc=/opt/oss/include

or

./configure --prefix=/tmp/qemu --with-oss=no   # if you don't have the OSS Sound drivers

If you're wondering why I used the prefix to be in /tmp/qemu - The code is quite "beta". You don't have to be root to run the code and given the fact that development on the code is going pretty quickly, I find this a very easy way to test out a set of patches or changes, without having to worry about stuff being left on the hard drive.

Here's an example shell script I use to start most images. I edit the first lines if I want to be able to get back to the virtual machine using the redir parameters :

#!/bin/bash
#
#CONFIG - set the USE variables to a value if you want their config
#         value below to propogated into the QEMU start line.
#
USE_LOCALTIME=1
USE_SSH=1
USE_TELNET=1
USE_FTP=1
USE_FDA=
USE_CDROM=1
USE_AUDIO=1
USE_BOOT=1
USE_KEYMAP=1
USE_LOADVM=
USE_MEMORY=1   # should always have this defined
USE_HDA=1
#
##########################################################################################
#
# LOCALTIME_CONFIG : "-localtime" # whether to use localtime from the bios or not
#
LOCALTIME_CONFIG="-localtime"
#
# SSH_CONFIG : -redir tcp:5022:10.0.2.15:22 - redirect tcp port 5022 on the host OS
#                                             to tcp port 22 in the guest OS.
#            : -redir udp:5022:10.0.2.15:22 - redirect udp port 5022 on the host OS
#                                             to udp port 22 in the guest OS
#  This SSH_CONFIG is for allowing systems outside the HOST system to access the
#  guest OS via SSH on port 5022 on the Host.  The TELNET_CONFIG and FTP_CONFIG
#  work the same way.  FTP is a bit funky, in that you have to use passive mode
#  often because of how the port redirector code works since it's part of the host.
#
SSH_CONFIG="-redir tcp:5022:10.0.2.15:22 -redir udp:5022:10.0.2.15:22"
TELNET_CONFIG="-redir tcp:5023:10.0.2.15:23"
FTP_CONFIG="-redir tcp:5021:10.0.2.15:21 -redir udp:5021:10.0.2.15:21"
#
# FDA_CONFIG : "-fda /vol/dev/rdsk/floppy/unnamed_floppy" to give access to the real floppy
#              "-fda /export/src/iso/DU.img" to give access to the virtual floppy image DU.img
#
FDA_CONFIG="-fda /export/src/image/Knoppix_3.8/save_floppy.img"
#
# CDR_CONFIG : "-fda /vol/dev/rdsk/c0t0d0/knoppix" to give access to the real cdrom
#              "-fda /export/src/iso/Knoppix-3.8.iso" to give access to a virtual cdrom through the iso image
#
CDR_CONFIG="-cdrom /vol/dev/dsk/c0t0d0/knoppix"
#
AUDIO_CONFIG="-enable-audio"
#
# BOOT_CONFIG : "-boot c" to boot off of first hard disk
#               "-boot d" to boot off of second hard disk or CDROM
#
BOOT_CONFIG="-boot d"
#
# KEYMAP_CONFIG : "-k "
#
KEYMAP_CONFIG="-k en-us"
#
# LOADVM_CONFIG : "-loadvm ${LOADVM_FILE}"
#
LOADVM_FILE=knoppix_save.img
if [ ! -f "${LOADVM_FILE}" ]; then
  unset USE_LOADVM
else
  LOADVM_CONFIG="-loadvm ${LOADVM_FILE}"
fi
#
# MEMORY_CONFIG : "-m 512" to allocate 512MB for the QEMU instance
#
MEMORY_CONFIG="-m 384"  # to allocated 384MB of memory for QEMU
#
# HDA_CONFIG : "-hda imagefile.img" # which file is ide controller 0, target 0
#
HDA_CONFIG="-hda knoppix.img"

set -x
/tmp/qemu/bin/qemu ${USE_LOCALTIME:+${LOCALTIME_CONFIG}} \
        ${USE_SSH:+${SSH_CONFIG}} \
        ${USE_TELNET:+${TELNET_CONFIG}} \
        ${USE_FTP:+${FTP_CONFIG}} \
        ${USE_FDA:+${FDA_CONFIG}} \
        ${USE_CDROM:+${CDR_CONFIG}} \
        -user-net  \
        ${USE_AUDIO:+${AUDIO_CONFIG}} \
        ${USE_BOOT:+${BOOT_CONFIG}} \
        ${USE_KEYMAP:+${KEYMAP_CONFIG}} \
        ${USE_LOADVM:+${LOADVM_CONFIG}} \
        ${USE_MEMORY:+${MEMORY_CONFIG}} \
        ${USE_HDA:+${HDA_CONFIG}}
stty sane
exit 0

This script sets up to run a qemu session with a 384MB memory segment, has a virtual CDROM that is actually the physical cdrom device in Solaris containing the Knoppix CD. You may also use an ISO image of a disk. "-boot d" tells qemu to boot from the CDROM, as opposed to "-boot c" which would be to boot from the first hard disk. The "-user-net" is a slirp based network config which uses a NE2000 driver interface to the guest OS, to make a connection out of the machine.

SLIRP networking in QEMU

The QEMU process provides a network layer to the Guested OS. As an example, to get Solaris 10 working under QEMU, you have to use the Install Time Update found at the Community Boot Driver ITU Floppy which has a specially modified version of the ni (ne2000) driver written by Masayuki Murayama and modified by Juergen Keil to work in the QEMU guest running Solaris. If you are using the -user-net option, the QEMU process sets up it's own internal DHCP and DNS servers, as well as a default router. The default address for the client is normally 10.0.2.15. This works very well, but because it's a user level process, it's much harder to get back to the virtual machine. The above redir parameters setup access back to the virtual machine, by having QEMU open up listeners on the ports specified, and then redirecting the traffic down through the slirp stack to the guest OS. An annoying side effect of using some internal software I use called inetmenu (which configures the networking correctly at boot time) caused my systems' default name to have a address of 127.0.0.1. What this did was screw up the "source" ip address on the packets entering the QEMU guest OS, which it never responded to. Eventually I figured out that problem, and solved it by doing a "telnet 192.168.30.151 5023" instead of a "telnet 127.0.0.1 5023". The reason this works on most systems is the hostname is usually not tied to localhost or 127.0.0.1. Once I use the real IP address of the Host system, qemu properly translated the packets to the guest OS, and I was able to log in.

Video as part of QEMU

The video config of the Virtual machine is a Cirrus Logic 5446. Under Solaris Xfree86 (Xsun), you can squeeze 1152x900 at 16-bits no problem. Getting it to work with Xorg is a little trickier. Knoppix has no problem driving the video at 1152x900 @ 16-bits.

Audio as part of QEMU

Audio config is a sound blaster 16, which seems to work ok. Though I've noticed better sound quality when using the OSS sound drivers. I think the libSDL is not very well optimized and I haven't had time to recompile it with a better compiler and under Solaris 10.

Another nice feature is the image files are easy to backup. All you need is disk space. I used a copy of a Win98SE image that I had installed to do the XP Home Upgrade. I didn't have any luck booting from the WinXP Home upgrade disk and having it recognize the Win98SE disk (or NT4 or any other valid Windows media I had). However, since you can start the upgrade to XP Home by booting into Win98SE and running the setup off the CDROM image, that's how I successfully got a working Win XP Home image. But QEMU is probably the biggest use of disk space that I have these days as I'm constantly retesting images and such.

Anyone who has jumpstarted a Sun box using Solaris anytime in the last 13 years has relied on the add_install_client script to do all the hard bits, like setting up the inetboot file in /tftpboot, and adding entries to the /etc/ethers and /etc/bootpararms. The add_install_script took a fair amount of the complexity that you might have to do by hand and did it for you, despite the fact that the parameters required to drive add_install_client can give the typing fingers a real workout.

Ever really looked at /etc/bootparams?

There's about a paragraph worth of stuff in there for each system that will get built describing to the client what things it needs to boot and install Solaris. I remember having to setup multiple servers and all the typing that entailed. I seem to recall building a table of systems and using a while; do loop to call add_install_client repeatedly. Not that I'm a bad typist, but why should I type all that crud out manually? I can't recall the number of times I couldn't get a system net-booting because I fat-fingered something like the ether address or path or something. And how would I really know, unless I had been using a command line shell with history features to see what I had really called add_install_client with?

However, bootp is not a great protocol for booting unless it's contained to a single subnet. The BOOTP requests don't pass through routers, and therefore, you needed to have a boot server on each subnet, so that a net-booting client could get it's initial boot load. Once the intial boot load happened, it could then talk to an install server on a different subnet. Not terrible, but it further complicates the issue by requiring a setup for the install server as well as the boot server.

As it turns out, since DHCP configurations can be passed through routers, you no longer need a boot server on each network to PXE or DHCP boot a Solaris system. This allows you to keep all your jumpstart configurations in one place, even if you have multiple jumpstart servers, as is done in large installations or like some of the factories in Sun.

Dawn of DHCP support for jumpstarting

I don't know exactly when this happened. It's only been in the last four or five years because an old Ultra-AXe box I have in the lab won't dhcp boot. (boot net:dhcp - install) The support for DHCP in add_install_client then fundamentally changed the behavior of add_install_client. Now, when you call add_install_client with the appropriate parameters to do a DHCP/PXE jumpstart, what you get is a list of DHCP macros that you have to plug into your local DHCP server, along with the requisite files in /tftpboot to get your client netbooting. If you thought /etc/bootparams was bad, configuring the DHCP macros is substantially worse, especially if you've never configured a DHCP server before. Not to mention, each different DHCP server has it's own syntax (SUN, ISC, Microsoft, etc et al). And it's not just a set of information that you have to configure, but there's also this concept of "supported architectures". Assuming you've been DHCP booting a bunch of V240's and the boss brings a new V440 up to the lab for you to jumpstart. You configure the add_install_client, punch boot net:dhcp - install and *nothing* happens. Well, not in the way you expect. You'll see things like timeouts and fallbacks to bootp perhaps, but it won't boot.

Whiskey-Tango-Foxtrot? Turns out that you have to inform the DHCP server, and all the SYMBOL macros that you now support the V240 and the V440, which requires approximately X lines of macro changes, based on the number of SYMBOL macros you have defined in your DHCP server which supports jumpstarting. A minimum number will be something like 8 and could be as high as 16. You go back and you fix your DHCP tables, and maybe you just go ahead a pre-populate any of the architectures for systems you might be DHCP jumpstarting, remembering to transform the "," to a "." so you dont' have to fool with the SYMBOL macros again. ( uname -i | sed 's/,/./' )

DHCP Macros for jumpstarting

/etc/bootparams looks really good right now, especially if you're fighting the DHCP server and trying to get things working. If you think the pain is not worth the effort, let me suggest the Jumpstart Enterprise Toolkit, aka JET which takes the complexity of jumpstart and put's all the typing and configuration into a single file. Make a typo? Review your template and remake the client. Need to tweak something, modify the template and remake the client. For some people, this is just *too* hard, but if you're like me and you want to figure out how you jumpstarted a system from 6 months ago, this is the tool to use. Do you have to care about DHCP servers when using JET? Nope. Well, unless you aren't running a Sun DHCP server. Do you have to worry about parsing the output of add_install_client? Nope. So where's the rub? Do you really want to become a DHCP/PXE macro guru? Do you think it's more important that you have really cool and complicated macros, or do you want a tool which sets up the client for you? If you fall into the first category, have at it. If all you really want is a framework to help you jumpstart your systems, then JET is the tool for you.

PXE Booting and how it relates to DHCP

When you call add_install_client (either manually or using a front end like JET) for a DHCP configuration, add_install_client sets up the information in /tftpboot, like you would expect to see. However, instead of populating /etc/bootparams, it nows gives you a list of DHCP macros, with values that you need to populate. JET does all the population of the DHCP macros for you (and if you run remove_client, will take them out of the DHCP tables).

The two DHCP macros that are required to PXE boot a system are the BootSrvA and BootFile. add_install_client tells you to put the BootSrvA macro in the PXE macro (PXEClient:Arch:00000:UNDI:002001). For the most part, we have figured out that this is kind of problem. If you happen to have a multihomed jumpstart server, having the BootSrvA macro in the PXE macro means that you can only have one network you boot from. Unless you then populate other macros with a BootSrvA macro, such as a network or client macro. So what's the point? Why bother with the PXE macro??? I think it just complicates the issue and by putting the BootSrvA in the client macro, you control that client. Some folks like to do anonymous jumpstarts where you can come in and plug your laptop and install it without having to do anything else than pinch a network port. From a security perspective, this is a bad idea. This means that at any time, a user could take one of your PC's and install an OS from the network and you don't have any control over it. The second macro required to PXE boot is BootFile, which is the Network Boot Program (nbp). It is currently a real mode binary which is responsible for collecting the rest of the client configuration information from the DHCP server, checking for a custom bootenv.rc file (usually named /tftpboot/01<MAC address>.bootenv.rc) on the tftpboot server and the inetboot file specified by the SbootURI macro. It starts the inetboot binary which then starts the kernel and you are on your way to PXE booting a Solaris X86/X64 system.

Per node bootenv.rc file

In Solaris 9, U6 (04/04), the concept of a per-node bootenv.rc was introduced. This file is created when you pass add_install_client a "-b variable=parameter" construct. If add_install_client sees that flag, it generates a custom bootenv.rc taken from the net-boot image, and applies the changes you specified with the -b flags. In this case, you can do things like disable acpi for the install, set the serial port to be the console, or set the boot-args to "- install nowin" like:

add_install_client [bunch of jumpstart variables] -b acpi-user-options=0x2

or

add_install_client [bunch of jumpstart variables] -b input-device=ttya -b output-device=ttya

or

add_install_client [bunch of jumpstart variables] -b boot-args=\"- install nowin\"

The last variable sets up the boot-args so that when the system boots, you no longer have to hit "2" to do a customized jumpstart, such that an Solaris X86/X64 jumpstart can be completely hands off (at in Solaris 10).

Somehow I just think I've made a bad decision. I just signed up to put my personal thoughts on the corporate sponsored blarg site, and I am not one to sit back idly and keep quiet. I'm sure a bunch of people who know me would agree. I don't know why I've not bothered up to this point. I've been posting on Usenet and other mailing lists for years, and with things like google, everything I might have ever written is kept someplace. Yeah, that makes me really happy. I think back to all the stupid posts when I thought *I* knew it all, had really bad days [the kind that makes people think you didn't take your medication], or was just being argumentative to the point of flames, and just shake my head. At least after all this time, I've realized the futility in arguing with folks who already have their minds made up, and just try to help those willing to ask for help or at least consider an alternative. For the most part that's worked. I've met a bunch of folks in person, both inside and outside of Sun, from my dealings with mailing lists such as the Yahoo Groups SolarisonX86 list and while it may just have been a lunch with a guy in Dallas in the West End, or a long phone call from someone in Canada whom I'd never talk to before who thought I was having a really bad day, I'm happy to call those folks friends. It makes me realize how very lucky I am, despite the fact that we don't have much interpersonal interaction without influences of a sequence of zeroes and ones.

But since I just did my first ever presentation, at CEC 2005 in San Francisco, I realized I'm now officially published with something more than a stream of consiousness response in a public forum. With a deck of slides I put together by myself with Star Office, and the requisite head-banging having to learn the ins and outs of Star Present, formatting help from some co-workers and a fair amount of reviews from peers, it's official now. Obviously, the CEC selection panel must have thought I had something important to talk about. So here I am, jumping into the deep end of the blarg. Great, now I have to think about what I say and how I say it. Thank goodness for "Save as Draft" At least if I'm not talking technical, I can write without having to go check the list of "do's" and "don'ts" to make sure I haven't violated rules of the blarg. (And wondering if my English professor from WVU would approve of my grammar. I have a tendency to run-on...) Unless I'm talking about something like how things work, what management is doing (which by the way, is one of the great reorg-ing mysteries of the world) or the big list of "do nots", I hope I can stay below the blarg police radar.

But, other engineers in the company are doing it, and it's a fairly innocuous way for other folks who don't troll mailing lists in the company to see what guys like me are dealing with from a corporate, technical or social perspective. Another way to get published. And as I sit and write this stuff, I realize I'm now going to have to learn HTML, something I've been avoiding for at least 10 years. What's a systems guy to do? I didn't get a degree in English, Creative Writing or Desktop Publishing. My degree is in Computer Science. The stuff in my head is all very technical and process driven, and up to this point, I've never really considered how to pretty it up on a web page. Heck, it's not like there's 10,000 other pieces of technology that I need to learn for my job. And this past weekend, even mangement has admitted they don't know what an "Engagement Architect" is supposed to do, which is now my official job title having originally been re-hired as a "Systems Engineer" after being RIF'd as a Cluster Services Methodology Developer. :-(

So will anyone really care what I have to say? Probably not. There's a bunch of stuff going on in my life that won't appear here because it's not really an appropriate forum for it. But folks will care about the technical content, which according to the blog stats, is what seems to tweak the public's interest.

CLB: may as well start early.

So what's the point of $7.6B in cash if the GAAP rules won't let you spend it because it comes off an asset? I'm really getting tired of that story, especially because I've come to realize all it is, is "spin". As far as I'm concerned, it's just a virtual carrot. It doesn't really exist other than to serve up as good press for the employees to hear at corporate meetings to believe that the company will stay afloat. The constant reorging, rifs, travel restrictions and expense reduction policies only serve as a warning sign. Yes, this is a technology business and the market changes very quickly. Hardware is more and more becoming commodity, and the JES software licensing at $100/head only validates the point. So where is Sun going? Services seem to be a very good way to bring in recurring revenue of substantial margin, but the mantra we hear is that "we don't want to become IBM GS". I look at their stock price and I have to wonder. The model of recurring revenue streams in services works really well if you get the long term contracts, even if the going rate is less. Having to do a couple of weeks at a time only incurs the overhead of creating, funding, staffing and implementing a project. The only thing the $7.6B in cash probably does is stave off someone buying Sun for $6B in cash and floating a $7.6B loan for the duration of procedures. In that context, I wonder what the future will really bring.

Enough rantings of another pony-tailed geek. Later!