mrj's Weblog

Thursday Jan 21, 2010

The EU clears Oracle's acquisition of Sun Microsystems

The EU press release is here.

Posted at 06:17AM Jan 21, 2010 by mrj in General | Comments[1] | Permalink

Tuesday Jan 19, 2010

Solaris 10 and OpenSolaris on the same zfs root pool

Sharing more info.. Here are some hints for installing both Solaris 10 and OpenSolaris on the same zfs rpool.

I did this on a VirtualBox guest.. I did a fresh install of s10u8 on a zfs root (you need to use the text installer or do a net based install to install on a zfs root).

I then booted an OpenSolaris iso, and after setting my hostname and hostid, ran the create-be script to create a new OpenSolaris BE on the S10's zfs root.

DISCLAIMER: this is totally unsupported by Sun, could mess up your system, etc. etc.

Write down your hostname, hostid, IP addr, netmask, gateway, and NIS domain.

	bash-3.00# uname -a
	SunOS unknown 5.10 Generic_141445-09 i86pc i386 i86pc
	bash-3.00# hostid
	10fa4034
	bash-3.00# echo "hw_serial,0xa?B" | mdb -k
	hw_serial:
	hw_serial:32 38 34 38 33 35 38 39 32 0
	bash-3.00#

Boot your OpenSolaris iso.. Set your hostname and hostid.

        jack@opensolaris:~$ pfexec su -
	root@opensolaris:~# hostname unknown
	root@unknown:~# hostid
	00041f55
	root@unknown:~# echo "hw_serial/v 32 38 34 38 33 35 38 39 32 0" | mdb -kw
	root@unknown:~# hostid
	10fa4034
	root@unknown:~#

Get the create-be script...

        root@opensolaris:~# wget http://blogs.sun.com/mrj/resource/create-be
        root@opensolaris:~# chmod a+x create-be

Import your Solaris 10 rpool... Create the OpenSolaris BE (we'll need to manually create the menu.lst entry later since the script doesn't handle s10 menu.lst entries right now.

        root@opensolaris:~# zpool import rpool
        root@opensolaris:~# /root/create-be --build=129 --bename=osol129

Add your new BE into your menu.lst. e.g. here is my entry.

  title osol129
  findroot (pool_rpool,0,a)
  bootfs rpool/ROOT/snv129
  kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS
  module$ /platform/i86pc/$ISADIR/boot_archive

Install a newer version of grub

        root@opensolaris:~# /mnt/sbin/installgrub /mnt/boot/grub/stage1 /mnt/boot/grub/stage2 /dev/rdsk/c0d0s0

Now, reboot into your OpenSolaris BE, configure it, and look around. You can reboot back into s10 at any time...

root@unknown:~# uname -a
SunOS unknown 5.11 snv_129 i86pc i386 i86pc Solaris
root@unknown:~# 

root@unknown:~# zfs list
NAME                        USED  AVAIL  REFER  MOUNTPOINT
rpool                      10.4G  28.8G    40K  /rpool
rpool/ROOT                 8.37G  28.8G    21K  legacy
rpool/ROOT/s10x_u8wos_08a  3.67G  28.8G  3.67G  /
rpool/ROOT/snv129          4.70G  28.8G  4.70G  /mnt
rpool/dump                 1.00G  28.8G  1.00G  -
rpool/export                 44K  28.8G    23K  /export
rpool/export/home            21K  28.8G    21K  /export/home
rpool/swap                    1G  29.8G    16K  -
root@unknown:~# 

root@unknown:~# beadm list
BE             Active Mountpoint Space Policy Created          
--             ------ ---------- ----- ------ -------          
s10x_u8wos_08a R      -          3.67G static 2010-01-13 12:02 
snv129         N      /          4.70G static 2010-01-14 14:24 
root@unknown:~# beadm activate s10x_u8wos_08a
root@unknown:~# reboot

...

bash-3.00# uname -a
SunOS unknown 5.10 Generic_141445-09 i86pc i386 i86pc
bash-3.00# zfs list
NAME                        USED  AVAIL  REFER  MOUNTPOINT
rpool                      10.4G  28.8G    40K  /rpool
rpool/ROOT                 8.37G  28.8G    21K  legacy
rpool/ROOT/s10x_u8wos_08a  3.67G  28.8G  3.67G  /
rpool/ROOT/snv129          4.70G  28.8G  4.70G  /mnt
rpool/dump                 1.00G  28.8G  1.00G  -
rpool/export                 44K  28.8G    23K  /export
rpool/export/home            21K  28.8G    21K  /export/home
rpool/swap                    1G  29.8G    16K  -
bash-3.00#

Posted at 10:09AM Jan 19, 2010 by mrj in General | Comments[1] | Permalink

Friday Jan 15, 2010

Solaris 10 & OpenSolaris p2p, p2v, v2p

Solaris has always been one of the easier OSes to move between machines. When we would try to boot Solaris on a newly prototyped SPARC machine (back in the day when I worked in the SPARC H/W org), we would install a Solaris image on an older SPARC machine, and then fix it up so that it could boot on the prototype. Once OBP could load unix, it usually took us a day or two to work through the H/W, FPGA, OBP, and Solaris bugs so we could get to multiuser login.

Now that we have zfs root, it's more difficult, and at the same time, much more powerful. zfs stores platform specific boot information in it's meta data which isn't easily accessed, making it more difficult. But, zfs supports live snapshots which makes it much more powerful.

With zfs, we can "easily move" the machine from one "machine" to another. This generally applies to S10 and Opensolaris, as long as it's running a zfs root.. You can go from a physical machine, VirtualBox guest, xVM guest, etc. to a different physical machine, VirtualBox guest, xVM guest, etc.

For an example, I thought I would share some tricks on how you can transform a x86 box running OpenSolaris to a VirtualBox guest without ever shutting down or rebooting the x86 box.

The first thing you want to do is boot a new VirtualBox guest with an OpenSolaris live install iso. You want to make sure that the zfs version in the install iso matches the zfs version on your x86 box.

Once you have booted the install iso, open a shell. Enable ssh, and then run format. Write down the disk your going to use (e.g. c4t0d0s0), then run fdisk from within format. Usually you will create a single disk partition for Solaris here.. Exit fdisk saving your changes.

Now partition your Solaris disk. Select the 0 partition, set the tag to "root" (without the quotes), and set the range from 1 to the last cylinder. Make sure partition 8 is set to 0 - 0 cylinders, then label the disk and exit format.

On your x86 box, write down your hostname and hostid info.

	: core2[1]#; hostname
        core2
	: core2[1]#; hostid
	05bdb9c2
	: core2[1]#; echo "hw_serial,0xa?B" | mdb -k
	hw_serial:
	hw_serial: 39 36 33 31 39 39 33 38 0 0

On your VirtualBox guest, update hostname and hostid to match your x86 system.

	root@opensolaris:~# hostname core2
	root@core2:~# hostid
	00041f55
	root@core2:~# echo "hw_serial/v 39 36 33 31 39 39 33 38 0 0" | mdb -kw
	root@core2:~# hostid
	05bdb9c2
	root@core2:~#

Now, create the zfs root on the VirtualBox guest using the disk you saw in format. Make sure that you create the zpool on slice 0 (s0). If your moving multiple pools, you'll probably want to setup multiple pools on the guest now too.

   zpool create -R /a -f rpool /dev/dsk/c4t0d0s0

Back to the x86 system, snapshot the root pool, then send it to the opensolaris guest (which is 192.168.0.117 in my example). Do the same for all pools you want to move.

   zfs snapshot -r rpool@p2v
   zfs send -R rpool@p2v | ssh jack@192.168.0.117 pfexec /usr/sbin/zfs receive -dF rpool

Once this completes, if the x86 system is actively being used, you'll want to shut down the apps your using (e.g. databases, etc.), take a snapshot again, then do a differencing zfs send to do a final sync.

Back on the VirtualBox guest, lets finalize the disk. Set bootfs to the BE you want to boot.

   zpool set bootfs='rpool/ROOT/--your-bootfs--' rpool

On the VirtualBox guest, install grub

   /a/sbin/installgrub /a/boot/grub/stage1 /a/boot/grub/stage2 /dev/rdsk/c4t0d0s0

If your NICs are different, you need to update them. If you have hostname.--nic-- and dhcp.--nic-- files, update them to point to your new NIC(s). You may have neither. Or you may only have a hostname.--nic--. You may also have to update /a/etc/nwam/llp. For multiple BEs, don't forget to update the NICs in all the BEs you want to boot.

   devfsadm -r /a -i e1000g
   mv /a/etc/hostname.--oldnic-- /a/etc/hostname.e1000g0
   mv /a/etc/hostname.--oldnic-- /a/etc/dhcp.e1000g0

If your using the same IP, you may want to take your x86 box off the net now... Eject the CDROM and reboot your VirtualBox guest... Hopefully it booted right up :-)

Posted at 12:21PM Jan 15, 2010 by mrj in Solaris | Comments[4] | Permalink

Thursday Dec 17, 2009

Switching from Nevada to OpenSolaris

I've fixed another bug (findroot grub menu entry) in the create-be script and added support for additional zpools (i.e. along with rpool, other zpools should now work seamlessly across Nevada and OpenSolaris BEs. NOTE: for additional zpools, the nevada and OpenSolaris BEs should be the exact same build).

The scripts lets you create a new (non COW) OpenSolaris BE on a nevada zfs root based system (or OpenSolaris system). You can use this to transition a Nevada zfs root based system to OpenSolaris. You can also choose to install an arbitrary OpenSolaris build (i.e. if you want to downgrade).

DISCLAIMER: this is totally unsupported by Sun, could mess up your system, etc. etc.

I strongly recommend creating a scratch BE to run this script out of, in case something goes wrong.

You can grab an updated copy here. Remember the DISCLAIMER above.. Back up your data first!

For folks inside of SWAN, here's a cheatsheet...

Write down your IP addr, netmask, gateway, NIS domain

Create & switch to a scratch BE

# lucreate -n scratch-be
# luactivate scratch-be
# init 6

Install the new OpenSolaris BE (this is not an upgrade, it's a fresh install into a BE on the same rpool as your current BEs)

# pkgadd -d /net/girltalk2/export/mrj/pkg-gate/packages/i386/ \
  SUNWipkg SUNWpython-ply SUNWpython-pycurl
# wget http://blogs.sun.com/mrj/resource/create-be
# chmod a+x create-be
# /root/create-be --build=129 --bename=osol129 --repo=http://ipkg.sfbay/dev --menu="osol129"

If you want to install additional software, i.e. setup a build machine

# zfs set mountpoint="/mnt" rpool/ROOT/osol129
# zfs mount rpool/ROOT/osol129
# pkg -R /mnt set-publisher -O http://ipkg.sfbay/extra extra
# pkg -R /mnt install developer/opensolaris/osnet@0.5.11-0.129
# zfs umount rpool/ROOT/osol129

If you want to be able to build xvm-gate

# zfs set mountpoint="/mnt" rpool/ROOT/osol129
# zfs mount rpool/ROOT/osol129
# pkg -R /mnt install \
  SUNWgmake@3.81-0.129 \
  SUNWbcc@0.16.17-0.129 \
  SUNWgnu-readline@5.2-0.129 \
  SUNWxwinc@0.5.11-0.129 \
  SUNWgnome-common-devel@0.5.11-0.129 \
  SUNWlibtool@1.5.22-0.129 \
  SUNWgnu-automake-110@1.10-0.129 \
  SUNWaconf@2.63-0.129 \
  SUNWgit@1.5.6.5-0.129 \
  SUNWxvm@3.3.2-0.129
# zfs umount rpool/ROOT/osol129

Reboot into your new be, configure the network, etc. migrate over other BE settings, e.g. sshd config.

# bootadm list-menu
# bootadm set-menu default=....your-osol129-menu-number....
# reboot
: run through sysconfig, reboot
: login
# beadm mount ....your-old-be.... /mnt
# cp /mnt/etc/ssh/sshd_config /etc/ssh/
# cp /mnt/etc/ssh/*key* /etc/ssh/
# svcadm refresh ssh;svcadm restart ssh
: migrate over other customizations you might have
# beadm umount ....your-old-be....

Posted at 10:00AM Dec 17, 2009 by mrj in General | Comments[1] | Permalink

Thursday Dec 03, 2009

Updated create-be

I've fixed a few bugs in the create-be script. The scripts lets you create a new (non COW) OpenSolaris BE on a nevada zfs root based system or OpenSolaris system. You can use this to transition a Nevada zfs root based system to OpenSolaris. You can also choose an arbitrary OpenSolaris build (i.e. if you want to downgrade).

DISCLAIMER: this is totally unsupported by Sun, could mess up your system, etc. etc.

You can grab an updated copy here.

create-be --build=128a --bename=osol128 --repo=http://pkg.opensolaris.org/dev

Posted at 09:53AM Dec 03, 2009 by mrj in Solaris | Comments[2] | Permalink

Saturday Nov 21, 2009

Install Nevada and OpenSolaris on the same zfs rpool

I wrote a relatively simple python script, here, which will allow you to install OpenSolaris on an existing Nevada, zfs root based x86 system. They will use different root filesystems (e.g. rpool/ROOT/*) of course, but will share the other filesystems (e.g. /export).

DISCLAIMER: this is totally unsupported by Sun, could mess up your system, etc. etc. It works great for me.. But may not for you. :-)

You can also use this script on an OpenSolaris system to install a new Boot Environment (BE) from scratch. When you do a beadm create, OpenSolaris will create a copy-on-write (COW) clone of the current filesystem (which is what you want normally). There are cases where you may not want to clone your existing config. e.g. If you really messed up your system configuration, you can re-install into a new BE (allows you to keep the old config around) instead of re-installing from scratch. Or you can use this to install an older build in a new BE. e.g. say you need to test something in an older build and you didn't keep any of your older BEs around. Here's an example invocation...

create-be --build=121 --bename=osol-121 --repo=http://pkg.opensolaris.org/dev

To install OpenSolaris on a Nevada system, you first need to install the OpenSolaris packaging system on your Nevada system. I'm not going to go over how to build the ipkg gate, but this should get you going in the right direction if you can't find binary packages.

hg clone ssh://anon@hg.opensolaris.org/hg/pkg/gate pkg-gate
** build everything
cd packages
pkgadd -d . SUNWipkg SUNWpython-ply SUNWpython-pycurl

You need to be using a zfs root of course.. i.e.

root@pico:~# df /
Filesystem           1K-blocks      Used Available Use% Mounted on
rpool/ROOT/snv_127 ...

Now use the script to create a new OpenSolaris BE on your nevada rpool, giving it a BE name, and tell it to create a grub menu entry. This will take a while :-).. You can override the default build (--build=) or point to a different repo (--repo=)

create-be --bename=osol-127 --menu="OpenSolaris b127"

This script does a fresh install... So you will have to reconfigure the system when you reboot into the new BE (the current BE is unaffected). The only settings I presently migrate over is the grub menu entry and the console settings in bootenv.rc.

Once the install completes (repeats the "takes a while"), you can now reboot into the OpenSolaris BE.. A note here.. you should update the grub menu so the default menu entry is the new entry which was added. This is because you will reboot into the new BE, configure it, and it will reboot again after it completes configuring the OpenSolaris BE. fastreboot will then boot into the original BE if you don't change the default grub menu entry. Also, make sure you do a reboot -p!!!!

-bash-4.0# bootadm list-menu
the location for the active GRUB menu is: /rpool/boot/grub/menu.lst
default 0
timeout 10
0 Solaris Express Community Edition snv_127 X86
1 Solaris failsafe
2 OpenSolaris b127
-bash-4.0# bootadm set-menu default=2

Now that you have booted and configured your OpenSolaris BE, lets poke around... run beadm list to make sure you actually booted into the right BE first :-). NOTE: OpenSolaris thinks the Nevada BE is an opensolaris BE which works out nice... I like to copy over my ssh keys since I use the same IP address for the nevada BE and OpenSolaris BE. I'll also create a new OpenSolaris BE using beadm..

root@pico:/etc/ssh# beadm list
BE       Active Mountpoint Space Policy Created          
--       ------ ---------- ----- ------ -------          
osol-127 NR     /          3.79G static 2009-11-20 20:03 
snv_127  -      -          7.97G static 2009-11-19 10:48
root@pico:/etc/ssh# beadm mount snv_127 /mnt
root@pico:/etc/ssh# cp /mnt/etc/ssh/*key* /etc/ssh/
root@pico:/etc/ssh# cp /mnt/etc/ssh/sshd_config /etc/ssh/
root@pico:/etc/ssh# beadm umount snv_127
root@pico:/etc/ssh# svcadm refresh ssh
root@pico:/etc/ssh# svcadm restart ssh
root@pico:/etc/ssh# beadm create test-osol-be
root@pico:/etc/ssh# beadm list
BE           Active Mountpoint Space Policy Created          
--           ------ ---------- ----- ------ -------          
osol-127     NR     /          3.79G static 2009-11-20 20:03 
snv_127      -      -          7.97G static 2009-11-19 10:48 
test-osol-be -      -          51.0K static 2009-11-21 06:54

Finally, lets reboot back into nevada and look around (I love fast reboot).

root@pico:/etc/ssh# beadm activate snv_127
root@pico:/etc/ssh# reboot
Nov 21 06:57:26 pico reboot: initiated by root on /dev/console
syncing file systems... done
rebooting...
SunOS Release 5.11 Version snv_127 32-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: pico
Reading ZFS config: done.
Mounting ZFS filesystems: (7/7)

pico console login: root
Password: 
Nov 21 06:59:06 pico login: ROOT LOGIN /dev/console
Last login: Sat Nov 21 05:10:55 on console
Sun Microsystems Inc.   SunOS 5.11      snv_127 November 2008
-bash-4.0# beadm list
-bash: beadm: command not found
-bash-4.0# lustatus
ERROR: No boot environments are configured on this system
ERROR: cannot determine list of all boot environment names
-bash-4.0# zfs list
NAME                      USED  AVAIL  REFER  MOUNTPOINT
rpool                    13.2G   132G  35.5K  /rpool
rpool/ROOT               11.8G   132G    21K  legacy
rpool/ROOT/osol-127      3.80G   132G  3.79G  /
rpool/ROOT/snv_127       7.97G   132G  7.97G  /
rpool/ROOT/test-osol-be    51K   132G  3.79G  /
rpool/dump                959M   132G   959M  -
rpool/export               44K   132G    23K  /export
rpool/export/home          21K   132G    21K  /export/home
rpool/swap                512M   133G  9.05M  -
-bash-4.0# df /
/                  (rpool/ROOT/snv_127):277843413 blocks 277843413 files
-bash-4.0#

Posted at 06:21AM Nov 21, 2009 by mrj in General | Permalink

Tuesday Sep 15, 2009

Installing a small xVM or VirtualBox OpenSolaris guest

I have a little python script that I use for for personal use to do some small OpenSolaris guest installs. I use it for VirtualBox and xVM guests, but it should work fine for metal too assuming you add the correct drivers needed for your system.

It's a simple text based installer. Now, I'm not much of a Python coder, the script is something I play around with in my spare time, it's not finished, and it's not supported by Sun, etc, etc.. :-) But I though some folks would find it useful so I'm sharing it

A 2009.06 based xVM install is something around 380M vs 3G+. b122 is quite a bit bigger due to some dependency bloat. For a xVM guest, boot the 2009.06 iso, login as jack, grab the installer and run it...

: core2[1]#; virt-install -n opensolaris -r 1024 -p --nographics -l /net/192.168.0.71/tank/isos/solaris/os2009.06.iso -f /vdisks/opensolaris 
[CUT]
opensolaris console login: jack
Password: 
Last login: Tue Sep 15 05:19:57 from core2.lan
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
jack@opensolaris:~$ wget http://blogs.sun.com/mrj/resource/slim-guest-installer
[CUT]
05:28:04 (69.94 KB/s) - `slim-guest-installer' saved [16096/16096]
jack@opensolaris:~$ chmod a+x slim-guest-installer 
jack@opensolaris:~$ pfexec ./slim-guest-installer

Thanks for choosing to install the OpenSolaris OS! Before you start, review
the Release Notes for this release for a list of known problems. The release
notes can be found at
   http://opensolaris.org/os/project/indiana/resources/relnotes/200906/x86

****
NOTICE: THIS INSTALLER ONLY SUPPORTS INSTALLING TO A WHOLE DISK. ALL DATA
ON THE DISK YOU INSTALL TO WILL BE DESTROYED.
****

Please Select Install Disk

  AVAILABLE DISK SELECTIONS:
	0.  /dev/dsk/c7t0d0p0  21459755520 bytes
Specify disk (enter its number or 'q' to quit): 0

NOTE: ALL DATA ON THIS DISK WILL BE DESTROYED.

Install on /dev/rdsk/c7t0d0p0 (yes or no): yes
Configuring ZFS Root:................ COMPLETE
Installing packages...
DOWNLOAD                                    PKGS       FILES     XFER (MB)
SUNWopenssl                                39/67   6042/8542   57.27/93.32
[CUT]

For VirtualBox, boot the 2009.06 iso, login as jack, grab the installer and run it with an additional option (--profile=vbox-guest) to specify virtualBox packages.

opensolaris console login: jack
Password: 
Last login: Tue Sep 15 05:19:57 from core2.lan
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
jack@opensolaris:~$ wget http://blogs.sun.com/mrj/resource/slim-guest-installer
[CUT]
05:28:04 (69.94 KB/s) - `slim-guest-installer' saved [16096/16096]
jack@opensolaris:~$ chmod a+x slim-guest-installer 
jack@opensolaris:~$ pfexec ./slim-guest-installer --profile=vbox-guest

Have Fun!

Posted at 12:50PM Sep 15, 2009 by mrj in General | Permalink

Monday Aug 17, 2009

How to do a fresh install into a BE on OpenSolaris

Lately, I've been playing around with smaller OpenSolaris guests. Not wanting to muck around with setting up an AI installer, I ended up doing a "fresh install" into a new Boot Environment (BE). Although none of this is overly practical, it is interesting.. I thought I would share what I learned..

This works just as well on metal, in a VirtualBox OpenSolaris guest, and in a xVM OpenSolaris guest. For this example, I'll run through an xVM OpenSolaris guest. You'll need to change your package list slightly for the other ones.

First, a little off subject, lets install a fresh OpenSolaris guest.. This assumes your already running a OpenSolaris dom0.

: core2[1]#; virt-install -n opensolaris -r 1024  -p --nographics \
--noautoconsole -l /net/192.168.0.71/tank/isos/solaris/os2009.06.iso \
-f /vdisks/opensolaris -s 20

Starting install...
Retrieving file unix...   100% |=========================| 1.4 MB    00:00
Retrieving file x86.micro 100% |=========================|  36 MB    00:01
Creating storage file...  100% |=========================|   20 B    00:00
Creating domain...                                                 0 B 00:05
Domain installation still in progress. You can reconnect to
the console to complete the installation process.
: core2[1]#; virsh console opensolaris
v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b'
SunOS Release 5.11 Version snv_111b 32-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: opensolaris
Remounting root read/write
Probing for device nodes ...
Preparing live image for use
Done mounting Live image
USB keyboard
 1. Albanian                      23. Lithuanian
 2. Belarusian                    24. Latvian
 3. Belgian                       25. Macedonian
 4. Brazilian                     26. Malta_UK
 5. Bulgarian                     27. Malta_US
 6. Canadian-Bilingual            28. Norwegian
 7. Croatian                      29. Polish
 8. Czech                         30. Portuguese
 9. Danish                        31. Russian
10. Dutch                         32. Serbia-And-Montenegro
11. Finnish                       33. Slovenian
12. French                        34. Slovakian
13. French-Canadian               35. Spanish
14. Hungarian                     36. Swedish
15. German                        37. Swiss-French
16. Greek                         38. Swiss-German
17. Icelandic                     39. Traditional-Chinese
18. Italian                       40. TurkishQ
19. Japanese-type6                41. TurkishF
20. Japanese                      42. UK-English
21. Korean                        43. US-English
22. Latin-American
To select the keyboard layout, enter a number [default 43]:

 1. Arabic
 2. Chinese - Simplified
 3. Chinese - Traditional
 4. Czech
 5. Dutch
 6. English
 7. French
 8. German
 9. Greek
10. Hebrew
11. Hungarian
12. Indonesian
13. Italian
14. Japanese
15. Korean
16. Polish
17. Portuguese - Brazil
18. Russian
19. Slovak
20. Spanish
21. Swedish
To select desktop language, enter a number [default is 6]:
User selected: English
Configuring devices.
Mounting cdroms
Reading ZFS config: done.

opensolaris console login: jack
Password:
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
jack@opensolaris:~$
jack@opensolaris:~$ Aug 17 02:58:54 opensolaris in.routed[696]: route 0.0.0.0/8 --> 0.0.0.0 nexthop is not directly connected

jack@opensolaris:~$ ifconfig xnf0
xnf0: flags=1004843 mtu 1500 index 2
        inet 192.168.0.147 netmask ffffff00 broadcast 192.168.0.255
jack@opensolaris:~$
jack@opensolaris:~$ 
(^] to exit console)
: core2[1]#; /usr/lib/xen/bin/xenstore-ls | grep passwd
     passwd = "EJzbFnyg"
: core2[1]#;

Connect a vncviewer to the OpenSolaris guest, using the vnc password above.

: core2[1]#; vncviewer 192.168.0.147:0 &> /dev/null &

Once you've completed the OpenSolaris install, lets disable gdm and intrd (I like to do this for small guests), and apply the workaround for a zfs bug (6840704 osol_0906 PV guests sometimes hang at login prompt). Then lets create a working BE. If we mess anything up, we can always go back to where we started from.

: core2[1]#; virsh start opensolaris;virsh console opensolaris
Domain opensolaris started

v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b'
SunOS Release 5.11 Version snv_111b 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: opensolaris
Configuring devices.
Loading smf(5) service descriptions: 150/150
svccfg import warnings. See /var/svc/log/system-manifest-import:default.log .
Reading ZFS config: done.
Mounting ZFS filesystems: (6/6)
Creating new rsa public/private host key pair
Creating new dsa public/private host key pair

opensolaris console login: myuser
Password:
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
myuser@opensolaris:~$ pfexec su -
Aug 17 10:58:54 opensolaris su: 'su root' succeeded for myuser on /dev/console
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
root@opensolaris:~# svcadm disable gdm
root@opensolaris:~# svcadm disable intrd
root@opensolaris:~# echo -e "\n"\
"forceload: drv/domcaps\n"\
"forceload: drv/xencons\n"\
"forceload: drv/xenbus\n"\
"forceload: drv/balloon\n"\
"forceload: drv/evtchn\n"\
"forceload: drv/privcmd\n"\
"forceload: drv/xdf\n"\
"forceload: drv/xnf\n\n" >> /etc/system 
root@opensolaris:~# bootadm update-archive
updating //platform/i86pc/boot_archive
updating //platform/i86pc/amd64/boot_archive
root@opensolaris:~# beadm create snv111b
root@opensolaris:~# beadm activate snv111b
root@opensolaris:~# reboot
Aug 17 11:36:01 opensolaris reboot: initiated by myuser on /dev/console
syncing file systems... done
rebooting...
v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b'
SunOS Release 5.11 Version snv_111b 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: opensolaris
Reading ZFS config: done.
Mounting ZFS filesystems: (6/6)

opensolaris console login:

Now lets get to the point of this blog entry. Lets do a fresh install to a new BE.

When you do a beadm create, you are creating a copy-on-write(COW) based clone of your current root. We don't want that though.. We want an empty root directory to start with.. So we'll create it by hand.. We'll need a uuid. You can write a little program using libuuid(3LIB) or just make one up. Also, we'll put the mountdir to a temporary location during the install.

opensolaris console login: myuser
Password:
Last login: Mon Aug 17 11:33:40 on console
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
myuser@opensolaris:~$ pfexec su -
Aug 17 11:39:04 opensolaris su: 'su root' succeeded for myuser on /dev/console
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
root@opensolaris:~# zfs create rpool/ROOT/small-be
root@opensolaris:~# zfs set canmount=noauto rpool/ROOT/small-be
root@opensolaris:~# zfs set mountpoint="/mnt" rpool/ROOT/small-be
root@opensolaris:~# zfs set org.opensolaris.libbe:uuid=f0fa607f-7d1c-66ca-caf9-e04cbf rpool/ROOT/small-be
root@opensolaris:~# zfs mount rpool/ROOT/small-be

Next, we'll setup a new packaging environment in the new "BE", install a custom set of packages, seed SMF, setup vfstab and /dev, cleanup some OpenSolaris cruft, setup the new BE to prompt for configuration on the next boot, and then apply the workaround for the zfs bug mentioned above.

root@opensolaris:~# export ROOTDIR=/mnt
root@opensolaris:~# pkg image-create -f -F -a opensolaris.org=http://pkg.opensolaris.org/ $ROOTDIR
root@opensolaris:~# export PKGS="entire \
        SUNWcsd \
        SUNWcs \
        SUNWcarx \
        SUNWcakrx \
        SUNWos86r \
        SUNWkvm \
        SUNWrmodr \
        SUNWpsdcr \
        SUNWpsdir \
        SUNWcnetr \
        SUNWesu \
        SUNWkey \
        SUNWuprl \
        SUNWkrb \
        SUNWbip \
        SUNWzfskr \
        SUNWbash \
        SUNWipf \
        SUNWbash \
        SUNWgrub \
        SUNWtoo \
        SUNWbind \
        SUNWrcmdc \
        SUNWmkcd \
        SUNWPython \
        SUNWPython-extra \
        SUNWipkg \
        SUNWinstall \
        SUNWbeadm \
        SUNWadmap \
        SUNWadmlib-sysid \
        SUNWadmr"
root@opensolaris:~# pkg -R $ROOTDIR install $PKGS
DOWNLOAD                                    PKGS       FILES     XFER (MB)
Completed                                  67/67   8542/8542   93.32/93.32

PHASE                                        ACTIONS
Install Phase                            14726/14726
PHASE                                          ITEMS
Reading Existing Index                           8/8
Indexing Packages                              67/67
Optimizing Index...
PHASE                                          ITEMS
Indexing Packages                              67/67
root@opensolaris:~# rm -rf $ROOTDIR/var/pkg/download/*
root@opensolaris:~# /usr/bin/cp $ROOTDIR/lib/svc/seed/global.db $ROOTDIR/etc/svc/repository.db
root@opensolaris:~# chmod 600 $ROOTDIR/etc/svc/repository.db
root@opensolaris:~# cd $ROOTDIR/var/svc/profile/
root@opensolaris:/mnt/var/svc/profile# ln -s generic_limited_net.xml generic.xml
root@opensolaris:/mnt/var/svc/profile# ln -s ns_files.xml name_service.xml
root@opensolaris:/mnt/var/svc/profile# cd
root@opensolaris:~# cp /etc/vfstab $ROOTDIR/etc/vfstab
root@opensolaris:~# /usr/sbin/devfsadm -R $ROOTDIR
root@opensolaris:~# echo -e "/lib/svc/method/sshd\n\
/usr/sbin/sysidkbd\n\
/usr/sbin/sysidpm\n\
/lib/svc/method/net-nwam\n\
/usr/lib/cc-ccr/bin/eraseCCRRepository" > $ROOTDIR/etc/.sysidconfig.apps
root@opensolaris:~# /usr/sbin/sys-unconfig -R $ROOTDIR
sys-unconfig started Mon Aug 17 12:32:56 2009
rm: cannot remove `/mnt/etc/vfstab.sys-u': No such file or directory
grep: /mnt/etc/dumpadm.conf: No such file or directory
sys-unconfig completed Mon Aug 17 12:32:56 2009
root@opensolaris:~# cat $ROOTDIR/etc/passwd | sed '/^jack/d' > $ROOTDIR/etc/passwd.new;mv -f $ROOTDIR/etc/passwd.new $ROOTDIR/etc/passwd
root@opensolaris:~# cat $ROOTDIR/etc/shadow | sed '/^jack/d' > $ROOTDIR/etc/shadow.new;mv -f $ROOTDIR/etc/shadow.new $ROOTDIR/etc/shadow
root@opensolaris:~# cat $ROOTDIR/etc/user_attr | sed 's/^root::::type=role;/root::::/g' > $ROOTDIR/etc/user_attr.new;mv -f $ROOTDIR/etc/user_attr.new $ROOTDIR/etc/user_attr
root@opensolaris:~# echo -e "\n"\
"forceload: drv/domcaps\n"\
"forceload: drv/xencons\n"\
"forceload: drv/xenbus\n"\
"forceload: drv/balloon\n"\
"forceload: drv/evtchn\n"\
"forceload: drv/privcmd\n"\
"forceload: drv/xdf\n"\
"forceload: drv/xnf\n\n" >> $ROOTDIR/etc/system
root@opensolaris:~# /usr/sbin/bootadm update-archive -R $ROOTDIR
updating /mnt//platform/i86pc/boot_archive
updating /mnt//platform/i86pc/amd64/boot_archive

Now lets unmount our new BE, and setup the correct mountpoint.

root@opensolaris:~# zfs umount rpool/ROOT/small-be
root@opensolaris:~# zfs set mountpoint="/" rpool/ROOT/small-be
root@opensolaris:~# beadm list
BE          Active Mountpoint Space   Policy Created
--          ------ ---------- -----   ------ -------
opensolaris -      -          4.01M   static 2009-08-17 10:05
small-be    -      -          357.76M static 2009-08-17 11:49
snv111b     NR     /          3.16G   static 2009-08-17 11:39
root@opensolaris:~#

Time to switch to our new BE and run through the configure.

root@opensolaris:~# beadm activate small-be
root@opensolaris:~# beadm list
BE          Active Mountpoint Space   Policy Created
--          ------ ---------- -----   ------ -------
opensolaris -      -          4.01M   static 2009-08-17 10:05
small-be    R      -          357.76M static 2009-08-17 11:49
snv111b     N      /          3.16G   static 2009-08-17 11:39
root@opensolaris:~# reboot
Aug 17 12:38:24 opensolaris reboot: initiated by myuser on /dev/console
syncing file systems... done
rebooting...
v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b'
SunOS Release 5.11 Version snv_111b 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: unknown
Configuring devices.
Loading smf(5) service descriptions: 78/78
Reading ZFS config: done.
Mounting ZFS filesystems: (8/8)


What type of terminal are you using?
 1) ANSI Standard CRT
[CUT]
Configuring network interface addresses: xnf0.
System identification is completed.

unknown console login: root
Password:
Aug 17 09:45:08 unknown login: ROOT LOGIN /dev/console
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
root@unknown:~#

We went from around 3G to around 370M for disk footprint using the custom set of packages above. Not too bad, but we can improve this over time.

Filesystem            kbytes    used   avail capacity  Mounted on
rpool/ROOT/small-be  20514816  373259 15924049     3%    /
rpool/ROOT/snv111b   20514816 3005343 15920934    16%    /mnt

Now lets create a COW based clone of out new BE and switch to it.


root@unknown:~# beadm list
BE          Active Mountpoint Space   Policy Created
--          ------ ---------- -----   ------ -------
opensolaris -      -          4.01M   static 2009-08-17 07:05
small-be    NR     /          364.43M static 2009-08-17 08:49
snv111b     -      -          3.17G   static 2009-08-17 08:39
root@unknown:~# bootadm update-archive
updating //platform/i86pc/boot_archive
updating //platform/i86pc/amd64/boot_archive
root@unknown:~# beadm create small-be-clone
root@unknown:~# beadm activate small-be-clone
root@unknown:~# beadm list
BE             Active Mountpoint Space   Policy Created
--             ------ ---------- -----   ------ -------
opensolaris    -      -          4.01M   static 2009-08-17 07:05
small-be       N      /          19.5K   static 2009-08-17 08:49
small-be-clone R      -          364.55M static 2009-08-17 09:56
snv111b        -      -          3.17G   static 2009-08-17 08:39
root@unknown:~#

root@unknown:~# reboot
Aug 17 09:57:20 unknown reboot: initiated by root on /dev/console
syncing file systems... done
rebooting...
v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b'
SunOS Release 5.11 Version snv_111b 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: unknown
Reading ZFS config: done.
Mounting ZFS filesystems: (9/9)

unknown console login: root
Password:
Aug 17 13:46:34 unknown login: ROOT LOGIN /dev/console
Last login: Mon Aug 17 13:44:19 on console
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
root@unknown:~#
root@unknown:~# beadm list
BE             Active Mountpoint Space   Policy Created
--             ------ ---------- -----   ------ -------
opensolaris    -      -          4.01M   static 2009-08-17 10:05
small-be       -      -          3.78M   static 2009-08-17 11:49
small-be-clone NR     /          420.45M static 2009-08-17 12:56
snv111b        -      -          3.17G   static 2009-08-17 11:39

From here, we are going to try two different things.. The first thing we want to try is to make sure we can delete the new BE and it's clone. We also want to get a little crazy and see if we can get rid of the original opensolaris and snv111b snapshot. But before we continue on, lets snapshot the vdisk so we can rollback to this point so we don't need to install from scratch again.

root@unknown:~# poweroff
Aug 17 13:47:20 unknown poweroff: initiated by root on /dev/console
syncing file systems... done
: core2[1]#; vdiskadm -u xvm snapshot /vdisks/opensolaris@pre-destroy 
: core2[1]#; virsh start opensolaris;virsh console opensolaris

Lets remove the small-be-clone clone and the small-be BE. For small-be, since we created this by hand, we will want to remove it by hand.

opensolaris console login: root
Password:
Last login: Mon Aug 17 11:46:42 on console
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
root@unknown:~# beadm activate snv111b
root@unknown:~# reboot
Aug 17 09:01:18 unknown reboot: initiated by root on /dev/console
syncing file systems... done
rebooting...
v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b'
SunOS Release 5.11 Version snv_111b 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: opensolaris
Reading ZFS config: done.
Mounting ZFS filesystems: (9/9) 

opensolaris console login: myuser
Password:
Last login: Mon Aug 17 11:46:42 on console
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008 
myuser@opensolaris:~$ pfexec su -
Aug 17 13:10:19 opensolaris su: 'su root' succeeded for myuser on /dev/console
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
root@opensolaris:~# beadm destroy small-be-clone
Are you sure you want to destroy small-be-clone? This action cannot be undone(y/[n]): y
root@opensolaris:~# zfs destroy rpool/ROOT/small-be
root@opensolaris:~# beadm list
BE          Active Mountpoint Space Policy Created
--          ------ ---------- ----- ------ -------
opensolaris -      -          4.01M static 2009-08-17 10:05
snv111b     NR     /          3.18G static 2009-08-17 11:39
root@opensolaris:~#

Now, lets rollback the vdisk and try the second part of our test.

root@unknown:~# poweroff
Aug 17 15:14:27 unknown poweroff: initiated by root on /dev/console
syncing file systems... done
: core2[1]#; vdiskadm -u xvm rollback /vdisks/opensolaris@pre-destroy
: core2[1]#; virsh start opensolaris;virsh console opensolaris 
Domain opensolaris started

v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b'
SunOS Release 5.11 Version snv_111b 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: unknown
Reading ZFS config: done.
Mounting ZFS filesystems: (9/9)

unknown console login: root
Password: 
Aug 17 15:22:00 unknown login: ROOT LOGIN /dev/console
Last login: Mon Aug 17 13:46:34 on console
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
root@unknown:~# beadm list
BE             Active Mountpoint Space   Policy Created          
--             ------ ---------- -----   ------ -------          
opensolaris    -      -          4.62M   static 2009-08-17 10:05 
small-be       -      -          1.72M   static 2009-08-17 13:23 
small-be-clone NR     /          377.48M static 2009-08-17 13:45 
snv111b        -      -          3.17G   static 2009-08-17 13:20 
root@unknown:~# 
root@unknown:~# beadm destroy snv111b
Are you sure you want to destroy snv111b? This action cannot be undone(y/[n]): y
root@unknown:~# beadm destroy opensolaris
Are you sure you want to destroy opensolaris? This action cannot be undone(y/[n]): y
root@unknown:~# beadm list
BE             Active Mountpoint Space   Policy Created
--             ------ ---------- -----   ------ -------
small-be       -      -          1.72M   static 2009-08-17 13:23
small-be-clone NR     /          377.48M static 2009-08-17 13:45
root@unknown:~# beadm activate small-be
root@unknown:~#
root@unknown:~# reboot
Aug 17 13:52:22 unknown reboot: initiated by root on /dev/console
syncing file systems... done
rebooting...
v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b'
SunOS Release 5.11 Version snv_111b 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: unknown
Reading ZFS config: done.
Mounting ZFS filesystems: (7/7)

unknown console login: root
Password:
Aug 17 13:53:22 unknown login: ROOT LOGIN /dev/console
Last login: Mon Aug 17 13:44:19 on console
Sun Microsystems Inc.   SunOS 5.11      snv_111b        November 2008
root@unknown:~# beadm destroy small-be-clone
Are you sure you want to destroy small-be-clone? This action cannot be undone(y/[n]): y
root@unknown:~#

Finally, lets create a new cloned BE and upgrade it to snv118. Here we'll run into some minor pkg bugs.. They don't hurt anything though. You see that we grew the root a little after our upgrade.

root@unknown:~# beadm create snv118
root@unknown:~# beadm mount snv118 /mnt
root@unknown:~# pkg -R /mnt set-publisher -O http://pkg.opensolaris.org/dev opensolaris.org
root@unknown:~# pkg -R /mnt install SUNWipkg
No updates available for this image.
root@unknown:~# pkg -R /mnt install entire@0.5.11-0.118
DOWNLOAD                                    PKGS       FILES     XFER (MB)
Completed                                  73/73   4311/4311   79.34/79.34

PHASE                                        ACTIONS
Removal Phase                              1569/1569
Install Phase                              2619/2619
Update Phase                               5470/5698 
driver (softmac) upgrade (removal of policy'read_priv_set=net_rawaccess write_priv_set=net_rawaccess) failed: minor node spec required.
driver (vnic) upgrade (removal of policy'read_priv_set=net_rawaccess write_priv_set=net_rawaccess) failed: minor node spec required.
driver (aggr) upgrade (removal of policy'read_priv_set=net_rawaccess write_priv_set=net_rawaccess) failed: minor node spec required.
Update Phase                               5610/5698 
driver (dnet) upgrade (removal of policy'read_priv_set=net_rawaccess write_priv_set=net_rawaccess) failed: minor node spec required.
driver (elxl) upgrade (removal of policy'read_priv_set=net_rawaccess write_priv_set=net_rawaccess) failed: minor node spec required.
driver (iprb) upgrade (removal of policy'read_priv_set=net_rawaccess write_priv_set=net_rawaccess) failed: minor node spec required.
Update Phase                               5698/5698
PHASE                                          ITEMS
Reading Existing Index                           8/8
Indexing Packages                              73/73
Optimizing Index...
PHASE                                          ITEMS
Indexing Packages                              73/73
root@unknown:~# bootadm update-archive -R /mnt
updating /mnt//platform/i86pc/boot_archive
updating /mnt//platform/i86pc/amd64/boot_archive
root@unknown:~# beadm umount snv118
root@unknown:~# beadm activate snv118
root@unknown:~# reboot
Aug 17 14:05:37 unknown reboot: initiated by root on /dev/console
syncing file systems... done
rebooting...
v3.3.2-xvm chgset 'Wed Aug 12 17:12:49 2009 -0700 18433:bd9f134b1e1b'
SunOS Release 5.11 Version snv_118 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: unknown
Configuring devices.
Loading smf(5) service descriptions: 6/6
Reading ZFS config: done.
Mounting ZFS filesystems: (7/7)

unknown console login: root
Password:
Aug 17 14:06:39 unknown login: ROOT LOGIN /dev/console
Last login: Mon Aug 17 13:53:22 on console
Sun Microsystems Inc.   SunOS 5.11      snv_118 November 2008
root@unknown:~# rm -rf /var/pkg/download/*
root@unknown:~# df -lk
Filesystem            kbytes    used   avail capacity  Mounted on
rpool/ROOT/snv118    20514816  418664 18756788     3%    /

Posted at 04:08PM Aug 17, 2009 by mrj in Solaris | Permalink

Friday Apr 10, 2009

How small can Solaris go (part 2)

First, a couple of answers to some questions from the first post..

James, re: "be able (after a PXE boot) to mount NFS or iSCSI and switch (or layer) root". No. I'm really going for a standalone, extremely stripped down, ramdisk based image right now. No swap. No disk support. This wouldn't be something you would use for a NAS appliance. For that, you would be better off going with a stripped down pkg based opensolaris image. I can get those down to ~350M using a "custom installer"

Benoit, re: "And in a memory usage how light can solaris go?" Good question, lets take a look :-)

When I do get a little free time to play around with this stuff, I generally do it using OpenSolaris xVM domUs since it takes about a second to test then reboot these images.

The last post I did was a i86pc based image.. Here's a domU with the same bits loaded.. We're at ~ 15M for disk usage.

Here's the py file for my domU.. Notice I'm using the dom0's unix (since I'm building the ramdisk based on dom0's bits).

: alpha[1]#; cat /tank/guests/micro/guest.py 
name = "micro"
vcpus = 1
memory = "256"
kernel = "/platform/i86xpv/kernel/unix"
ramdisk = "/tank/guests/micro/ramdisk"
extra = "/platform/i86xpv/kernel/unix"
vif = ['']
on_shutdown = "destroy"
on_reboot = "restart"
on_crash = "preserve"
: alpha[1]#; 

: alpha[1]#; xm create -c /tank/guests/micro/guest.py
Using config file "/tank/guests/micro/guest.py".
Started domain micro
v3.3.2-rc1-pre-xvm chgset 'Mon Apr 06 20:13:29 2009 -0400 18424:97250633e58b'
SunOS Release 5.11 Version onnv-3.3-mrj 32-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
strplumb: failed to initialize drv/dld
# df -lk
Filesystem            kbytes    used   avail capacity  Mounted on
/ramdisk:a             38255   15116   19314    44%    /
/devices                   0       0       0     0%    /devices
/dev                       0       0       0     0%    /dev
ctfs                       0       0       0     0%    /system/contract
proc                       0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
swap                  205012       0  205012     0%    /etc/svc/volatile
objfs                      0       0       0     0%    /system/object
sharefs                    0       0       0     0%    /etc/dfs/sharetab
#

Now, we are really stripped down (for Solaris)... Lets add in kmdb and enough of mdb to let us do a mdb -K.

I have a very very ugly python script I use to build up my ramdisk...

: alpha[1]#; ./micro.py 
USAGE: ./micro.py cfg disk sizeM

Here's the config file I'm using for my domu after uncommenting out kmdb... Notice I don't even have syscalls in at this point...

: alpha[1]#; cat domu.files
@kmdb32

@kernel32
@i86xpv32
@init32
#@syscall32

#@mount
#@uidcache

#@net32
#/usr/bin/ln

#@devfsadm32
#@basic32
#@ssh32

I just luupgraded my system to b112.. Lets build a new image using that... So how much memory are we using?

: alpha[1]#; ./micro.py domu.files /tank/guests/micro/ramdisk 40
NOTICE: overwriting disk: /tank/guests/micro/ramdisk

: alpha[1]#; xm create -c /tank/guests/micro/guest.py 
Using config file "/tank/guests/micro/guest.py".
Started domain micro
v3.1.4-xvm chgset 'Mon Mar 30 23:29:09 2009 -0700 15914:bb9557896640'
SunOS Release 5.11 Version snv_112 32-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
NOTICE: Invalid iBFT table 0x1
strplumb: failed to initialize drv/dld
# df -lk
Filesystem            kbytes    used   avail capacity  Mounted on
/ramdisk:a             38255   19246   15184    56%    /
/devices                   0       0       0     0%    /devices
/dev                       0       0       0     0%    /dev
ctfs                       0       0       0     0%    /system/contract
proc                       0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
swap                  204992       0  204992     0%    /etc/svc/volatile
objfs                      0       0       0     0%    /system/object
sharefs                    0       0       0     0%    /etc/dfs/sharetab
# mdb -K

Welcome to kmdb
kmdb: no terminal data available for TERM=
kmdb: failed to set terminal type to `', using `vt100'
Loaded modules: [ scsi_vhci mac xpv_psm ufs unix krtld genunix specfs xpv_uppc
 ]
[0]> ::memstat
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                       8922                34   14%
Anon                          140                 0    0%
Exec and libs                  78                 0    0%
Page cache                    550                 2    1%
Free (cachelist)              708                 2    1%
Free (freelist)             53089               207   84%
Balloon                         0                 0    0%

Total                       63487               247
[0]>

less than 40M... Not great, but not bad either... There some code changes we could do to get things smaller. But it's not big enough where it would matter at this point... Since I'm building a 40M ramdisk, we need space for that too.. So I would need around 80-90M total for this image..

Lets do a quick test, changing the domU's memory to 80M.

< memory = "256"

> memory = "80"

: alpha[1]#; ./micro.py domu.files /tank/guests/micro/ramdisk 40
NOTICE: overwriting disk: /tank/guests/micro/ramdisk
: alpha[1]#; xm create -c /tank/guests/micro/guest.py 
Using config file "/tank/guests/micro/guest.py".
Started domain micro
v3.1.4-xvm chgset 'Mon Mar 30 23:29:09 2009 -0700 15914:bb9557896640'
SunOS Release 5.11 Version snv_112 32-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
NOTICE: Invalid iBFT table 0x1
strplumb: failed to initialize drv/dld
# mdb -K
WARNING: retrying of kmdb allocation of 0x600000 bytes

Welcome to kmdb
kmdb: no terminal data available for TERM=
kmdb: failed to set terminal type to `', using `vt100'
Loaded modules: [ scsi_vhci mac xpv_psm ufs unix krtld genunix specfs xpv_uppc
 ]
[0]> ::memstat
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                       5766                22   31%
Anon                          140                 0    1%
Exec and libs                   0                 0    0%
Page cache                      1                 0    0%
Free (cachelist)             1259                 4    7%
Free (freelist)             11265                44   61%
Balloon                         0                 0    0%

Total                       18431                71
[0]>

Interesting.. What about a 64-bit kernel? It's going to bigger of course. But if you need a 64-bit kernel, memory shouldn't be an issue.

: alpha[1]#; xm create -c /tank/guests/micro/guest64.py 
Using config file "/tank/guests/micro/guest64.py".
Started domain micro
v3.1.4-xvm chgset 'Mon Mar 30 23:29:09 2009 -0700 15914:bb9557896640'
SunOS Release 5.11 Version snv_112 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
strplumb: failed to initialize drv/dld
# df -lk
Filesystem            kbytes    used   avail capacity  Mounted on
/ramdisk:a             38255   31187    3243    91%    /
/devices                   0       0       0     0%    /devices
/dev                       0       0       0     0%    /dev
ctfs                       0       0       0     0%    /system/contract
proc                       0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
swap                  195480       0  195480     0%    /etc/svc/volatile
objfs                      0       0       0     0%    /system/object
sharefs                    0       0       0     0%    /etc/dfs/sharetab
# mdb -K

Welcome to kmdb
kmdb: no terminal data available for TERM=
kmdb: failed to set terminal type to `', using `vt100'
Loaded modules: [ scsi_vhci mac xpv_psm ufs unix krtld genunix specfs xpv_uppc
 ]
[0]> ::memstat
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                      15575                60   25%
Anon                          192                 0    0%
Exec and libs                  93                 0    0%
Page cache                    771                 3    1%
Free (cachelist)             1203                 4    2%
Free (freelist)             45653               178   72%
Balloon                         0                 0    0%

Total                       63487               247
[0]>

OK, now lets do something interesting... Let bring up networking enough so we can ping, etc. In my domu.files file, I'm going to bring in @syscall32, @mount, @net32, and /usr/bin/ln.

One thing you notice when you start playing with this stuff, is that things can grow very fast when you start pulling in user bins (due to all the libraries which can be pulled in too). You would think things like reboot and poweroff would be a small impact. Not so :-)

I have a custom init bin which configures the system and starts up a shell (no SMF, etc). I'll manually configure it though so you can see what I'm doing to get the system up.

Notice I'm setting up a dev link for the NIC.. We don't have devfsadm in this particular ramdisk. Obviously you would pre-create the link on the ramdisk, or pull in the devfsadm bits.. But I though it was an interesting thing to show.

: alpha[1]#; xm create -c /tank/guests/micro/guest.py 
Using config file "/tank/guests/micro/guest.py".
Started domain micro
v3.1.4-xvm chgset 'Mon Mar 30 23:29:09 2009 -0700 15914:bb9557896640'
SunOS Release 5.11 Version snv_112 32-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
NOTICE: Invalid iBFT table 0x1
# mount -o remount,rw /devices/ramdisk:a /
# /sbin/soconfig -f /etc/sock2path
# ifconfig lo0 plumb 127.0.0.1 netmask 255.255.255.0 up
# cd /dev
# ln -s ../devices/xpvd/xnf@0:xnf0 xnf0
# ifconfig xnf0 plumb 192.168.0.91 netmask 255.255.255.0 up
# route add default 192.168.0.1
add net default: gateway 192.168.0.1
# ping 192.168.0.1
192.168.0.1 is alive
# df -lk
Filesystem            kbytes    used   avail capacity  Mounted on
/devices/ramdisk:a     38255   28442    5988    83%    /
/devices                   0       0       0     0%    /devices
/dev                       0       0       0     0%    /dev
ctfs                       0       0       0     0%    /system/contract
proc                       0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
swap                  198852       0  198852     0%    /etc/svc/volatile
objfs                      0       0       0     0%    /system/object
sharefs                    0       0       0     0%    /etc/dfs/sharetab
# mdb -K

Welcome to kmdb
kmdb: no terminal data available for TERM=
kmdb: failed to set terminal type to `', using `vt100'
Loaded modules: [ scsi_vhci mac xpv_psm ufs unix krtld genunix specfs xpv_uppc
 ]
[0]> ::memstat
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                      11628                45   18%
Anon                          140                 0    0%
Exec and libs                 129                 0    0%
Page cache                    711                 2    1%
Free (cachelist)             1435                 5    2%
Free (freelist)             49444               193   78%
Balloon                         0                 0    0%

Total                       63487               247
[0]>

What's next? I'm trying to get VirtualBox (CLI only) running (RDP for external access) :-)

Posted at 10:44AM Apr 10, 2009 by mrj in General | Comments[1] | Permalink

Monday Mar 30, 2009

Intel Xeon 5500 series

Congratulations to Intel on their new Intel Xeon 5500 series. It is a truly remarkable CPU, and in my opinion, will be looked back on as a game changer in the industry.

There are features which are easy to see, such as the integrated memory controllers and the Quick Path Interconnects (QPI) (which connect the CPUs and IO bridges to each other).

But, as you would expect, they also continue to improve the guts of the CPUs. For example, with the 5500 series and associated IO chipsets, comes virtualization improvements including better virtualized CPU performance, and the ability to safely passthrough an IO device to a virtualized guest.

With Intel's help, we backported a lot of the new functionality from xen-unstable.hg into OpenSolaris xVM in April of 2008, including the big ones, Extended Page Table (EPT) support and Virtual Processor IDs (VPID). So we've been ready for a while :-)

With the EPT support, you can bypass the shadow page code in the hypervisor for fully virtualized guests. This gives you a nice performance improvement, and has the added benefit of a not having to run a *lot* of complex, and occasionally, buggy code.

I've just scratched the surface of the virtualization improvements in the Intel Xeon 5500 series. For folks who enjoy CPU technology, we live in exciting times.

Posted at 08:00PM Mar 30, 2009 by mrj in General | Permalink

Thursday Mar 12, 2009

How small can Solaris go

I've been playing around lately to see how small of a Solaris image I can get that will boot to a shell prompt... It turns out, it can get pretty small.. And it boots to the prompt in ~ 1 second... Although there's not a lot you can do with it :-)

SunOS Release 5.11 Version onnv-3.3-mrj 32-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
WARNING: Last shutdown is later than time on time-of-day chip; check date.
strplumb: failed to initialize drv/dld
# df -lk
Filesystem            kbytes    used   avail capacity  Mounted on
/ramdisk:a             38255   15146   19284    44%    /
/devices                   0       0       0     0%    /devices
/dev                       0       0       0     0%    /dev
ctfs                       0       0       0     0%    /system/contract
proc                       0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
swap                 1768156       0 1768156     0%    /etc/svc/volatile
objfs                      0       0       0     0%    /system/object
sharefs                    0       0       0     0%    /etc/dfs/sharetab
# du -sk *
31      boot
952     dev
1       devices
67      etc
6087    kernel
4849    lib
8       lost+found
1931    platform
11277   proc
122     sbin
1441    system
1       tmp
59      usr
3       var
#

Posted at 08:38PM Mar 12, 2009 by mrj in Solaris | Comments[5] | Permalink

UPDATE: Developing with multiple BEs in OpenSolaris

Following up on Bart's comment, you can certainly use -R to perform the same operation... Not sure why I didn't think that could be used for pkg set-authority, but it can..

Funny since I use -R for my custom opensolaris builds..

root@unknown:~# df -lk
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0d0s0       491522  367690   74680    84%    /
/devices                   0       0       0     0%    /devices
/dev                       0       0       0     0%    /dev
ctfs                       0       0       0     0%    /system/contract
proc                       0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
swap                  767856     336  767520     1%    /etc/svc/volatile
objfs                      0       0       0     0%    /system/object
sharefs                    0       0       0     0%    /etc/dfs/sharetab
/usr/lib/libc/libc_hwcap3.so.1
                      491522  367690   74680    84%    /lib/libc.so.1
fd                         0       0       0     0%    /dev/fd
swap                  767520       0  767520     0%    /tmp
swap                  767536      16  767520     1%    /var/run
root@unknown:~# pkg list | wc -l
      65
root@unknown:~# ps -ef
     UID   PID  PPID   C    STIME TTY         TIME CMD
    root     0     0   0 06:52:45 ?           0:01 sched
    root     1     0   0 06:52:46 ?           0:00 /sbin/init
    root     2     0   0 06:52:46 ?           0:00 pageout
    root     3     0   0 06:52:46 ?           0:00 fsflush
    root     7     1   0 06:52:47 ?           0:02 /lib/svc/bin/svc.startd
    root     9     1   0 06:52:47 ?           0:27 /lib/svc/bin/svc.configd
    root   549     1   0 10:58:06 ?           0:00 /usr/lib/inet/inetd start
    root   201     1   0 06:53:19 ?           0:00 devfsadmd
  daemon   293     1   0 06:53:40 ?           0:00 /lib/crypto/kcfd
   dladm    15     1   0 06:52:48 ?           0:00 /sbin/dlmgmtd
    root   298     1   0 06:53:44 ?           0:00 /usr/lib/picl/picld
    root   198     1   0 06:53:19 ?           0:00 /usr/lib/sysevent/syseventd
    root   554   552   0 10:58:06 ?           0:00 /usr/lib/saf/ttymon
    root   552     7   0 10:58:06 ?           0:00 /usr/lib/saf/sac -t 300
    root   580     1   0 10:58:08 ?           0:00 /usr/lib/ssh/sshd
  daemon   525     1   0 06:57:48 ?           0:00 /usr/sbin/rpcbind
    root   544     1   0 10:58:03 ?           0:00 /usr/sbin/nscd
    root   512     1   0 06:56:07 ?           0:00 /sbin/dhcpagent
    root   553     1   0 10:58:06 ?           0:00 /usr/lib/utmpd
    root   571     7   0 10:58:08 console     0:00 -bash
    root   376     1   0 06:53:54 ?           0:00 /usr/sbin/cron
    root   667   571   0 11:03:32 console     0:00 ps -ef
    root   567     1   0 10:58:08 ?           0:00 /usr/sbin/syslogd
root@unknown:~#

Anyway, here is the sequence using a -R.. This works for 99.x% of the cases.. But I would expect to fail for the same cases lu will.. i.e. say you need a new version update_drv, etc. For those cases the chroot will get you through it with some skilled sequencing.. Of course, the chroot approach can have it's own set of problems :-)

beadm create snv109
beadm mount snv109 /mnt
pkg -R /mnt set-authority -O http://pkg.opensolaris.org/dev opensolaris.org
pkg -R /mnt refresh
pkg -R /mnt install SUNWipkg
pkg -R /mnt install entire@0.5.11-0.109
bootadm update-archive -R /mnt
beadm umount snv109
beadm activate snv109

Posted at 10:34AM Mar 12, 2009 by mrj in Solaris | Comments[2] | Permalink

Wednesday Mar 04, 2009

Developing with multiple BEs in OpenSolaris

One of the things I have found over the years to be helpful as a Solaris kernel developer is liveupgrade (lucreate(1M)).

For example, I will usually have 3 boot environments (BE) on my systems at one time. The first BE usually has the base build that my gate is a child of. The second BE is my test BE which I BFU, and generally seem to kill quite often. And the third BE is an old build so I can go move my test BE to the old build all the way up to the most recent build (luupgrade(1M)).

OpenSolaris uses beadm which is a nice replacement to the lu commands. I was a little disappointed when I first started using it because I didn't think it was as flexible as I wanted.. But after playing with it for a while, I have a setup which I'm pretty happy with at the moment.

What I want to be able to do is to have the stock OpenSolaris BE, which gets upgraded at major releases. But I also want to have BEs which are based off of various development builds, and test BEs which I can BFU, etc. This is really nice since these will be COW based clones due to zfs.

So the question was how to update an alternate BE and at the same time, not modify the current BE. i.e. not having to update the authority, upgrade SUNWipkg, etc. The solution is pretty simple, chroot...

Here's how I create a b108 BE on a stock 2008.11 system, while ensuring I keep the 2008.11 bits unmodified. I can switch back to the stock OpenSolaris bits at any time with a beadm activate opensolaris.

beadm create snv108
beadm mount snv108 /mnt
mount -F proc /proc /mnt/proc
chroot /mnt
pkg set-authority -O http://pkg.opensolaris.org/dev opensolaris.org
pkg refresh
pkg install SUNWipkg
pkg install entire@0.5.11-0.108
bootadm update-archive
exit
umount /mnt/proc
beadm umount snv108
beadm activate snv108

Posted at 09:31AM Mar 04, 2009 by mrj in Solaris | Comments[3] | Permalink

Monday May 05, 2008

Installing OpenSolaris on Xen

Here are some quick instructions on how to install a DHCP based PV OpenSolaris guest/domU on a hypervisor based on the Xen open source community.

First, download the OpenSolaris CDROM.

Here's the py file I'm using... Your path to pygrub will differ if your using a linux dom0.

: alpha[1]#; cat pv.py
name = "opensolaris-pv-install"
vcpus = 1
memory = "1024"
bootloader = "/usr/lib/xen/bin/pygrub"
kernel = "/platform/i86xpv/kernel/amd64/unix"
ramdisk = "/boot/x86.microroot"
extra = "/platform/i86xpv/kernel/amd64/unix -B console=ttya,livemode=text"
disk = ['file:/tank/guests/install/opensolaris/os200805.iso,6:cdrom,r',
        'file:/tank/guests/opensolaris/disk.img,0,w']
vif = ['']
on_shutdown = "destroy"
on_reboot = "destroy"
on_crash = "preserve"
: alpha[1]#;

Setup your paths correctly, create your disk, etc. Boot the OpenSolaris LiveCD

: alpha[1]#; xm create -c pv.py
Using config file "./pv.py".
Started domain opensolaris-pv-install
v3.1.4-xvm chgset 'Fri May 02 10:23:19 2008 -0700 15873:3e3bd3d19023'
SunOS Release 5.11 Version snv_86 64-bit
Copyright 1983-2008 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: opensolaris
Remounting root read/write
Probing for device nodes ...
Preparing live image for use
Done mounting Live image
USB keyboard
 1. Albanian                      22. Latvian                       
 2. Belarusian                    23. Macedonian                    
 3. Belgian                       24. Malta_UK                      
 4. Bulgarian                     25. Malta_US                      
 5. Croatian                      26. Norwegian                     
 6. Czech                         27. Polish                        
 7. Danish                        28. Portuguese                    
 8. Dutch                         29. Russian                       
 9. Finnish                       30. Serbia-And-Montenegro         
10. French                        31. Slovenian                     
11. French-Canadian               32. Slovakian                     
12. Hungarian                     33. Spanish                       
13. German                        34. Swedish                       
14. Greek                         35. Swiss-French                  
15. Icelandic                     36. Swiss-German                  
16. Italian                       37. Traditional-Chinese           
17. Japanese-type6                38. TurkishQ                      
18. Japanese                      39. TurkishF                      
19. Korean                        40. UK-English                    
20. Latin-American                41. US-English                    
21. Lithuanian                    
To select the keyboard layout, enter a number [default 41]:

1. Chinese - Simplified
2. Chinese - Traditional
3. English
4. French
5. German
6. Italian
7. Japanese
8. Korean
9. Portuguese - Brazil
10. Russian
11. Spanish
12. Swedish
To select the desktop language, enter a number [default 3]:
Configuring devices.
Mounting local partitions/cdroms
Reading ZFS config: done.

opensolaris console login: 
 May  5 08:06:30 opensolaris in.routed[639]: route 0.0.0.0/8 --> 0.0.0.0 nexthop ...
opensolaris console login:

Log into LiveCD (jack/jack). Make sure your networking is up (it can take a minute or two until the DHCP client runs).

opensolaris console login: jack
Password: 
Last login: Mon May  5 08:07:00 on console
Sun Microsystems Inc.   SunOS 5.11      snv_86  January 2008
jack@opensolaris:~$ 
jack@opensolaris:~$ ifconfig xnf0
xnf0: flags=201004843 mtu 1500 index 2
        inet 192.168.0.117 netmask ffffff00 broadcast 192.168.0.255

Start up a VNC Server, connect to it on port 5901, and run through the install.

jack@opensolaris:~$ mkdir .vnc;cp .Xclients .vnc/xstartup 
jack@opensolaris:~$ vncserver

You will require a password to access your desktops.

Password:
Verify:
xauth:  creating new authority file /jack/.Xauthority

New 'opensolaris:1 ()' desktop is opensolaris:1

Starting applications specified in /jack/.vnc/xstartup
Log file is /jack/.vnc/opensolaris:1.log

Once your install has completed, create a py file for the your new guest, and you are ready to go...

name = "opensolaris"
vcpus = 1
memory = "512"
disk = ['file:/tank/guests/opensolaris/disk.img,0,w']
vif = ['']
on_shutdown = "destroy"
on_reboot = "restart"
on_crash = "destroy"

Have Fun!

Posted at 03:44PM May 05, 2008 by mrj in Solaris | Comments[4] | Permalink

Thursday Jul 19, 2007

The Latest Solaris on Xen drop

I've learned something.. Never promise to write a follow up blog. :-) In keeping with my usual raw data dump style, here are some tips and tricks...

The latest Solaris on Xen drop is out. If you haven't seen it already, head over to here.

If you don't have a vncviewer installed, you can use the one which is part of vino (remote desktop). Here's what I have on some of my systems.

$ cat `which vncviewer`
#!/bin/sh
exec java -jar /usr/share/gnome/vino/vino-client.jar  ${1+"$@"}

When running windows XP as a guest, you can enable (crappy) sound by adding the following to your py file.

  soundhw='es1370'

The default NIC in HVM works fine, but is slowww. At this time, WinXP PV drivers aren't generally available (to speed the IO up).

  vif = [ 'type=ioemu,mac=.your mac here.' ]

Also in WinXP, if your using VNC, specifying a USB tablet is helpful for mouse tracking. This doesn't work in Linux HVM domains though (no driver available). e.g. add ...

  usb=1
  usbdevice="tablet"

Other than the NIC, these settings work equally well in Windows Vista. You need to use a different NIC to get networking to work in Vista, e.g.

  vif =  ['type=ioemu,mac=.your mac here.,model=ne2k_pci']

I have few guests on my system.

I need more memory on my system now. 4G isn't enough anymore. It takes a while to go through all my guests and install the updates weekly.

A couple of the more interesting, non traditional ones... for Ubunutu, I did a HVM install and then copied in a Linux domU kernel which I built (all static, no modules). One thing I haven't figured out yet is how to properly setup the console devices. I can't seem to get /dev/xvc0 or /dev/tty0 created even though the frontend and backend driver are present. I do get console output on xvc0 though, so it is usuable. For the PV framebuffer, I do get a connection between the frontend/backend, but can't run X since it wants to open /dev/tty0. You do see a lot of "/dev/mem: mmap: Bad address" messages which comes from /usr/sbin/dmidecode trying to call into the BIOS (which isn't support in a PV guest). Unfortunately it's not a trival thing to remove...

    root@mrj-desktop:/etc# apt-get remove dmidecode
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    The following packages will be REMOVED:
      acpi-support dmidecode gnome-power-manager gnome-session hotkey-setup
      laptop-detect powermanagement-interface powernowd tasksel tasksel-data
      ubuntu-desktop ubuntu-minimal ubuntu-standard
    0 upgraded, 0 newly installed, 13 to remove and 0 not upgraded.
    27 not fully installed or removed.
    Need to get 0B of archives.
    After unpacking 17.2MB disk space will be freed.
    Do you want to continue [Y/n]? n
    Abort.
    root@mrj-desktop:/etc#

It hasn't stopped me from doing anything other than upgrading hotkey-setup.

My gentoo guest is a staging area for building my via based router image. I have a compact flash in the system with a kernel and compressed ramdisk. The ramdisk is < 256M when uncompressed. This way I don't write to the compact flash very often, and if the system is hacked, I just need to power cycle to clean it up. I'm running a real gentoo/glibc (vs ulibc) distro which is why it's so big. I have a disk image which I loop mount and chroot to. I update all the packages then umount, copy the image file to a final image file. loop mount and chroot to new image file. emerge -C the man pages and gcc, rm -rf /usr/portage, portage files in /var, and some extra docs, then gzip it and scp it to the router. I have to install more memory this way but can run any package I want which is nice..

It's nice to be able to do all this from one system now :-) It's surprising how often I fire up the other OSes to try something out or look around, etc. I just need a laptop with 8G or more of memory now... :-)