mrj's Weblog

First, a couple of answers to some questions from the first post..

James, re: "be able (after a PXE boot) to mount NFS or iSCSI and switch (or layer) root". No. I'm really going for a standalone, extremely stripped down, ramdisk based image right now. No swap. No disk support. This wouldn't be something you would use for a NAS appliance. For that, you would be better off going with a stripped down pkg based opensolaris image. I can get those down to ~350M using a "custom installer"

Benoit, re: "And in a memory usage how light can solaris go?" Good question, lets take a look :-)

When I do get a little free time to play around with this stuff, I generally do it using OpenSolaris xVM domUs since it takes about a second to test then reboot these images.

The last post I did was a i86pc based image.. Here's a domU with the same bits loaded.. We're at ~ 15M for disk usage.

Here's the py file for my domU.. Notice I'm using the dom0's unix (since I'm building the ramdisk based on dom0's bits).

: alpha[1]#; cat /tank/guests/micro/guest.py 
name = "micro"
vcpus = 1
memory = "256"
kernel = "/platform/i86xpv/kernel/unix"
ramdisk = "/tank/guests/micro/ramdisk"
extra = "/platform/i86xpv/kernel/unix"
vif = ['']
on_shutdown = "destroy"
on_reboot = "restart"
on_crash = "preserve"
: alpha[1]#; 

: alpha[1]#; xm create -c /tank/guests/micro/guest.py
Using config file "/tank/guests/micro/guest.py".
Started domain micro
v3.3.2-rc1-pre-xvm chgset 'Mon Apr 06 20:13:29 2009 -0400 18424:97250633e58b'
SunOS Release 5.11 Version onnv-3.3-mrj 32-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
strplumb: failed to initialize drv/dld
# df -lk
Filesystem            kbytes    used   avail capacity  Mounted on
/ramdisk:a             38255   15116   19314    44%    /
/devices                   0       0       0     0%    /devices
/dev                       0       0       0     0%    /dev
ctfs                       0       0       0     0%    /system/contract
proc                       0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
swap                  205012       0  205012     0%    /etc/svc/volatile
objfs                      0       0       0     0%    /system/object
sharefs                    0       0       0     0%    /etc/dfs/sharetab
#

Now, we are really stripped down (for Solaris)... Lets add in kmdb and enough of mdb to let us do a mdb -K.

I have a very very ugly python script I use to build up my ramdisk...

: alpha[1]#; ./micro.py 
USAGE: ./micro.py cfg disk sizeM

Here's the config file I'm using for my domu after uncommenting out kmdb... Notice I don't even have syscalls in at this point...

: alpha[1]#; cat domu.files
@kmdb32

@kernel32
@i86xpv32
@init32
#@syscall32

#@mount
#@uidcache

#@net32
#/usr/bin/ln

#@devfsadm32
#@basic32
#@ssh32

I just luupgraded my system to b112.. Lets build a new image using that... So how much memory are we using?

: alpha[1]#; ./micro.py domu.files /tank/guests/micro/ramdisk 40
NOTICE: overwriting disk: /tank/guests/micro/ramdisk

: alpha[1]#; xm create -c /tank/guests/micro/guest.py 
Using config file "/tank/guests/micro/guest.py".
Started domain micro
v3.1.4-xvm chgset 'Mon Mar 30 23:29:09 2009 -0700 15914:bb9557896640'
SunOS Release 5.11 Version snv_112 32-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
NOTICE: Invalid iBFT table 0x1
strplumb: failed to initialize drv/dld
# df -lk
Filesystem            kbytes    used   avail capacity  Mounted on
/ramdisk:a             38255   19246   15184    56%    /
/devices                   0       0       0     0%    /devices
/dev                       0       0       0     0%    /dev
ctfs                       0       0       0     0%    /system/contract
proc                       0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
swap                  204992       0  204992     0%    /etc/svc/volatile
objfs                      0       0       0     0%    /system/object
sharefs                    0       0       0     0%    /etc/dfs/sharetab
# mdb -K

Welcome to kmdb
kmdb: no terminal data available for TERM=
kmdb: failed to set terminal type to `', using `vt100'
Loaded modules: [ scsi_vhci mac xpv_psm ufs unix krtld genunix specfs xpv_uppc
 ]
[0]> ::memstat
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                       8922                34   14%
Anon                          140                 0    0%
Exec and libs                  78                 0    0%
Page cache                    550                 2    1%
Free (cachelist)              708                 2    1%
Free (freelist)             53089               207   84%
Balloon                         0                 0    0%

Total                       63487               247
[0]> 

less than 40M... Not great, but not bad either... There some code changes we could do to get things smaller. But it's not big enough where it would matter at this point... Since I'm building a 40M ramdisk, we need space for that too.. So I would need around 80-90M total for this image..

Lets do a quick test, changing the domU's memory to 80M.

< memory = "256"

> memory = "80"

: alpha[1]#; ./micro.py domu.files /tank/guests/micro/ramdisk 40
NOTICE: overwriting disk: /tank/guests/micro/ramdisk
: alpha[1]#; xm create -c /tank/guests/micro/guest.py 
Using config file "/tank/guests/micro/guest.py".
Started domain micro
v3.1.4-xvm chgset 'Mon Mar 30 23:29:09 2009 -0700 15914:bb9557896640'
SunOS Release 5.11 Version snv_112 32-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
NOTICE: Invalid iBFT table 0x1
strplumb: failed to initialize drv/dld
# mdb -K
WARNING: retrying of kmdb allocation of 0x600000 bytes

Welcome to kmdb
kmdb: no terminal data available for TERM=
kmdb: failed to set terminal type to `', using `vt100'
Loaded modules: [ scsi_vhci mac xpv_psm ufs unix krtld genunix specfs xpv_uppc
 ]
[0]> ::memstat
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                       5766                22   31%
Anon                          140                 0    1%
Exec and libs                   0                 0    0%
Page cache                      1                 0    0%
Free (cachelist)             1259                 4    7%
Free (freelist)             11265                44   61%
Balloon                         0                 0    0%

Total                       18431                71
[0]> 

Interesting.. What about a 64-bit kernel? It's going to bigger of course. But if you need a 64-bit kernel, memory shouldn't be an issue.

: alpha[1]#; xm create -c /tank/guests/micro/guest64.py 
Using config file "/tank/guests/micro/guest64.py".
Started domain micro
v3.1.4-xvm chgset 'Mon Mar 30 23:29:09 2009 -0700 15914:bb9557896640'
SunOS Release 5.11 Version snv_112 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
strplumb: failed to initialize drv/dld
# df -lk
Filesystem            kbytes    used   avail capacity  Mounted on
/ramdisk:a             38255   31187    3243    91%    /
/devices                   0       0       0     0%    /devices
/dev                       0       0       0     0%    /dev
ctfs                       0       0       0     0%    /system/contract
proc                       0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
swap                  195480       0  195480     0%    /etc/svc/volatile
objfs                      0       0       0     0%    /system/object
sharefs                    0       0       0     0%    /etc/dfs/sharetab
# mdb -K

Welcome to kmdb
kmdb: no terminal data available for TERM=
kmdb: failed to set terminal type to `', using `vt100'
Loaded modules: [ scsi_vhci mac xpv_psm ufs unix krtld genunix specfs xpv_uppc
 ]
[0]> ::memstat
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                      15575                60   25%
Anon                          192                 0    0%
Exec and libs                  93                 0    0%
Page cache                    771                 3    1%
Free (cachelist)             1203                 4    2%
Free (freelist)             45653               178   72%
Balloon                         0                 0    0%

Total                       63487               247
[0]> 

OK, now lets do something interesting... Let bring up networking enough so we can ping, etc. In my domu.files file, I'm going to bring in @syscall32, @mount, @net32, and /usr/bin/ln.

One thing you notice when you start playing with this stuff, is that things can grow very fast when you start pulling in user bins (due to all the libraries which can be pulled in too). You would think things like reboot and poweroff would be a small impact. Not so :-)

I have a custom init bin which configures the system and starts up a shell (no SMF, etc). I'll manually configure it though so you can see what I'm doing to get the system up.

Notice I'm setting up a dev link for the NIC.. We don't have devfsadm in this particular ramdisk. Obviously you would pre-create the link on the ramdisk, or pull in the devfsadm bits.. But I though it was an interesting thing to show.

: alpha[1]#; xm create -c /tank/guests/micro/guest.py 
Using config file "/tank/guests/micro/guest.py".
Started domain micro
v3.1.4-xvm chgset 'Mon Mar 30 23:29:09 2009 -0700 15914:bb9557896640'
SunOS Release 5.11 Version snv_112 32-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
NOTICE: Invalid iBFT table 0x1
# mount -o remount,rw /devices/ramdisk:a /
# /sbin/soconfig -f /etc/sock2path
# ifconfig lo0 plumb 127.0.0.1 netmask 255.255.255.0 up
# cd /dev
# ln -s ../devices/xpvd/xnf@0:xnf0 xnf0
# ifconfig xnf0 plumb 192.168.0.91 netmask 255.255.255.0 up
# route add default 192.168.0.1
add net default: gateway 192.168.0.1
# ping 192.168.0.1
192.168.0.1 is alive
# df -lk
Filesystem            kbytes    used   avail capacity  Mounted on
/devices/ramdisk:a     38255   28442    5988    83%    /
/devices                   0       0       0     0%    /devices
/dev                       0       0       0     0%    /dev
ctfs                       0       0       0     0%    /system/contract
proc                       0       0       0     0%    /proc
mnttab                     0       0       0     0%    /etc/mnttab
swap                  198852       0  198852     0%    /etc/svc/volatile
objfs                      0       0       0     0%    /system/object
sharefs                    0       0       0     0%    /etc/dfs/sharetab
# mdb -K

Welcome to kmdb
kmdb: no terminal data available for TERM=
kmdb: failed to set terminal type to `', using `vt100'
Loaded modules: [ scsi_vhci mac xpv_psm ufs unix krtld genunix specfs xpv_uppc
 ]
[0]> ::memstat
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                      11628                45   18%
Anon                          140                 0    0%
Exec and libs                 129                 0    0%
Page cache                    711                 2    1%
Free (cachelist)             1435                 5    2%
Free (freelist)             49444               193   78%
Balloon                         0                 0    0%

Total                       63487               247
[0]> 

What's next? I'm trying to get VirtualBox (CLI only) running (RDP for external access) :-)