Notes from a Carbon Based Life Form

Using Solaris 10 Update 3, Sun Cluster 3.2, Zones, & ZFS in a Multi-Node Cluster of Sun Fire T-2000s

It all started with a conference call with one of our customers. We wanted a way to set up some highly available systems that could be used for various beta or QA purposes, or production services, or anywhere in between as needed. We also wanted a way to maximize the resources. We had 4 servers available to us, all Sun Fire T-2000s. If we used them as straight servers, they'd be great at anything they do, right? 8 cores, 4 threads per core, 32GB of RAM. Nice. Capable of running dozens of zones without skipping a beat. Perhaps even hundreds of zones.

Zones make perfect development boxes, right? You can blow them away and re-install them in a matter of minutes, or even seconds on ZFS. Zones also do pretty good as production environments as well. We're currently using a large number of zones in production, to supply a variety of services.

Zones on ZFS make particularly good dev boxes because you can take frequent snapshots and roll back as desired.

** Zones with their zoneroot on ZFS do encounter bug #6356600, which relates to how the live upgrade scratch zone used for installing packages into local zones can't access ZFS filesystems to upgrade zones with a ZFS zoneroot.

Sun Cluster 3.2 introduced support for ZFS as a failover filesystem, and for failover zones as well. We decided to make use of both of these features.

We built a 4-node cluster out of the 4 T-2000s, and began exporting individual disks from our SAN. Put 3 disks in a ZFS pool as a raidz filesystem, and installed the zone root at a ratio of 1 zone per zpool. (We're still doing some testing with our SAN and comparing performance of ZFS on individual disks, or ZFS on a RAID5 LUN exported by the SAN, but so far the way we're doing it is working nicely.)

So we built the cluster, and got it all configured and running. We then installed the first zone onto the ZFS pool. Then I copied the relevant portions of the zones configuration (in /etc/zones/*) to the other nodes in the cluster.

We then created a resource group in Sun Cluster 3.2, and added the zone into the resource group. We also added the ZFS pool into the resource group as an HA-Storage resource, and created a quick set of control scripts to start and stop the zone. The zone itself takes care of bringing up it's ip addresses, and starting the various applications installed within.

End result: Highly available servers, on a failover basis, that take less than 30 seconds to fail over from one host to another.

So far it's working really well. We're already getting more requests to build more of these multi-noded clusters, with zone/zpool combo's as the resource group. It's been a great solution for us.

So. It turns out I went to a lot of needless-overly-complicated-thought-process-work figuring out a way for a local zone to login to a global zone and run a command, within an RBAC profile to automatically set the date and time of a Global Zone.

global# zonecfg -z localzone
zonecfg:localzone> set limitpriv="default,sys_time"
zonecfg:localzone> commit
zonecfg:localzone> exit

I would then have been all set to just run `ntpdate` or something to set the date/time from the localzone.

At least my last experience was good some usefull experience that I've been able to re-use that's not related to setting the time and date, but rather giving people limited access to run specific pre-configured commands in the global zone, from the local zone, without having an account on the global zone, and without opening the global zone to any overt risk.

Well. I finally managed to get the Sun Solaris Operating Environment installed on my Intel-based Apple Mac Mini. More specifically, I managed to get a patched version of Solaris Express build 46, which should be available to the public VSN, installed onto the Mini.

*edit: It appears that changes required to multiboot, grub, stage2 (and stage2_eltorito), as well as to fdisk, should be available in build 47 of Solaris Express, which should be available by mid-September at the latest, considering build 46 is available now as the Solaris Express Community Release as linked from the OpenSolaris ON Downloads Page.

tcsh-[107]% prtdiag
System Configuration: Apple Computer, Inc. Macmini1,1
BIOS Configuration: Apple Computer, Inc. MM11.88Z.0055.B03.0604071521 04/07/06

==== Processor Sockets ====================================

Version Location Tag
-------------------------------- --------------------------
Genuine Intel(R) CPU T U2E1
Genuine Intel(R) CPU T U2E1

==== Memory Device Sockets ================================

Type Status Set Device Locator Bank Locator
------- ------ --- ------------------- --------------------
DDR2 in use 0 DIMM0 BANK 0
DDR2 in use 0 DIMM1 BANK 1

==== On-Board Devices =====================================
Integrated Graphics Controller
Yukon Ethernet Controller
Azalia Audio Codec
SATA
PATA

==== Upgradeable Slots ====================================

ID Status Type Description
--- --------- ---------------- ----------------------------
2 available PCI Express AirPort

I did have one little bug, though, (6374895) related to the Solaris fdisk binary, and the Apple EFI disk label. The Solaris install seems to have eaten my Mac OS X partition. It did, however, leave Apple's Boot Camp as the bootloader though, so that's nice. With a keyboard, and a Left-ALT key, I can boot from CD if I need to. :)

I guess now I'll have to bootstrap a gcc just to compare it against my linux server, and my Sun Ultra 20 Workstation.

A shell script was running just fine interactively, but when I tried to automate it, it wasn't behaving as expected, and I hadn't set it up to log it's output to a file.

#!/bin/sh

DATE=`date`
echo "Begin: `basename $0` :: $DATE"
echo
echo " Testing for /tmp/foo"
if [ -f /tmp/foo ]; then
echo " /tmp/foo exists, we're good to go"
else
echo " /tmp/foo doesn't exist, not so good to go"
fi
echo " Testing for /tmp/bar"
if [ -f /tmp/bar ]; then
echo " /tmp/bar exists, we're still good to go"
else
echo " /tmp/bar doesn't exist, Doh! Double not so good to go"
fi

Now, that's well, and good, but when it's running, automated, on boot, the output wasn't being captured so I couldn't see what the problem was, or what errors occured. So I started looking for an easy lazy way to log the output of my shell script, which was being run non-interactively. I certainly didn't want to go through the script line by line, and add a " >> $logfile" to the end of every echo, and every command.

#!/bin/sh

LOGFILE="/var/adm/`basename $0`.log"
exec 1>>$LOGFILE
exec 2>&1

DATE=`date`
echo "Begin: `basename $0` :: $DATE"
echo
echo " Testing for /tmp/foo"
if [ -f /tmp/foo ]; then
echo " /tmp/foo exists, we're good to go"
else
echo " /tmp/foo doesn't exist, not so good to go"
fi
echo " Testing for /tmp/bar"
if [ -f /tmp/bar ]; then
echo " /tmp/bar exists, we're still good to go"
else
echo " /tmp/bar doesn't exist, Doh! Double not so good to go"
fi

And surprise, surprise. That worked perfectly. :)
Turns out my problem was a typo. I had a 't' where I should have had an 'r'.
Here's what my log looked like:

% cat /var/adm/foo.sh.log
Begin: foo.sh :: Fri Feb 24 01:48:59 EST 2006

Testing for /tmp/foo
/tmp/foo exists, we're good to go
Testing for /tmp/bar
/tmp/bar doesn't exist, Doh! Double not so good to go

I was presented with a question a while back. Is there a way to have a non-global zone set the time and date of a Solaris 10 system? The answer I came it with is "Sort of." Here's why:

I looked at setting up an RBAC profile assiging the command /usr/bin/rdate, with uid=0 and privs=sys_time, but that didn't work. There is no way to assign an RBAC profile from a global zone to a user from a non-global zone, and there is no way to assign an RBAC profile directly to a zone. (That's a neat idea, though, isn't it?)

Then I thought about using an ssh key to permit passphrase-less authentication from a user in the non-global zone, to the global zone to log in and run a command (set up in an RBAC profile) to allow them to set the date. Here's my solution to this problem:

First, I create the proxy user, and then I set up the RBAC profile in the global zone:

1. Create the user account (I use proxy, because I also use this account for other purposes):

bash-3.00# useradd -u 600 -g 10 -c "Special Proxy User" -s /usr/bin/rbash -d /export/home/proxy -m proxy
bash-3.00# passwd proxy
New Password: ********
Re-enter new Password: ********
passwd: password successfully changed for proxy

4. Add this line to /etc/security/exec_attr:
Proxy:solaris:cmd:::/usr/bin/rdate:uid=0;privs=sys_time

Now, if you were to su to proxy and run rdate 192.43.244.18 you would effect a setting of the time and date on your system, hopefully without errors.

Now, to set up the access from the non-global zone, `su` to the proxy user account, and generate an ssh-key. I'm using a commented, 1024-bit Diffie-Hellman key, just for example purposes. Yours can be whatever bit-level and algorithm you prefer.

bash-3.00# su - proxy
bash-3.00$ ssh-keygen -f .ssh/rdate.key -t dsa -b 1024 -C "Rdate Command Key"
Generating public/private dsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in .ssh/rdate.key.
Your public key has been saved in .ssh/rdate.key.pub.
The key fingerprint is:
e4:02:8e:94:b7:c5:87:30:ca:fe:ce:c3:09:98:26:e3 Rdate Command Key

Once the key, is generated, copy the contents of your public key file to your authorized_keys file:

bash-3.00$ cp .ssh/rdate.key.pub .ssh/authorized_keys

ssh-dss AAAAB3NzaC1kc3MAAACBANhSeknty5MNVDTg7o8fJ3WYhsDd2k/RaFIWPXvROLCgP4BduWFw
AJw/bTRVLVah5Jk52EIbKtst4hZ2EJdVtIxOo/WI+6IE5c8shDoo2JjMl3Qkyim8kIl/g8aOi094yZe7
oYcrL8+Zo5I/PaNLOS6JN6VFa3lp1zAbEMeHaoQBAAAAFQD0X4fWRQBVF7z9J2TblQsCaaCJJwAAAIBt
DBmKQ/OyOd5fMbp8e8PXCasv84smq3vrHNZzDjA+pSVfQHcQ87s79IiHBWtZwawVZlzAZzk9kykZBZjl
lRPVAFm2dFZbDtadmOk9L2yK/oj6k9h5POBXEJS4HZ7ry1vDsgFV5kOqFppeMtynbhgE2YsYK64nzeFQ
nAr1uJQJXQAAAIEAtfaiakkVWQlgZnu89LTEPYOmUbM4MziGpKqZyaBBpjyT/6ugTGahuRKI4br0ESCV
jxmKqGkR1O5jOSNBvMr2zbWRpnVgCM8ThDH1OuvPdZu2aSAX2KCnIga+Wzj8/uWCCDHRj+2D0l7vGgQX
ImJMlJ4Sa9xyOMJH8qU6Un0Gscs= Rdate Command Key

command="pfexec rdate 192.43.244.18" ssh-dss AAAAB3NzaC1kc3MAAACBANhSeknty5MNVDT
g7o8fJ3WYhsDd2k/RaFIWPXvROLCgP4BduWFwAJw/bTRVLVah5Jk52EIbKtst4hZ2EJdVtIxOo/WI+6I
E5c8shDoo2JjMl3Qkyim8kIl/g8aOi094yZe7oYcrL8+Zo5I/PaNLOS6JN6VFa3lp1zAbEMeHaoQBAAA
AFQD0X4fWRQBVF7z9J2TblQsCaaCJJwAAAIBtDBmKQ/OyOd5fMbp8e8PXCasv84smq3vrHNZzDjA+pSV
fQHcQ87s79IiHBWtZwawVZlzAZzk9kykZBZjllRPVAFm2dFZbDtadmOk9L2yK/oj6k9h5POBXEJS4HZ7
ry1vDsgFV5kOqFppeMtynbhgE2YsYK64nzeFQnAr1uJQJXQAAAIEAtfaiakkVWQlgZnu89LTEPYOmUbM
4MziGpKqZyaBBpjyT/6ugTGahuRKI4br0ESCVjxmKqGkR1O5jOSNBvMr2zbWRpnVgCM8ThDH1OuvPdZu
2aSAX2KCnIga+Wzj8/uWCCDHRj+2D0l7vGgQXImJMlJ4Sa9xyOMJH8qU6Un0Gscs= Rdate Command
Key

As root, in the global zone, lets apply a restrictive shell, and limit the PATH with a profile that's owned by root (this helps limit destructive potential, if someone were to try to use the passphrase-less ssh-key for no good:

bash-3.00# cp /usr/bin/bash /usr/bin/rbash
usermod -s /usr/bin/rbash proxy
echo "export PATH=\"~/bin\"" > /export/home/proxy/.profile
chmod 444 /export/home/proxy/.profile
chown root /export/home/proxy/.profile

Play around with an rbash shell, and you'll see that commands are restricted to what's in the path, and you can't have a '/' in any commands to be run.

Now, you could use a script like Brendan Gregg's zcp to copy files from the global zone to the non-global zone, but sometimes if it's just a little file like this, I'll just go ahead an copy it to the non-global zone's TMP directory.

cp /export/home/proxy/.ssh/rdate* /zones/zone1/root/tmp/

Now, let's log into the zone, and setup a script that can set the date and time
for the whole system.

zlogin -e @ -C zone1

mkdir /etc/proxy
mv /tmp/rdate* /etc/proxy/
cd /etc/proxy
vi setdate
-----------------------------------------------------------------------
#!/bin/sh
# quick script to set the date
/usr/bin/ssh -T -a -i /etc/proxy/rdate.key proxy@<global zone IP>
-----------------------------------------------------------------------

So there you are. A simple command you can run in a non-global zone, to set the date and time for the entire system, which until zones support a separate clock from the global, is as best you can get. (Timezone's not included)

However, you're not really setting the time in the non-global zone, you're causing a process to log into the global zone and set the time there.

If you're primary concern is an application that runs in a zone, that needs permission to set the time/date, then this should work for you, as long as you don't have any objections to using ssh keys this way, from a philosophical or security-minded point of view.

So, really:
Q: Can you set the time and date from a non-global zone?
A: No. but you can automate setting it in the global zone, from a non-global zone.

If you're behind a firewall that doesn't allow access to anywhere on port 37, and thus your Global zone also cannot set the time and date from a Time Server, then you'll need to set up some kind of dynamic ssh tunnel to ssh one step further out, and tunnel that rdate command through that ssh tunnel.

It took me two days to realize that my sound wasn't working on the Solaris Community Express installation on my Toshiba Tecra M2. It took me a couple of hours of digging, both on the system, and on Google, to find links to a couple of articles that between them had the information I needed to get sound working. It's actually very simple.

I downloaded and installed the TOOLSi810 package from the tools.de site, and then did the following:

update_drv -d -i '"pci8086,24c5"' audio810
update_drv -a -i '"pci8086,24c5"' audioi810

The first line removes the Sun driver that doesn't work with my hardware, and the second line adds the TOOLS driver_alias.
Note that the PCI addresses are wrapped in single-quotes outside of double-quotes.
Now, reboot and you'll be all set. Hopefully.

So. I deleted the Linux JDS installation from my laptop, reclaimed the filesystems, and prepared myself to install Solaris Community Express build nv_18. I downloaded and burned the 4 CD images, plus the Software Companion CD image for Solaris 10, and proceeded to to the install.

With the size of disks these days, I tend to just do an "Entire Distribution + OEM" install, and today was no exception. Everything went very smoothly, and I didn't have any problems with the installation.

I saw where some people had issues with X.org, and the Toshiba Tecra M2. I didn't have those problems, as the screen detection worked just fine for installation purposes, 1024 x 768 x 256 colors. Not great, but usable. Then I downloaded and installed the Nvidia Drivers for Solaris x86. Use the Nvidia README. If you follow the instructions step by step, you'll have no problems. I found that the xorg.conf installed by the Nvidia drivers works just fine on my Tecra M2.

I found quite a few blogs here @ blogs.sun.com talking about Solaris 10 and Community Express x86 on the Toshiba Tecra M2. Folks like Chris Gerhard and Josh Simons have some good tips, and advice, and Eric Boutilier has some generally interesting content regarding Solaris and OpenSolaris, and what's going on in that whole 'movement'.

Thanks to Chris, Josh, and Eric, for the information they've had available, as well as the anonymous masses on various forums, bulletin-boards, and to the Casper Dik's of the world for their nifty tools, and tips, and stuff.