Friday April 28, 2006

Zones Top Ten

For quite a while I've played with Solaris 10 Zones. Even made very good use of them for isolating development projects in a sandbox environment. And so far, all I needed could be done with a simple 'zonecfg', 'zoneadm install', 'zoneadm boot' and then 'zlogin'. Recently I started to dig a little deeper and ran into problems with networking between local zones. And I had issues with that, partly by not enough RTFM :-) and partly by real issues. The real problems were mainly caused by trying this on a system running Nevada with BFU-ed on top of that the BrandZ stuff. Some piece in ZFS broke a dependency in my SMF.

This article is about where I went "off the right track" while setting up my zones. What was obvious and what wasn't. Often because somewhere I read something that was dead wrong (yes, also on blogs :-) or at least incomplete. Because I'm a Letterman fan, I've decided to do this in a Top Ten format. Similar to the Late-Show, don't pay too much attention to the ranking.

1		This is important: during the install of a zone, never-ever start to "peek 'n poke" around in your local zone's directory to see how far things have come. Don't even think about it!! I know, it is especially tempting when the install takes a little longer, which happens on a slower system. Just by doing a 'cd' in to your zones directory tree, you will prevent nfs mounts to be unmounted and the install will mess up hopelessly. Underwater, what is happening during install is a combination of nfs mounting and hard links (not symbolic links).
2		There are full and sparse zones. The thing that matters here is whether you specify a "-b" at your "create" in zonecfg. The man-page says about this "Use the -b to create a blank configuration. Without arguments, create applies the Sun default settings." Now you would expect that the default is to inherit nothing and therefore create a full zone. But it is the other way round. With the default come automatic inherits for /lib, /platform, /sbin and /usr. Because /bin is linked to /usr/bin, that one is also inherited from the global zone. Now by adding the "-b" when you create a zone, you don't have these inherits and therefore you'll get a full zone, which takes roughly two gigs per zone. I always end my zonecfg with the sequence "verify", "commit" and "info". The latter shows you clearly what your inherited packages are, even when you use the default.
3		Many READMEs and HOWTOs on Zones only mention 'zonecfg', 'zoneadm install', 'zoneadm boot' and then 'zlogin'. What is missing there is that the first time you do a 'zoneadm boot' all the services get initialized and the system needs to be network configured. So, before booting the zone for the first time, open another window and enter 'zlogin -C thezone'. This will open a console for that zone (close it with '~.'). When you now boot, you will see that first the services get initialized and then you have to answer the typical questions about network, naming service, locale, etc. At the end the zone will reboot and now you can straight away 'zlogin' into your zone.
4		Instead of doing 'zoneadm -z thezone reboot' from the global zone, you can just as well do a normal Solaris 'reboot' command from within the local zone. This makes even more sense when you are connected to the zones console ('zlogin -C').
5		If you failed to have a console open when the zone was booted for the first time, you can do the following: use 'zlogin -C' to open a console into your zone and login as root. Execute 'sys-unconfig', this will halt the zone. Now, in another window, boot the zone again with 'zoneadm -z thezone boot'. In the console window you will now see the same questions asked as during the first boot.
6		The forums are full of people having trouble with not being able to ssh into their zones. For a while I had the same issues. Many places recommend that you manually create your ssh keys (when missing) with "/lib/svc/method/sshd -c". Which will create the keys, but doesn't solve the underlying problem. Which is that you didn't go trough a proper configuration process after installing the zone. See at [3].
7		If you've configured DNS as your naming service and you get after reboot that typical message from sendmail with "unable to qualify my own domain name", the solution is to edit your /etc/hosts file from 127.0.0.1 localhost 192.168.1.91 thezone loghost into 127.0.0.1 localhost 192.168.1.91 thezone thezone.thedomain.the loghost IMHO it doesn't make sense, but it fixes the problem.
8		I discovered that hacking the zoneconfigs is pretty straightforward and fun. You know the slew of disclaimers that should follow here of course . In your global zone, go to /etc/zones. There is an 'index' and for each zone an XML file that has the zone's configuration. In the file 'index' you can change the zone directory, but I wouldn't start messing with that GUID. Also, I think the zone status field is better left alone. If you start editing the index or the XML file, keep the two consistent. This hacking is of course dangerous stuff, but I used it successfully when I couldn't delete a corrupted zone.
9		To change the behaviour of what happens with a 'create' and 'create -b' in zonecfg, the /etc/zones directory contains two xml files: SUNWdefault.xml and SUNWblank.xml. At least have a look at these to determine what the difference is between the two.
10		Somewhere along the way in Nevada, I guess around build #30, an option was added to move or clone a zone. Very cool!!

That's it for now. Happy hacking and have fun with your zones ...

Friday April 21, 2006

Serial ports on Sun Ultra 20

Like so many PCs today, the Sun Ultra 20 workstation doesn't come with "legacy" ports. No serial or parallel anymore, it's all USB. And for some that's a problem, like when you need to connect serial matrix printers for receipts, invoices, etc.

The Ultra 20 is based on a Tyan S2865 motherboard. But with some modifications: the COM1 port has been removed completely (although the solderpins are still there) and COM2 is only a 10 pin header.

Trying to use the COM2 port failed at first, because by default all legacy ports are disabled in the BIOS. To fix this press F2 while booting, go to the "Integrated Peripherals", select "Onboard Serial Port 2" menu and enable the port. You can leave the IO and IRQ as default (2F8 / IRQ3). Finally leave the BIOS by choosing "Save & Exit Setup".

Now you need a cable and a bracket with a DB9 connector. You can do this two ways. You can probably avoid a lot of trouble by buying this from Cables To Go. I didn't test it myself, but these seem to be the right type. If you want to go the DIY route, you must be careful. On many serial or serial/parallel I/O boards, the DB9 plugs on the bracket are connected to the PCB header, according to the right side of the diagram.

click to enlarge

The wiring on the left shows how it must be. If you would use a cramp-on DB-9 plug you're automagically good-to-go. Hoever all the cables on serial/parallel boards follow a strict 1-1, 2-2, 3-3 to 9-9 wiring, while it needs to be 1-1, 2-6, 3-2, 4-7, etc. The reason is clear when you look how a header plug is numbered differently from a DB-9 plug.

OK, long story, but knowing the solution, it's all pretty simple. Use a cramp-on DB-9 (male !!), 30 cm of flatcable, find in your junk box a bracket with a hole for the DB-9 or buy one on eBay and off-you-go.

Sunday April 16, 2006

Shell Hell

Unix shell programming appears to be an art, not a science. That's probably also why I'm not so much of a Perl fan. Take note, I do like both scripting and art a lot, but when it comes to Computer Science (mind that last word :) I think computer languages should be very unambiguous and easy to read. If that makes a compiler's life a little harder or means a bit more typing for the programmer, so be it....

But this story is not about bashing Perl, it's about unexpected problems with simple shell scripting. I was installing W3C's libwww package to get going with some XML-RPC development. It was the typical process: "./configure" followed by "gmake", nothing fancy. The configure step went fine, but while gmaking, it hicked up severely. I traced it back to some "test" commands in the libtool script, when the condition of 'test' had an empty variable.

Something like:

#!/bin/sh
something=""
if test $something = "yes"
then
    echo "something is yes"
fi

Or when the something="" statement had been absent at all. In both cases, the result is an error at the "if test" line.

The cause of all this trouble is that for undefined variables there is a difference between

if test $something = "yes"; then

and

if test "$something" = "yes"; then

Mind the additional quotes. The libwww package is missing those a couple of times, which appears to be fine when the variable has a value, but not when it is undefined.

Back to art vs science. That things like this are illegal syntax I'm OK with. But it should be made clear up front. Preferably as a compile error, although I realize that for scripting languages that is not too applicable. I had hoped for an error like "line 13: illegal syntax" or something similar. But that's not what I got:

./test.sh: test: argument expected

So it took me 10 mins and many more echo statements to find out which of all those test commands in the 5000+ lines of code script was the culprit. After finding it, of course the problem repeated itself, quite a couple of times. It took a while to get this script fixed, but don't worry, I did ... :-)

Later, when redoing the install on RedHat, I discovered RH8 is having the same issues with the test command, but it states:

./test.sh: line 13: test: =: unary operator expected

The message is even more cryptic, but it has the big advantage of giving you a line number. It's the small things that matter :-). Which would have saved me many echo statements and a couple of hours.

With all these problems out of the way, I could dig into XMLRPC-C to build an interface between a Windows client and a Solaris backend. So far, I find XML-RPC to be an elegant protocol, which results in quick implementations and simple solutions.

Sunday April 09, 2006

the Ultra computer lab in my basement

Time to get this WebLog started. I waited long with initiating this blog, but that doesn't mean I'm "late on the web". My first web-server got fired up in 1995, at which time probably the toughest job was to put your hands on an old PC. Hard to imagine today, where you have to pay to throw away a computer, but at the time all PCs -- even a 386 -- were first generation and still hard to get at.

Anyway, I managed to steal one from somewhere, convinced the network administrator to give me a second static IP, put my first Linux (SlackWare) on the box, followed by the ncsa web-server. There wasn't much else, but it worked and then all the focus was on writing content, of course in raw HTML.

Let's skip forward, it's more than 10 years later. This picture of the lab in my basement is to illustrate what this weblog will be all about. You're looking at a mixture of Solaris X86, RedHat, Xinerama, TheWeatherNetwork, BrandZ, Cobalt Qube/RAQ, etc. That's what keeps me awake at night. :-)

I don't intend to write this blog about my food preferences, vacation travels, home reno's, etc. You can find all of that at www.schaik.com on my own domain, which I was lucky enough to secure in 1998.

On this WebLog, it will be all about hacking Solaris and Linux. A few years ago, when Sun's opinion of Linux was still very devided, I used (like many others within Sun) Linux in stealth mode at home and on my laptop. But my colleagues in the office knew how to find me to solve their "triple-boot" problems. Luckily Solaris and Linux are now treated equally, with for me the funny result that I'm currently using much more Solaris than Linux.

Of course with a good amount of hurdles. And that's the main reason for starting this blog. This will become my way to share my "solutions for problems" with the world. Yesterday, a Friday, it was one of those days to be frustrated about half the world just trying to make problems, while the other half is doing all their best to solve them. Unfortunately, I really beleave that that's the ratio. But this site will for 100% be part of the latter: "solving problems".

Final note: I saw this banner sign with "Unix Papeleria" during our last vacation into Mexico. I came it across in the beautiful historic university town of Guanajuato. My Spanish is pretty non-existent, but if I'm right, a papeleria is a shop where you buy your stationary. There is not much synergy between that and this blog, but simply because of the word Unix, I thought this good enough reason to make it my banner. Of course I'm still wondering if this is Mr. Unix or that there's something else going on .....