everyone has a shangri-la to find: Lewis Thompson's weblog

The Problem

Recently one of my customers migrated a legacy SPARC system to a Solaris 8 branded zone on a new T5220 platform. This system provided an anonymous FTP service which no longer worked correctly within the branded Zone. More specifically, files could be uploaded & downloaded but directory listings always returned empty. Non-anonymous users reported no problems with any of the functionality

The Investigation

Initial investigation began by trussing in.ftpd within the Solaris 8 zone to see what was happening. Very quickly it was possible to see that in.ftpd forks and execs /bin/ls to generate the directory listing and this was failing. It's worth pointing out that on Solaris 8 anonymous FTP requires a chroot() environment (as per in.ftpd(1M)) and this was configured correctly

19710/1: execve("/bin/ls", 0x000353F0, 0xFFBFFE0C) Err#2 ENOENT

Very odd as ENOENT indicates 'no such file or directory' and /bin/ls definitely exists. Next step was to look more closely at the execve() syscall... I knocked up a very quick fbt DTrace script that would do this for me before realising that this was Solaris 8! Fortunately we can DTrace from the global zone, so after a few changes I can up with the following to trace the Solaris 8 exec calls:

#pragma D option flowindent

fbt::s8_elfexec:entry
/ execname == "in.ftpd" /
{ printf("s8_exec: execname: %s", execname); self->follow = 1; tracing = 1; }

fbt:::
/ self->follow /
{}

fbt:::return
/ self->follow /
{ trace(arg1); }

fbt::s8_elfexec:return
/ self->follow /
{ trace(arg1); self->follow = 0; exit(0); }

Running this from the global zone at the same time as issuing an LS in the anonymous FTP client and I could see that ENOENT was being returned from a call to lookupnameat(). The DTrace was updated with the following to print out the full filename of the file that was being looked up:

fbt::lookupnameat:entry
/ self->follow /
{ printf("%s", stringof(args[0])); }

Another LS from the client and the problem becomes clear -- we are trying to load /.SUNWnative/usr/lib/s8_brand.so.1. Until now I was not familiar with the /.SUNWnative directory but this exists on all Solaris 8 & 9 zones. Within this directory are three lofs mounts that present /lib, /usr and /platform to the local zone from the global:

/zones/s8_lt203398/root/.SUNWnative/lib on /lib read only/setuid/nodevices/dev=4010002 on Tue Apr 14 16:59:39 2009
/zones/s8_lt203398/root/.SUNWnative/platform on /platform read only/setuid/nodevices/dev=4010002 on Tue Apr 14 16:59:39 2009
/zones/s8_lt203398/root/.SUNWnative/usr on /usr read only/setuid/nodevices/dev=4010002 on Tue Apr 14 16:59:39 2009

Those that have kept up will now know why things are failing -- everything here hinges around anonymous FTP using a chroot() environment and those Solaris 8 zone-specific mounts not existing (why would they? in.ftpd(1M) was written years before Zones were around)

The Fix

Despite all of this investigation the real fix is far more obvious: don't use the legacy Solaris 8 FTP daemon when your system is running Solaris 10 (or OpenSolaris)!

The same functionality can be achieved by creating a minimal Solaris 10 local zone & setting up anonymous FTP there. The newer Solaris 10 ftpd provides additional functionality over Solaris 8. The last step in this method is to set up a loopback mount in the global zone to present the relevant directory into the Solaris 8 zone

The Workaround

Not everybody will have the option of adding an additional Solaris 10 zone to deliver "The Fix" so here is a workaround to get Solaris 8 anonymous FTP working. The good news is that it isn't at all hacky, it's just a few extra steps that are missing from in.ftpd(1M)

Armed with the knowledge that we are missing the /.SUNWnative mounts within the chroot() environment we simply add these in and things will begin to work. The updated mount output will now include:

/export/home/ftp/.SUNWnative on /.SUNWnative read/write/setuid/zone=s8_lt203398/dev=4010009 on Wed Apr 15 02:20:15 2009

Here my chroot environment exists under /export/home/ftp. And a more permanent way to achieve this is by updating your zone's vfstab with:

/.SUNWnative    -    /export/home/ftp/.SUNWnative    lofs    -    yes    ro

As a final note, in.ftpd(1M) didn't seem to mention that /lib/ld.so.1 should also be included in the chroot() environment. It should, and things won't work unless it is there. Don't forget to ensure the permissions are all set correctly

I have a three button mouse, but I mainly use two buttons: left and middle click. From time to time I use the right button

These days most mice have a scroll-wheel that doubles as middle click. They're not bad, but I prefer the left and right mouse buttons. So why not remap them?

On my Sun Ray I run the following:

% xmodmap -e "pointer = 1 3 2 4 5"

I think the syntax is very clear... 5 logical buttons are assigned by listing the physical buttons in order. i.e. the default is "1 2 3 4 5" but as I want to swap buttons 2 & 3, I transpose them in the list

I spend the larger part of my day collating outputs from customer systems (be they crash dumps, LDOM outputs, messages, etc.) and adding notes where appropriate. For example I might dump a vnode structure in mdb and want to draw particular attention to a specific member of that structure

Most CDAs I've read through use !! or <-- to draw attention to notes, or distinguish notes from output. I've followed this pattern and find it very useful day to day... but this can be improved

As a (g)vim user I added the following to my .vimrc file:

highlight CDANote ctermbg=red
match CDANote /<--.*$/
highlight CDANote2 ctermbg=red
2match CDANote2 /!!.*$/

And an obligatory screenshot:

Almost certainly there are better ways to achieve the same, but for day-to-day use, this makes reading notes and comments a great deal easier

udp_get_next_priv_port() from http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/inet/udp/udp.c
Assuming the system is unlabeled, what port will be returned when we're called (hint: it isn't IPPORT_RESERVED-1)?
No doubt this will be trivial for the C guru, but it had me scratching my head for a little while. It's probably time that I went back and read Programming 101...

/*
 * Return the next anonymous port in the privileged port range for
 * bind checking.
 */
static in_port_t
udp_get_next_priv_port(udp_t *udp)
{
        static in_port_t next_priv_port = IPPORT_RESERVED - 1;
        in_port_t nextport;
        boolean_t restart = B_FALSE;
        udp_stack_t *us = udp->udp_us;

retry:
        if (next_priv_port < us->us_min_anonpriv_port ||
            next_priv_port >= IPPORT_RESERVED) {
                next_priv_port = IPPORT_RESERVED - 1;
                if (restart)
                        return (0);
                restart = B_TRUE;
        }

        if (isystem_labeled() &&
            (nextport = tsol_next_port(crgetzone(udp->udp_connp->conn_cred),
            next_priv_port, IPPROTO_UDP, B_FALSE)) != 0) {
                next_priv_port = nextport;
                goto retry;
        }

        return (next_priv_port--);
}

Thanks to an invitation from Clive King I attended Cardiff university for the day to present on Solaris CIFS and VSCAN as part of a set of tech talks for academic customers

As promised to those guests I'm providing a copy of my slides (thanks Jarod & our lab interns), at the end of which there is the list of references along with links to the relevant sites. You can also check out the sites I visited while gathering the information for the slides as well as my notes

If there were any other questions, just drop a comment and I'll endeavour to give you an answer

Video playback (with codecs) is an absolute must to make OpenSolaris a workable alternative to Ubuntu for my home machine

What follows is a very quick guide to compiling MPlayer v1.0rc2 on OpenSolaris

Fetch gcc, gmake and gawk to allow us to compile MPlayer with minimum fuss; we also pull SUNWxorg-headers to allow us to compile the Xv video-out plugin (I also have FSWxorg-headers installed; if things don't work out, you may want to add that to the end of the list). As IPSgawk is in the Blastwave IPS repository, we add that also:

$ pfexec pkg set-authority -O http://blastwave.network.com:10000/ Blastwave
$ pfexec pkg install SUNWgcc SUNWgmake IPSgawk SUNWxorg-headers

Compiling MPlayer is now very straightforward, the only special thing we need to do is promote /opt/csw/gnu in our PATH during compile time. This is because various parts of the build fail with the standard Solaris awk, so we override with GNU awk:

$ echo $PATH
/usr/gnu/bin:/usr/bin:/usr/X11/bin:/usr/sbin:/sbin
$ export PATH=/usr/gnu/bin:/opt/csw/gnu:/usr/bin:/usr/X11/bin:/usr/sbin:/sbin
$ wget http://www8.mplayerhq.hu/MPlayer/releases/MPlayer-1.0rc2.tar.bz2
$ tar jxf MPlayer-1.0rc2.tar.bz2 && cd MPlayer-1.0rc2
$ ./configure
$ gmake

Key points to watch out for are the list of enabled audio and video plugins upon configure completing (you almost certainly want Xv)

If all went well, you should be able to run MPlayer with a simple ./mplayer -vo xv /path/to/video

In the meantime, I'm looking into the best way to get MPlayer packages available via one of the standard IPS repositories

I recently had a customer contact the Solutions Centre requesting a procedure to grow a Solaris Volume Manager (previously known as Solstice DiskSuite) RAID1 mirror using an old, unwanted volume.

In order to answer the customer's question authoritatively I turned to Sun's extensive 'Global Labs', where I booked a suitable host and installed the same version of Solaris as the customer. In this circumstance the global lab I picked was actually in the same building as me, but the procedure was done remotely using the lab tools. For all intents and purposes, the system could very well have been in the global lab in Singapore.

With Solaris installed I quickly duplicated the customer's configuration. He was using two disks in his mirror, giving him good redundancy:

d10 (RAID 1 mirror, 20GB; mounted on /export/home)
|- d11 (c1t0d0s0, 20GB)
\- d12 (c2t0d0s0, 20GB)

d20 (RAID 1 mirror, 30GB; mounted on /export/backup)
|- d21 (c1t0d0s1, 30GB)
\- d22 (c2t0d0s1, 30GB)


In this example, the customer is now using a remote host for backup, so the d20 metadevice is no longer being used. He wanted to grow d10 by the capacity of d20.

The following simple steps achieve this with 0% downtime. It is not even necessary to unmount the volume that is being grown:

1. Take a backup of /export/home. Verify this backup.


2. Unmount d20:

# umount /export/backup

3. Remove all references to the d20 mirror and all sub-mirrors with metaclear. In this step the -r flag indicates that d20, d21 and d22 will be removed:

# metaclear -r d20

4. Attach the slices previously used in the d20 mirror to each of the d10 sub-mirrors:

# metattach d11 c1t0d0s1
# metattach d12 c2t0d0s1

5. Tell d10 that the sub-mirrors have changed and that the size of the metadevice has changed:

# metattach d10

6. Finally, we need to use growfs to grow the UFS filesystem that lives on the d10 metadevice:

# growfs -M /export/home /dev/md/rdsk/d10

Hopefully this simple six-step procedure will be of use to those new to Volume Manager.

This is the info I've either been told or figured out while playing around with a couple of Sun Ray DTUs at home. It's possible it might not be 100% accurate, or I may have failed to include some behind-the-scenes operations that are not normally seen by the end-user (me). Hopefully it will still be some help.

The following steps are performed as part of a Sun Ray DTU (desk top unit) boot-up:

1. POST


2. Firmware verify


3. DHCP


4. Connect to server

During the DHCP stage the DTU expects all of the regular information (IP address, netmask, gateway, etc.), but in addition to this it needs some vendor-specific information. The vendor-specific tags tell the DTU which IP address to contact to check for new firmware, where the Sun Ray services are running, and which host to use for logging.

On Solaris, the standard utadm, dhcpconfig and dhtadm commands are used to manage the DHCP service. However, if you're trying to do interesting things without Solaris you'll quickly hit problems as it is very unlikely your DHCP server will understand the Sun Ray-specific options.

Linux systems often use the ISC dhcpd, which can be taught about these tags by adding the following to your dhcpd.conf file (from the standard Linux sample dhcpd.conf):

option space SunRay;
option SunRay.AuthSrvr code 21 = ip-address;
option SunRay.AuthSrvr IPADDRESSHERE;
option SunRay.FWSrvr code 31 = ip-address;
option SunRay.FWSrvr IPADDRESSHERE;
option SunRay.NewTVer code 23 = text;
option SunRay.NewTVer "3.0_51,REV=2004.11.10.16.18";
option SunRay.Intf code 33 = text;
option SunRay.Intf "INTERFACENAMEHERE";
option SunRay.LogHost code 24 = ip-address;
option SunRay.LogHost 192.168.1.101;
option SunRay.LogKern code 25 = integer 8;
option SunRay.LogKern 6;
option SunRay.LogNet code 26 = integer 8;
option SunRay.LogNet 6;
option SunRay.LogUSB code 27 = integer 8;
option SunRay.LogUSB 6;
option SunRay.LogVid code 28 = integer 8;
option SunRay.LogVid 6;
option SunRay.LogAppl code 29 = integer 8;
option SunRay.LogAppl 6;

Replace IPADDRESSHERE with the valid IPs of your authentication and firmware servers. INTERFACENAMEHERE should be set to the name of the interface DHCP requests are being accepted on (e.g. eth1). By default these options are not handed out to all clients. A group must be configured to use the specific options. e.g.

group {
    vendor-option-space SunRay;
    subnet 192.168.1.0 netmask 255.255.255.0 {
        authoritative;
        option routers 192.168.1.1;
        range 192.168.1.50 192.168.1.100;
    }
}

By using the above you gain full control over the information passed to your DTU. Using this it is very easy to allow a DTU to sit on a network well away from your Sun Ray and DHCP server... but wait, if simply allowing a DTU to contact a Sun Ray server on a different network is all you are looking to do, you can overload DHCP option 49: "X Window System Display Manager Option", RFC2132.

With ISC dhcpd this is configured by adding:

option x-display-manager IPADDRESSHERE(, IPADDRESSHERE, IPADDRESSHERE);

to the configuration file. Replace IPADDRESSHERE with the IP address of the Sun Ray server.

If, like me, you're using an embedded Linux device (e.g. Linksys WRT54g) to provide your DHCP, the chances are Dnsmasq is handling both DNS and DHCP for you. Add the following to your dnsmasq.conf to replicate the functionality just discussed:

dhcp-option=49,IPADDRESSHERE

If you want to know more, check out Configuring DHCP options 49 and 66 for Sun Ray use (used as a reference), this blog also covers configuring Windows DHCP server to provide option 49.

A little background information first...

I joined Sun in July 2006 as a student lab intern working for Paul Humphreys where I worked with four other students to provide hardware (and sometimes software) setups for Sun support engineers.

During this time I moved towards high-end hardware such as the E15/25k, Serengeti and Niagara-based systems. All the while I was doing my best to beef my knowledge of the Solaris kernel and userland and other assorted goodies (including the incredible Sun Ray). Solaris as a whole is the way I'd like to take things in the future, at least until I figure out what specifically interests me the most.

Two days ago I started as a full-time employee within the solutions centre where I will be working within the BT group as a support engineer.

And now...

Here I am, I have finally have the coveted blogs.sun.com blog and I plan to use this space to cover technical issues in some way related to Sun. Most entries will be basic, the odd one might be a little more advanced but overall I hope to provide some useful tips, tricks and info that might help somebody out when they're in a fix...