Kool Aid Served Daily

I haven't blogged much about company culture and the shock I had coming to Sun. I guess I felt I was integrated. I was wrong. In many ways, I am still an outsider. Day in and out, I sit in my office at home, disconnected from the company hivemind. I sometimes get asked by people outside Sun what the plans are and I feel they get offended when I say I don't know. I'm not holding out, I literally do not know.

Sure, I could look at past actions and what I know about the business to make predictions. But at best, they are just as likely to be correct as an industry analyst. It is like Battlestar Galactica, I don't know the writers/producers, but I can predict where the story arcs are going. Can I be wrong? You betcha, which is why I watch the show anyway.

One of the things which is really exciting me about Sun right now are the UI improvements being introduced between builds 50-55 of nevada. They might have started creeping in before then, but I really started to notice along then. The difference is drastic from Solaris 9 and even Solaris 10. I'm going to write about what I think is going on, but please remember, I'm not representing Sun or even the development engineers.

We had the great browser war and before that we had the great windowing system war. Microsoft seemed to win both. Netscape Navigator won on technical merits, but the masses adopted Internet Explorer. As for the Windows versus X battle, I hated it when I was at NetApp and everyone pretty much had to use Windows and Outlook. We needed the extra internal test cycles on hosting Exchange databases, so we converted desktops to Win 2000 and then WinXP.

I think Sun lost the desire to prove it was the better desktop. Either all the engineers got old enough or turned their attention to other matters. Let the young Linux upstarts do it. And they tried, but they never won. Full disclosure, my home terminal is a WinXP box running PuTTY: a free telnet/ssh client and my backup is Fedora Core 4 Linux. My work terminal is a Sun Ray 1G connected to a Solaris Nevada b55 w2100z box. I went back to WinXP because of games and I'm very happy with PuTTY's interface to the Solaris and Linux servers I have at home.

But a funny thing happened, Apple and MAC OS X showed that you could have a BSD core and pretty UI and win people over from WinXP. The installation of new hardware, the interaction between software, etc, was so easy that non-technical people could do it. I know of a lot of diehard Solaris developers who flocked to the Mac Book Pro. And the remains of Netscape reared their head in the form of Firefox to show that the browser war wasn't over either. Suddenly, it became viable to have a non-Microsoft OS which seamlessly handled multimedia and the Internet.

Now I don't think anyone at Sun woke up and said, "You know, we could grab the desktop." I'm not even sure such secret plans exist. But just as Sun has been pouring Star Office, NFSv4, dtrace, and zfs contributions down the open source well, they have been drawing out the innovations made in Gnome, Thunderbird (the mail client which got me off of mutt by the way), Firefox, etc. And you know what, the Solaris desktop experience is pretty fun. Put a dead quiet Sun Ray thin client on someone's desk and it becomes really pleasant.

I've found that I don't have to turn from my Sun Ray 1G to my WinXP box to go look at something on the web. Or to get a crisp terminal session going. As a matter of fact, I want to retire that box to just being a game box. Or even better, I'll have it dual boot WinXP and some OpenSolaris variant. Or perhaps get Xen or vmware running on it to have it serve up a bunch of OSes. It really doesn't matter, as long as I can kick off a game whenever my son feels like pitting his skills against mine.

I just saw an interesting question on one of the internal Sun developer lists:

ssh -fN -L "3049:somehost.someext:2049" jonb@somehost.someext
mount -o port=3049 localhost:/f /f

This allows you to mount a filesystem from a remote machine over a ssh tunnel. The question was why doesn't automounting also work?

so I have a

/-              auto_direct     -nosuid,rw

in auto_master

/f -rw,port=3049 localhost:/f

in auto_direct.

And since this is over Solaris, by default it is NFSv4 and we shouldn't need anything but port 2049. The question was why didn't this work. A good resource on this is at: Tunneling NFS traffic via ssh, which is provided by Spencer Shepler. In the comments, it specifically mentions that all you need is port 2049 for NFSv4 traffic.

In my reply on the mailing list, I pointed out that this does not apply to the auotmounter. It has to call on the server's portmapper, which is port 111. Let's repeat the experiment and see if we can prove this to be what is happening.

[tdh@sandman ~]> ssh -fN -L "3049:ultralord:2049" ultralord
The authenticity of host 'ultralord (192.168.2.104)' can't be established.
RSA key fingerprint is a5:e6:e6:9c:e2:9c:2e:d9:6c:d2:91:c0:ed:41:8f:38.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ultralord,192.168.2.104' (RSA) to the list of known hosts.
Password:

And

# mount -o port=3049 localhost:/f/1 /f
# ls -la /f
total 4
drwxr-xr-x   2 root     root         512 Dec 30 15:08 .
drwxr-xr-x  24 root     root         512 Dec 30 15:08 ..
# tcsh
# ls -la /f
total 4
drwxr-xr-x   2 root     root         512 Dec 30 15:11 .
drwxr-xr-x  24 root     root         512 Dec 30 15:08 ..
-rw-r--r--   1 root     root           0 Dec 30 15:11 ultralord
# showmount -e localhost
showmount: localhost: RPC: Program not registered

I'll claim I don't need to setup the automounter. I'll do that later to confirm things. Okay, let's try some snoop. Krap! snoop only works on non-loopback interfaces. I found a dtrace script which is supposed to work: DTrace TCP Snoop, but I need to work on it a bit more. Anyway, let's look at how showmount works over a regular interface:

[tdh@sandman ~]> showmount -e ultralord
export list for ultralord:
/f/1 (everyone)
/f/2 (everyone)

And the snoop:

# snoop sandman ultralord
Using device /dev/hme (promiscuous mode)
     sandman -> ultralord PORTMAP C GETPORT prog=100005 (MOUNT) vers=1 proto=TCP
     ultralord -> sandman PORTMAP R GETPORT port=36795
     sandman -> ultralord MOUNT1 C Get export list
     ultralord -> sandman MOUNT1 R Get export list 2 entries

Which can be translated as:

sandman asks ultralord's portmapper: "Do you support MOUNT vers 1 over TCP?"
ultralord replies: "Yeah and you can get it on port 36795."
sandman then asks ultralord's mountd: "What exports do you have?"
ultralord then replies: "/f/1 and /f/2"

It makes sense that the automounter would also use the mount protocol. It basically needs to do a MOUNTPROC3_DUMP (see RFC 1813) in order to determine what the server has available. In effect you would have to punch a hole for both port 111 and the port ultralord advertises mountd on. And nothing specifies that the server has to use the same port everytime.

With NFSv4, you do not need an automounter to discover all of the exports on a server. But the implementation shipped in Solaris 10 does not support the automatic mounting of new filesystems as you cross boundaries. In part, it was an engineering decision to ship NFSv4 without this functionality. You can use automounters to get some of the equivalent functionality. You also have to look at typical usuage of shares in Solaris pre-zfs - small sets of shares per server and no child shares.

The problem is given this set of shares on a server:

/    ro
/f   rw=.eng.sun.com,ro=@172.16
/f/1 rw=.lab.sun.com,anon=0
/f/2 rw,root=.lab.sun.com

How do you dynamically determine where you can go? We haven't specified that this is a Solaris server, so let's assume that '/', '/f', '/f/1', and '/f/2' are all different filesystems. Via the NFSv4 spec (RFC 3530) you should be able to do the following:

# mount -o vers=4 sandman:/ /sandman
...
[tdh@sandman ~]> cd /sandman
[tdh@sandman ~]> ls -la
total 6
drwxr-xr-x   3 tdh      staff        512 Dec 30 16:18 .
drwxr-xr-x   5 tdh      staff        512 Dec 30 16:18 ..
drwxr-xr-x   2 tdh      staff        512 Dec 30 16:18 f
[tdh@sandman ~]> cd f
[tdh@sandman ~]> ls -la
total 10
drwxr-xr-x   5 tdh      staff        512 Dec 30 16:18 .
drwxr-xr-x  24 root     root         512 Dec 30 15:07 ..
drwxr-xr-x   2 tdh      staff        512 Dec 30 15:11 1
drwxr-xr-x   2 tdh      staff        512 Dec 30 15:11 2
drwxr-xr-x   3 tdh      staff        512 Dec 30 16:18 3
[tdh@sandman ~]> cd 1
[tdh@sandman ~]> ls -la
total 4
drwxr-xr-x   2 tdh      staff        512 Dec 30 15:11 .
drwxr-xr-x   5 tdh      staff        512 Dec 30 16:18 ..
-rw-r--r--   1 tdh      staff          0 Dec 30 15:11 ultralord
[tdh@sandman ~]> cd ../3
[tdh@sandman ~]> ls -la
total 6
drwxr-xr-x   3 tdh      staff        512 Dec 30 16:21 .
drwxr-xr-x   5 tdh      staff        512 Dec 30 16:18 ..
drwxr-xr-x   2 tdh      staff        512 Dec 30 16:18 g

Under the current Solaris implementations, you would need to manually make the mounts before each of the cd commands. Except for the last, in that case you are staying on the filesystem represented by '/f'.

The NFS development team is actively working on fixing this issue, which we call Mirror Mounts. Once that functionality is in place, you will be able to explore a remote host across both a firewall and a ssh tunnel via only one port: 2049.

In Not all USB Thumbdrives are created equal, I made a zfs filesystem from a stack of 6 USB thumbdrives. Once I was done playing with them, I wanted to reformat them back to their stock configuration. No problem, right? WinXP does everything for you. Wrong. I tried to manage them via start->Control Panel->Administrative Tools->Computer Management->Storage->Disk Management(Local). Phew, what a mouthfull. Anyway, I got a menu much like this one:

You can right-click on the image to get a menu which will allow you to view the image. That way you can see it in full splendor.

No matter where I clicked, I could not change the partition. After a week or so, i.e., today, I put them in and tried again. This time instead of going to the Disk Managment tool, I clicked on the drives inside Explorer. As I clicked on them, I was told the format was unknown and did I want to reformat the drives:

I'd answer yes and get the above. Also, remember the drives were a mix of SMI and EMI labels.