Kool Aid Served Daily

I've seen a big spike on my blog today. Normally, it means I've put on a lot of new content. But in this case, it means I'm finally on the blog list for OpenSolaris.

Now I blog quite a bit about OpenSolaris and I never bothered seeing if I was on that page. But I started looking at it, saw the stuff for NFSv4 in Action, and wondered what the criteria was to get my feed pulled there. It is simple, follow the instructions on the site page.

I also like playing games with google and my latest is with kanigix. I invented the term last week and registered a couple of domains - kanigix.org. It took a couple of days for it to show up, mainly after my posts on creating a bootable x86 DVD.

But what I found was interesting were the sites which are Agrregators. Now, I saw someone else get mad about this, (wow, I was able to hit this one: Making Money With Someone Else's Writing?), but I figure the intent is to be seen. If you are ranting, who cares how the message gets out? If you are doing a technical entry, who cares how the other person gets help from you?

But, it should be the case that a reader can easily get back to where you blog. I don't care for the page hits, but I do care about being quoted or allowing someone to find something I did before I was put on the agrregator. So I've stuck a copyright section on each of my posts - it also contains a link back to Kool Aid Served Daily.

By the way, I found myself on planetsolaris.org and planetsun.org. Both of these made sense. Also, so does the The League of Professional System Administrators. Some of the others do not.

Also, if I was concerned with page hits, I think I would take the javascript from statcounter.com and figure a way to embed it in every entry and only have it count once.

Or perhaps it is as simple as a protocol for Agrregators to pass back hits to the site being fed from?

So, I'm trying to get kanigix.org in shape and I wanted to add one of those little icons up in the URL bar of the browsers. One google later and I'm at http://www.html-kit.com/favicon.

I followed the instructions, upload my .jpg file, and got a new resource. I downloaded the favicon.ico, did not get a virus, and then uploaded the resulting file to my website.

I then added the following snippet of html to my pages:

<link rel="shortcut icon" href="/images/favicon.ico" >

Note that I have to use special escape characters in the raw html here in order to keep a blank line from appearing. It should really be '<' and '>'.

Okay, I got hot and bothered - I thought I found a bug in the ZFS code. Work with me as I show my reasoning and see if you say "Doh!" before I do...

I create some zfs filesystems:

# zpool create zoo c0d1s0 c0d1s1
# zfs create zoo/home
# zfs set mountpoint=/export/zfs zoo/home
# zfs set sharenfs=on zoo/home
# zfs set compression=on zoo/home
# zfs set quota=10G zoo/home
# zfs create zoo/home/nfsv2
# zfs create zoo/home/nfsv3
# zfs create zoo/home/nfsv4
# zfs list
NAME                   USED  AVAIL  REFER  MOUNTPOINT
zoo                   3.21G   131G  99.5K  /zoo
zoo/home               395K  10.0G  99.5K  /export/zfs
zoo/home/nfsv2        98.5K  10.0G  98.5K  /export/zfs/nfsv2
zoo/home/nfsv3        98.5K  10.0G  98.5K  /export/zfs/nfsv3
zoo/home/nfsv4        98.5K  10.0G  98.5K  /export/zfs/nfsv4
zoo/x86               3.21G   131G  3.21G  /zoo/x86

I then create some user accounts to go along with the new filesystems:

# useradd -m -u 1094 -g 100 -c "Mr. NFSv2" -d /export/zfs/nfsv2 nfsv2
# chown nfsv2:100 /export/zfs/nfsv2
# useradd -m -u 1813 -g 100 -c "Mr. NFSv3" -d /export/zfs/nfsv3 nfsv3
# chown nfsv3:100 /export/zfs/nfsv3
# useradd -m -u 3530 -g 100 -c "Mr. NFSv4" -d /export/zfs/nfsv4 nfsv4
# chown nfsv4:100 /export/zfs/nfsv4
# ls -al /export/zfs
total 10
drwxr-xr-x   5 root     sys            5 Mar 20 00:33 .
drwxr-xr-x   4 root     sys          512 Mar 20 00:31 ..
dr-xr-xr-x   3 root     root           3 Mar 20 00:40 .zfs
drwxr-xr-x   2 nfsv2    protos         2 Mar 20 00:33 nfsv2
drwxr-xr-x   2 nfsv3    protos         2 Mar 20 00:33 nfsv3
drwxr-xr-x   2 nfsv4    protos         2 Mar 20 00:33 nfsv4

This is on wont, an Opteron box, and I'm checking things out on sandman, an Ultra 5 (sparc). I haven't created users on it yet. What do I see:

# cd /net/wont/export/zfs/
# ls -la
total 6
drwxr-xr-x   5 root     sys            5 Mar 20 00:33 .
dr-xr-xr-x   2 root     root           2 Mar 20 00:56 ..
dr-xr-xr-x   3 root     root           3 Mar 20 01:00 .zfs
dr-xr-xr-x   1 root     root           1 Mar 20 00:56 nfsv2
dr-xr-xr-x   1 root     root           1 Mar 20 00:56 nfsv3
dr-xr-xr-x   1 root     root           1 Mar 20 00:56 nfsv4

So the bug is obviously the owner and group are not being reported correctly. It could be ZFS or NFSv4. I add an user account to check to see if it is an NFSv4 ID mapping problem:

#  useradd -m -u 3530 -m -g 100 -c "Mr. NFSv4" -d /export/home/nfsv4 nfsv4
64 blocks
# pwd
/net/wont/export/zfs
# ls -la
total 6
drwxr-xr-x   5 root     sys            5 Mar 20 00:33 .
dr-xr-xr-x   2 root     root           2 Mar 20 00:56 ..
dr-xr-xr-x   3 root     root           3 Mar 20 01:00 .zfs
dr-xr-xr-x   1 root     root           1 Mar 20 00:56 nfsv2
dr-xr-xr-x   1 root     root           1 Mar 20 00:56 nfsv3
dr-xr-xr-x   1 root     root           1 Mar 20 00:56 nfsv4

Nope, no luck. How about some content?

# su - nfsv4
Sun Microsystems Inc.   SunOS 5.11      snv_35  October 2007
$ ls -la
total 4
drwxr-xr-x   2 nfsv4    protos         2 Mar 20 00:33 .
drwxr-xr-x   5 root     sys            5 Mar 20 00:33 ..
dr-xr-xr-x   3 root     root           3 Mar 20 01:04 .zfs
$ touch it
$ ls -la
total 5
drwxr-xr-x   2 nfsv4    protos         3 Mar 20 01:04 .
drwxr-xr-x   5 root     sys            5 Mar 20 00:33 ..
dr-xr-xr-x   3 root     root           3 Mar 20 01:04 .zfs
-rw-r--r--   1 nfsv4    protos         0 Mar 20 01:04 it

And then if I check on the client:

> cd nfsv4
> ls -la
total 5
drwxr-xr-x   2 nfsv4    protos         3 Mar 20 01:04 .
drwxr-xr-x   5 root     sys            5 Mar 20 00:33 ..
dr-xr-xr-x   3 root     root           3 Mar 20 01:06 .zfs
-rw-r--r--   1 nfsv4    protos         0 Mar 20 01:04 it
> ls -la
total 7
drwxr-xr-x   5 root     sys            5 Mar 20 00:33 .
dr-xr-xr-x   2 root     root           2 Mar 20 00:56 ..
dr-xr-xr-x   3 root     root           3 Mar 20 01:07 .zfs
dr-xr-xr-x   1 root     root           1 Mar 20 00:56 nfsv2
dr-xr-xr-x   1 root     root           1 Mar 20 00:56 nfsv3
drwxr-xr-x   2 nfsv4    protos         3 Mar 20 01:04 nfsv4

And now the correct owner appears... Was it some name caching issue? Does it repeat?

We can see the NFSv4 ID mapping taking place:

> cd nfsv3
> ls -la
total 4
drwxr-xr-x   2 nobody   protos         2 Mar 20 00:33 .
drwxr-xr-x   5 root     sys            5 Mar 20 00:33 ..
dr-xr-xr-x   3 root     root           3 Mar 20 01:08 .zfs
> cd ..
> ls -la
total 8
drwxr-xr-x   5 root     sys            5 Mar 20 00:33 .
dr-xr-xr-x   2 root     root           2 Mar 20 00:56 ..
dr-xr-xr-x   3 root     root           3 Mar 20 01:08 .zfs
dr-xr-xr-x   1 root     root           1 Mar 20 00:56 nfsv2
drwxr-xr-x   2 nobody   protos         2 Mar 20 00:33 nfsv3
drwxr-xr-x   2 nfsv4    protos         3 Mar 20 01:04 nfsv4

So, before I cd into a directory, the client should Doh!

The problem here is that before I cd into the directory, the mount has not taken place. This is in /net and thus we are using the automounter. You could argue that since we are presenting a list of exported filesystems to the user, we should have already gone down into them and gathered some information.

What happens if we have 1300 or 25,000 ZFS user accounts here? And we have 1k or 25k clients trying to connect?

The answer is what we call a mount storm and it kills server performance.

I've seen this really kill NAS boxes in customer's data centers (Linux clients with the AMD automounter). I'm glad the Solaris automounter is not getting the data ahead of time.

Now that we know what is going on, lets have some fun. First, we let the automounts time out:

# ls -la
total 6
drwxr-xr-x   5 root     sys            5 Mar 20 00:33 .
dr-xr-xr-x   2 root     root           2 Mar 20 00:56 ..
dr-xr-xr-x   3 root     root           3 Mar 20 01:27 .zfs
dr-xr-xr-x   1 root     root           1 Mar 20 01:22 nfsv2
dr-xr-xr-x   1 root     root           1 Mar 20 01:22 nfsv3
dr-xr-xr-x   1 root     root           1 Mar 20 01:22 nfsv4

Lets confirm our hypothesis:

# cd nfsv4
# ls -la
total 5
drwxr-xr-x   2 nfsv4    protos         3 Mar 20 01:04 .
drwxr-xr-x   5 root     sys            5 Mar 20 00:33 ..
dr-xr-xr-x   3 root     root           3 Mar 20 01:27 .zfs
-rw-r--r--   1 nfsv4    protos         0 Mar 20 01:04 it
# cd ..
# ls -la
total 7
drwxr-xr-x   5 root     sys            5 Mar 20 00:33 .
dr-xr-xr-x   2 root     root           2 Mar 20 00:56 ..
dr-xr-xr-x   3 root     root           3 Mar 20 01:27 .zfs
dr-xr-xr-x   1 root     root           1 Mar 20 01:22 nfsv2
dr-xr-xr-x   1 root     root           1 Mar 20 01:22 nfsv3
drwxr-xr-x   2 nfsv4    protos         3 Mar 20 01:04 nfsv4

And lets try to nuke the stuff:

# chown nfsv4 nfsv3
chown: nfsv3: Not owner
# ls -la
total 8
drwxr-xr-x   5 root     sys            5 Mar 20 00:33 .
dr-xr-xr-x   2 root     root           2 Mar 20 00:56 ..
dr-xr-xr-x   3 root     root           3 Mar 20 01:27 .zfs
dr-xr-xr-x   1 root     root           1 Mar 20 01:22 nfsv2
drwxr-xr-x   2 nobody   protos         2 Mar 20 00:33 nfsv3
drwxr-xr-x   2 nfsv4    protos         3 Mar 20 01:04 nfsv4
# useradd -m -u 1094  -m -g 100 -c "Mr. NFSv2" -d /export/home/nfsv2 nfsv2
64 blocks
# ls -la
total 8
drwxr-xr-x   5 root     sys            5 Mar 20 00:33 .
dr-xr-xr-x   2 root     root           2 Mar 20 00:56 ..
dr-xr-xr-x   3 root     root           3 Mar 20 01:30 .zfs
dr-xr-xr-x   1 root     root           1 Mar 20 01:22 nfsv2
drwxr-xr-x   2 nobody   protos         2 Mar 20 00:33 nfsv3
drwxr-xr-x   2 nfsv4    protos         3 Mar 20 01:04 nfsv4
# chown nfsv4 nfsv2
chown: nfsv2: Not owner
# ls -la
total 9
drwxr-xr-x   5 root     sys            5 Mar 20 00:33 .
dr-xr-xr-x   2 root     root           2 Mar 20 00:56 ..
dr-xr-xr-x   3 root     root           3 Mar 20 01:30 .zfs
drwxr-xr-x   2 nfsv2    protos         2 Mar 20 00:33 nfsv2
drwxr-xr-x   2 nobody   protos         2 Mar 20 00:33 nfsv3
drwxr-xr-x   2 nfsv4    protos         3 Mar 20 01:04 nfsv4

The automounter is lazy, it only goes to the server when it has to service a user request.

Evidently the name cache clears pretty quickly as well:

# useradd -m -u 1813 -m -g 100 -c "Mr. NFSv3" -d /export/home/nfsv3 nfsv3
64 blocks
# ls -la
total 9
drwxr-xr-x   5 root     sys            5 Mar 20 00:33 .
dr-xr-xr-x   2 root     root           2 Mar 20 00:56 ..
dr-xr-xr-x   3 root     root           3 Mar 20 01:32 .zfs
drwxr-xr-x   2 nfsv2    protos         2 Mar 20 00:33 nfsv2
drwxr-xr-x   2 nfsv3    protos         2 Mar 20 00:33 nfsv3
drwxr-xr-x   2 nfsv4    protos         3 Mar 20 01:04 nfsv4

Finally, why does it work correctly in the following sequence?

# mkdir -p /nfsv4/wont/zfs/nfsv2
# ls -la /nfsv4/wont/zfs/nfsv2
total 4
drwxr-xr-x   2 root     root         512 Mar 20 01:34 .
drwxr-xr-x   3 root     root         512 Mar 20 01:34 ..
# mount wont:/export/zfs/nfsv2 /nfsv4/wont/zfs/nfsv2
# ls -la /nfsv4/wont/zfs/nfsv2
total 4
drwxr-xr-x   2 nfsv2    protos         2 Mar 20 00:33 .
drwxr-xr-x   3 root     root         512 Mar 20 01:34 ..
dr-xr-xr-x   3 root     root           3 Mar 20 01:34 .zfs
# mkdir -p /nfsv4/wont/zfs/nfsv3
# ls -la /nfsv4/wont/zfs
total 8
drwxr-xr-x   4 root     root         512 Mar 20 01:35 .
drwxr-xr-x   3 root     root         512 Mar 20 01:34 ..
drwxr-xr-x   2 nfsv2    protos         2 Mar 20 00:33 nfsv2
drwxr-xr-x   2 root     root         512 Mar 20 01:35 nfsv3
# mount wont:/export/zfs/nfsv3 /nfsv4/wont/zfs/nfsv3
# ls -la /nfsv4/wont/zfs
total 8
drwxr-xr-x   4 root     root         512 Mar 20 01:35 .
drwxr-xr-x   3 root     root         512 Mar 20 01:34 ..
drwxr-xr-x   2 nfsv2    protos         2 Mar 20 00:33 nfsv2
drwxr-xr-x   2 nfsv3    protos         2 Mar 20 00:33 nfsv3

Because we explicitly do the mount request, which causes directory information to be gathered.