So I'm helping David with some ZFS issues over on the OpenSolaris NFS discussion forum, and I realized that for Solaris, NFSv4 is ubiquitous. The problem he was having was in getting FreeBSD to access a ZFS snapshot. I happened to know that with the releases in the wild, ZFS doesn't play well with NFSv3. And a FreeBSD client would by default be running NFSv3.

I showed an example:

• NFSv3 fails:

  [tdh@adept .zfs]> cd snapshot/
  [tdh@adept snapshot]> ls -al
  total 0
  dr-xr-xr-x+ 4 root root 4 Mar 21 17:52 .
  dr-xr-xr-x+ 3 root root 3 Mar 21 17:52 ..
  ?---------  ? ?    ?    ?            ? monday
  ?---------  ? ?    ?    ?            ? monday.1
  [tdh@adept snapshot]> cd monday
  monday: Object is remote.
  [tdh@adept snapshot]> uname
  Linux
  [tdh@adept snapshot]> mount | grep wont
  wont:/export/zfs on /net/wont/export/zfs type nfs
  (rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
  wont:/export/zfs/tdh on /net/wont/export/zfs/tdh type nfs
  (rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
  wont:/export/zfs/nfsv4 on /net/wont/export/zfs/nfsv4 type nfs
  (rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
  wont:/export/zfs/nfsv3 on /net/wont/export/zfs/nfsv3 type nfs
  (rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
  wont:/export/zfs/nfsv2 on /net/wont/export/zfs/nfsv2 type nfs
  (rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
  [tdh@adept snapshot]> mount | grep wont
  wont:/export/zfs on /net/wont/export/zfs type nfs
  (rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
  wont:/export/zfs/tdh on /net/wont/export/zfs/tdh type nfs
  (rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
  wont:/export/zfs/nfsv4 on /net/wont/export/zfs/nfsv4 type nfs
  (rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
  wont:/export/zfs/nfsv3 on /net/wont/export/zfs/nfsv3 type nfs
  (rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
  wont:/export/zfs/nfsv2 on /net/wont/export/zfs/nfsv2 type nfs
  (rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
  

• NFSv4 works:

  [tdh@adept snapshot]> sudo mkdir -p /nfsv4/wont/export/zfs/nfsv4
  [tdh@adept snapshot]> sudo mount -t nfs4 wont:/export/zfs/nfsv4
  /nfsv4/wont/export/zfs/nfsv4
  [tdh@adept snapshot]> cd !$
  cd /nfsv4/wont/export/zfs/nfsv4
  [tdh@adept nfsv4]> ls -la
  total 16
  drwxr-xr-x  2 nfsv4 nobody     4 Mar 20 01:44 .
  drwxr-xr-x  3 root  root    4096 Mar 21 18:01 ..
  -rw-r--r--  1 nfsv4 nobody     0 Mar 20 01:04 it
  dr-xr-xr-x  3 root  root       3 Mar 21 17:55 .zfs
  -rw-r--r--  1 nfsv4 nobody 13867 Mar 20 01:47 zfs.txt
  [tdh@adept nfsv4]> cd .zfs
  [tdh@adept .zfs]> ls -la
  total 1
  dr-xr-xr-x  3 root  root   3 Mar 21 17:56 .
  drwxr-xr-x  2 nfsv4 nobody 4 Mar 20 01:44 ..
  dr-xr-xr-x  4 root  root   4 Mar 21 17:56 snapshot
  [tdh@adept .zfs]> cd snapshot/
  [tdh@adept snapshot]> ls -la
  total 2
  dr-xr-xr-x  4 root  root   4 Mar 21 17:56 .
  dr-xr-xr-x  3 root  root   3 Mar 21 17:56 ..
  drwxr-xr-x  2 nfsv4 nobody 4 Mar 20 01:38 monday
  drwxr-xr-x  2 nfsv4 nobody 4 Mar 20 01:44 monday.1
  [tdh@adept snapshot]> cd monday
  [tdh@adept monday]> ls -la
  total 11
  drwxr-xr-x  2 nfsv4 nobody     4 Mar 20 01:38 .
  dr-xr-xr-x  4 root  root       4 Mar 21 17:56 ..
  -rw-r--r--  1 nfsv4 nobody     0 Mar 20 01:04 it
  -rw-r--r--  1 nfsv4 nobody 11808 Mar 20 01:38 zfs.txt
  

This says something quite powerful, ZFS was released to the public in OpenSolaris not working correctly with NFSv3. In turn, that shows the power of OpenSolaris, you get early access to cool technology. But also, it shows that NFSv4 is the defacto standard for development withing Sun. Think about it, NFSv4 is on by default - Solaris 10 clients connecting to Solaris 10 servers will speak NFSv4 to each other.

Another neat thing is that with Solaris 10, automounting will be default occur over NFSv4. The variables which control all of this are in /etc/default/nfs:

# Sets the minimum version of the NFS protocol that will be registered
# and offered by the server.  The default is 2.
#NFS_SERVER_VERSMIN=2

# Sets the maximum version of the NFS protocol that will be registered
# and offered by the server.  The default is 4.
#NFS_SERVER_VERSMAX=4

# Sets the minimum version of the NFS protocol that will be used by
# the NFS client.  Can be overridden by the "vers=" NFS mount option.
# The default is 2.
#NFS_CLIENT_VERSMIN=2

# Sets the maximum version of the NFS protocol that will be used by
# the NFS client.  Can be overridden by the "vers=" NFS mount option.
# If "vers=" is not specified for an NFS mount, this is the version
# that will be attempted first.  The default is 4.
#NFS_CLIENT_VERSMAX=4

# Determines if the NFS version 4 delegation feature will be enabled
# for the server.  If it is enabled, the server will attempt to
# provide delegations to the NFS version 4 client. The default is on.
#NFS_SERVER_DELEGATION=on

# Specifies to nfsmapid daemon that it is to override its default
# behavior of using the DNS domain, and that it is to use 'domain' as
# the domain to append to outbound attribute strings, and that it is to
# use 'domain' to compare against inbound attribute strings.
#NFSMAPID_DOMAIN=domain

When a client contacts a server, it says "I want access via NFSv4.". The server can reply that version is not supported and then negotiation occurs. Looking at the defaults, we see that a mount request will be via NFSv4. If a Solaris 9 client comes in, it doesn't know anything at all about NFSv4 and it will start with NFSv3.

I can of course get NFSv3 via a command sequence like:

# sudo mount -o vers=3 wont:/export/zfs/nfsv4 /mnt

The Solaris team made a great design choice here, they extended the existing option set to change the version. I.e., to explicitly get a NFSv4 mount:

# sudo mount -o vers=4 wont:/export/zfs/nfsv4 /mnt

Any existing systems which are upgraded to Solaris 10 will either:

• Use NFSv[2|3] if their automounts or /etc/vfstab already contain explicit 'vers=' option settings.
• Use NSFv4 by default.

A more subtle impact here is that the same automounter maps can be shared with earlier Solaris boxes, boxes which have their /etc/default/nfs defaults changed, other OSes. By not explictly mentioning the version, we keep flexibility.

In contrast the Linux designers chose to instead use a new filesystem type to represent NFSv4 - '-t nfs4'. If you look closely at my prior example, you would see I'm running Linux on adept:

[tdh@adept snapshot]> sudo mount -t nfs4 wont:/export/zfs/nfsv4

The Linux design was selected to make people aware that they were changing protocols. I don't like it, but I can understand why they made that choice.

The impact is that you do have to explicitly change your automount maps to get NFSv4 and you can't share them. I'm going to add this to the FAQ at NFSv4 in Action, because if you try to capture packet traces showing a NFSv4 problem and you inadvertantly go to /net, you can get confused.

So last night I reinstalled Nevada b35 on wont. Of course, I didn't bother nuking my ZFS filesystem. And of course it did not show up after the system came up. But that is the same as a UFS filesystem. For UFS you need to add it to /etc/vfstab to get it mounted automatically upon a reboot.

What do I need to do with a ZFS filesystem?

• Let's find it first:

  # zpool import
    pool: zoo
      id: 6577446991347315550
   state: ONLINE
  action: The pool can be imported using its name or numeric identifier.  The
          pool may be active on on another system, but can be imported using
          the '-f' flag.
  config:
  
          zoo         ONLINE
            mirror    ONLINE
              c0d1s0  ONLINE
              c0d1s1  ONLINE
            mirror    ONLINE
              c0d1s3  ONLINE
              c0d1s4  ONLINE
  

• And now lets import it:

  # zpool import zoo
  cannot import 'zoo': pool may be in use from other system
  use '-f' to import anyway
  # zpool import -f zoo
  cannot mount 'zoo/home/nfsv2': mountpoint or dataset is busy
  

• And lets verify that we can see it:

  # df -h
  Filesystem             size   used  avail capacity  Mounted on
  /dev/dsk/c0d0s0         30G   4.3G    25G    15%    /
  /devices                 0K     0K     0K     0%    /devices
  ctfs                     0K     0K     0K     0%    /system/contract
  proc                     0K     0K     0K     0%    /proc
  mnttab                   0K     0K     0K     0%    /etc/mnttab
  swap                   1.5G   716K   1.5G     1%    /etc/svc/volatile
  objfs                    0K     0K     0K     0%    /system/object
  /usr/lib/libc/libc_hwcap2.so.1
                          30G   4.3G    25G    15%    /lib/libc.so.1
  fd                       0K     0K     0K     0%    /dev/fd
  swap                   1.5G   108K   1.5G     1%    /tmp
  swap                   1.5G    44K   1.5G     1%    /var/run
  /dev/dsk/c0d0s7        5.7G   5.8M   5.6G     1%    /export/home
  /vol/dev/dsk/c1t0d0/"kanigixx86"
                         2.8G   2.8G     0K   100%    /cdrom/"kanigixx86"
  zoo                    134G   100K   128G     1%    /zoo
  zoo/x86                134G   2.8G   128G     3%    /zoo/x86
  zoo/home                10G   100K  10.0G     1%    /export/zfs
  zoo/home/nfsv2          10G    98K  10.0G     1%    /export/zfs/nfsv2
  zoo/home/tdh            10G    98K  10.0G     1%    /export/zfs/tdh
  zoo/home/nfsv3          10G    98K  10.0G     1%    /export/zfs/nfsv3
  zoo/home/nfsv4          10G   108K  10.0G     1%    /export/zfs/nfsv4
  zoo/isos               134G   2.8G   128G     3%    /zoo/isos
  

No idea why 'zfs import -f zoo' spit out a warning. Well, it might have to do with the fact that I didn't do a 'zfs export' before I did the reinstall.

And finally, lets make sure we can see the contents across the network on sandman:

# showmount -e wont
export list for wont:
/export/zfs       (everyone)
/export/zfs/tdh   (everyone)
/export/zfs/nfsv3 (everyone)
/export/zfs/nfsv4 (everyone)
/export/zfs/nfsv2 (everyone)
# cd /net/wont/export
# ls -la
total 3
dr-xr-xr-x   2 root     root           2 Mar 21 15:36 .
dr-xr-xr-x   2 root     root           2 Mar 21 15:36 ..
dr-xr-xr-x   1 root     root           1 Mar 21 15:36 zfs
# cd zfs
# ls -la
total 7
drwxr-xr-x   6 root     sys            6 Mar 20 23:12 .
dr-xr-xr-x   2 root     root           2 Mar 21 15:36 ..
dr-xr-xr-x   3 root     root           3 Mar 21 15:35 .zfs
dr-xr-xr-x   1 root     root           1 Mar 21 15:36 nfsv2
dr-xr-xr-x   1 root     root           1 Mar 21 15:36 nfsv3
dr-xr-xr-x   1 root     root           1 Mar 21 15:36 nfsv4
dr-xr-xr-x   1 root     root           1 Mar 21 15:36 tdh
# cd tdh
# ls -la
total 4
drwxr-xr-x   2 1066     staff          2 Mar 20 23:12 .
drwxr-xr-x   6 root     sys            6 Mar 20 23:12 ..
dr-xr-xr-x   3 root     root           3 Mar 21 15:36 .zfs

Obviously, the userid of 1066 has not been added to sandman. Notice that this is a NFSv4 mount and the ID mapping must be working correctly. I.e., if it were not, we would see nobody. A NFSv3 mount would show 1066 regardless of the ID domain settings.

I'm going to preface this with I should have been testing with my Sparc systems and not the x86. Can we say the default installation is hosed with respect to headless systems?

So I decided the next experiment was going to:

• Replace the grub splash screen with my own.
• Try and force the installation to automatically install the Software Companion packages.

I also decided to remove the Sparc packages from the DVD - eventually I am going to want that space for something else.

Well, I decided to not do my installation testing on wont. I want to do it on an old laptop I have, but I can't find its modular DVD reader and it won't boot from USB connected drives. I don't want another PC in the house, so I decided to use my w2100z.

It is hooked up via a serial console. This is a very important fact. And that is foreshadowing again - the sign of good journalism.

Pop in the new DVD, reboot and stare at the following for 5 minutes:

# sync; sync; sync; halt
Mar 21 00:49:07 warlock halt: initiated by root on /dev/console
syncing file systems... done
Press any key to reboot.
Resetting...

Add in the lack of a decent power reset switch on the w2100z, and the frustration mounts. I got it back up on Nevada b34 and then popped the DVD back into wont. And the reason why I didn't get anything on the earlier console was really evident. The default grub selection is to boot to the attached monitor and not the console. If you are on the box, you can use the attached keyboard to get the selection to go to the console.

Headless boxes are pretty common with Solaris. I'll have to look to see how this is being handled in the labs.

I'm sure I could easily make headed and headless versions of the DVDs. By why waste a DVD?

Oh, by the way, the DVD boots on wont. I didn't see the splash screen, perhaps when the system reboots from the HD? Also, when the system prompted me for where to search for software to install, the DVD was greyed out. That might have nuked my second goal.

As an added bonus, I have some ZFS filesystems on the disk I didn't nuke. It will be interesting to see how they come up.

Ugh, I have my test system back - the modular DVD on the old laptop does not go in the modular bay - it is integrated. :-> I can swap the CD writer out for a floppy drive. Oh well, the first DVD version is installing on it.

Okay, I changed too many things at once on wont - I also added the extra software - which messed the initial boot up.

I'm not getting my splash screen.

The laptop is hung on:

Discovering additional network configuration...

It has a cardbus based network card.

So the good news is that I can repeat my orginal experiment to create a bootable x86 DVD. The bad news is that nothing new worked.

Strike that, I was able to modify the string name which appeared on the Gnome display of the DVD:

I did that with this entry in my ~/.mkisofsrc: Doh!, I nuked that area on wont :->

I really need to get that laptop bootable - even without the NIC it does the same thing.

I'll try to tie up all the loose ends sometime in the next day or so. Oh, I also can't see my ZFS filesystem, but I'll recover it as well.

I added some links to allow the RSS feeds for both Kool Aid Served Daily and NFSv4 in Action to be added to user accounts of the Alesti RSS Reader.

They will look like this:

www.alesti.org