Tuesday March 21, 2006
Ubiquitous NFSv4 So I'm helping David with some ZFS issues over on the OpenSolaris NFS discussion forum, and I realized that for Solaris, NFSv4 is ubiquitous. The problem he was having was in getting FreeBSD to access a ZFS snapshot. I happened to know that with the releases in the wild, ZFS doesn't play well with NFSv3. And a FreeBSD client would by default be running NFSv3.
I showed an example:
- NFSv3 fails:
[tdh@adept .zfs]> cd snapshot/
[tdh@adept snapshot]> ls -al
total 0
dr-xr-xr-x+ 4 root root 4 Mar 21 17:52 .
dr-xr-xr-x+ 3 root root 3 Mar 21 17:52 ..
?--------- ? ? ? ? ? monday
?--------- ? ? ? ? ? monday.1
[tdh@adept snapshot]> cd monday
monday: Object is remote.
[tdh@adept snapshot]> uname
Linux
[tdh@adept snapshot]> mount | grep wont
wont:/export/zfs on /net/wont/export/zfs type nfs
(rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
wont:/export/zfs/tdh on /net/wont/export/zfs/tdh type nfs
(rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
wont:/export/zfs/nfsv4 on /net/wont/export/zfs/nfsv4 type nfs
(rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
wont:/export/zfs/nfsv3 on /net/wont/export/zfs/nfsv3 type nfs
(rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
wont:/export/zfs/nfsv2 on /net/wont/export/zfs/nfsv2 type nfs
(rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
[tdh@adept snapshot]> mount | grep wont
wont:/export/zfs on /net/wont/export/zfs type nfs
(rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
wont:/export/zfs/tdh on /net/wont/export/zfs/tdh type nfs
(rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
wont:/export/zfs/nfsv4 on /net/wont/export/zfs/nfsv4 type nfs
(rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
wont:/export/zfs/nfsv3 on /net/wont/export/zfs/nfsv3 type nfs
(rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
wont:/export/zfs/nfsv2 on /net/wont/export/zfs/nfsv2 type nfs
(rw,nosuid,nodev,hard,intr,addr=192.168.2.105)
- NFSv4 works:
[tdh@adept snapshot]> sudo mkdir -p /nfsv4/wont/export/zfs/nfsv4
[tdh@adept snapshot]> sudo mount -t nfs4 wont:/export/zfs/nfsv4
/nfsv4/wont/export/zfs/nfsv4
[tdh@adept snapshot]> cd !$
cd /nfsv4/wont/export/zfs/nfsv4
[tdh@adept nfsv4]> ls -la
total 16
drwxr-xr-x 2 nfsv4 nobody 4 Mar 20 01:44 .
drwxr-xr-x 3 root root 4096 Mar 21 18:01 ..
-rw-r--r-- 1 nfsv4 nobody 0 Mar 20 01:04 it
dr-xr-xr-x 3 root root 3 Mar 21 17:55 .zfs
-rw-r--r-- 1 nfsv4 nobody 13867 Mar 20 01:47 zfs.txt
[tdh@adept nfsv4]> cd .zfs
[tdh@adept .zfs]> ls -la
total 1
dr-xr-xr-x 3 root root 3 Mar 21 17:56 .
drwxr-xr-x 2 nfsv4 nobody 4 Mar 20 01:44 ..
dr-xr-xr-x 4 root root 4 Mar 21 17:56 snapshot
[tdh@adept .zfs]> cd snapshot/
[tdh@adept snapshot]> ls -la
total 2
dr-xr-xr-x 4 root root 4 Mar 21 17:56 .
dr-xr-xr-x 3 root root 3 Mar 21 17:56 ..
drwxr-xr-x 2 nfsv4 nobody 4 Mar 20 01:38 monday
drwxr-xr-x 2 nfsv4 nobody 4 Mar 20 01:44 monday.1
[tdh@adept snapshot]> cd monday
[tdh@adept monday]> ls -la
total 11
drwxr-xr-x 2 nfsv4 nobody 4 Mar 20 01:38 .
dr-xr-xr-x 4 root root 4 Mar 21 17:56 ..
-rw-r--r-- 1 nfsv4 nobody 0 Mar 20 01:04 it
-rw-r--r-- 1 nfsv4 nobody 11808 Mar 20 01:38 zfs.txt
This says something quite powerful, ZFS was released to the public in OpenSolaris not
working correctly with NFSv3. In turn, that shows the power of OpenSolaris, you get early
access to cool technology. But also, it shows that NFSv4 is the defacto standard for
development withing Sun. Think about it, NFSv4 is on by default - Solaris 10 clients
connecting to Solaris 10 servers will speak NFSv4 to each other.
Another neat thing is that with Solaris 10, automounting will be default occur over
NFSv4. The variables which control all of this are in /etc/default/nfs:
# Sets the minimum version of the NFS protocol that will be registered
# and offered by the server. The default is 2.
#NFS_SERVER_VERSMIN=2
# Sets the maximum version of the NFS protocol that will be registered
# and offered by the server. The default is 4.
#NFS_SERVER_VERSMAX=4
# Sets the minimum version of the NFS protocol that will be used by
# the NFS client. Can be overridden by the "vers=" NFS mount option.
# The default is 2.
#NFS_CLIENT_VERSMIN=2
# Sets the maximum version of the NFS protocol that will be used by
# the NFS client. Can be overridden by the "vers=" NFS mount option.
# If "vers=" is not specified for an NFS mount, this is the version
# that will be attempted first. The default is 4.
#NFS_CLIENT_VERSMAX=4
# Determines if the NFS version 4 delegation feature will be enabled
# for the server. If it is enabled, the server will attempt to
# provide delegations to the NFS version 4 client. The default is on.
#NFS_SERVER_DELEGATION=on
# Specifies to nfsmapid daemon that it is to override its default
# behavior of using the DNS domain, and that it is to use 'domain' as
# the domain to append to outbound attribute strings, and that it is to
# use 'domain' to compare against inbound attribute strings.
#NFSMAPID_DOMAIN=domain
When a client contacts a server, it says "I want access via NFSv4.". The server
can reply that version is not supported and then negotiation occurs. Looking
at the defaults, we see that a mount request will be via NFSv4. If a Solaris 9
client comes in, it doesn't know anything at all about NFSv4 and it will start
with NFSv3.
I can of course get NFSv3 via a command sequence like:
# sudo mount -o vers=3 wont:/export/zfs/nfsv4 /mnt
The Solaris team made a great design choice here, they extended the existing
option set to change the version. I.e., to explicitly get a NFSv4 mount:
# sudo mount -o vers=4 wont:/export/zfs/nfsv4 /mnt
Any existing systems which are upgraded to Solaris 10 will either:
- Use NFSv[2|3] if their automounts or /etc/vfstab already contain explicit 'vers=' option settings.
- Use NSFv4 by default.
A more subtle impact here is that the same automounter maps can be shared with earlier Solaris boxes, boxes which have their /etc/default/nfs defaults changed, other OSes. By not explictly mentioning the version, we keep flexibility.
In contrast the Linux designers chose to instead use a new filesystem type to represent NFSv4 - '-t nfs4'. If you look closely at my prior example, you would see I'm running Linux on adept:
[tdh@adept snapshot]> sudo mount -t nfs4 wont:/export/zfs/nfsv4
The Linux design was selected to make people aware that they were changing protocols. I don't like it, but I can understand why they made that choice.
The impact is that you do have to explicitly change your automount maps to get NFSv4 and you can't share them. I'm going to add this to the FAQ at NFSv4 in Action, because if you try to capture packet traces showing a NFSv4 problem and you inadvertantly go to /net, you can get confused.
Technorati Tags:
OpenSolaris Solaris NFSv4 ZFS
Recovering a ZFS filesystem after a reinstall So last night I reinstalled Nevada b35 on wont. Of course, I didn't bother nuking my ZFS filesystem. And of course it did not show up after the system came up. But that is the same as a UFS filesystem. For UFS you need to add it to /etc/vfstab to get it mounted automatically upon a reboot.
What do I need to do with a ZFS filesystem?
- Let's find it first:
# zpool import
pool: zoo
id: 6577446991347315550
state: ONLINE
action: The pool can be imported using its name or numeric identifier. The
pool may be active on on another system, but can be imported using
the '-f' flag.
config:
zoo ONLINE
mirror ONLINE
c0d1s0 ONLINE
c0d1s1 ONLINE
mirror ONLINE
c0d1s3 ONLINE
c0d1s4 ONLINE
- And now lets import it:
# zpool import zoo
cannot import 'zoo': pool may be in use from other system
use '-f' to import anyway
# zpool import -f zoo
cannot mount 'zoo/home/nfsv2': mountpoint or dataset is busy
- And lets verify that we can see it:
# df -h
Filesystem size used avail capacity Mounted on
/dev/dsk/c0d0s0 30G 4.3G 25G 15% /
/devices 0K 0K 0K 0% /devices
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 1.5G 716K 1.5G 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
/usr/lib/libc/libc_hwcap2.so.1
30G 4.3G 25G 15% /lib/libc.so.1
fd 0K 0K 0K 0% /dev/fd
swap 1.5G 108K 1.5G 1% /tmp
swap 1.5G 44K 1.5G 1% /var/run
/dev/dsk/c0d0s7 5.7G 5.8M 5.6G 1% /export/home
/vol/dev/dsk/c1t0d0/"kanigixx86"
2.8G 2.8G 0K 100% /cdrom/"kanigixx86"
zoo 134G 100K 128G 1% /zoo
zoo/x86 134G 2.8G 128G 3% /zoo/x86
zoo/home 10G 100K 10.0G 1% /export/zfs
zoo/home/nfsv2 10G 98K 10.0G 1% /export/zfs/nfsv2
zoo/home/tdh 10G 98K 10.0G 1% /export/zfs/tdh
zoo/home/nfsv3 10G 98K 10.0G 1% /export/zfs/nfsv3
zoo/home/nfsv4 10G 108K 10.0G 1% /export/zfs/nfsv4
zoo/isos 134G 2.8G 128G 3% /zoo/isos
No idea why 'zfs import -f zoo' spit out a warning. Well, it might have to do with
the fact that I didn't do a 'zfs export' before I did the reinstall.
And finally, lets make sure we can see the contents across the network on sandman:
# showmount -e wont
export list for wont:
/export/zfs (everyone)
/export/zfs/tdh (everyone)
/export/zfs/nfsv3 (everyone)
/export/zfs/nfsv4 (everyone)
/export/zfs/nfsv2 (everyone)
# cd /net/wont/export
# ls -la
total 3
dr-xr-xr-x 2 root root 2 Mar 21 15:36 .
dr-xr-xr-x 2 root root 2 Mar 21 15:36 ..
dr-xr-xr-x 1 root root 1 Mar 21 15:36 zfs
# cd zfs
# ls -la
total 7
drwxr-xr-x 6 root sys 6 Mar 20 23:12 .
dr-xr-xr-x 2 root root 2 Mar 21 15:36 ..
dr-xr-xr-x 3 root root 3 Mar 21 15:35 .zfs
dr-xr-xr-x 1 root root 1 Mar 21 15:36 nfsv2
dr-xr-xr-x 1 root root 1 Mar 21 15:36 nfsv3
dr-xr-xr-x 1 root root 1 Mar 21 15:36 nfsv4
dr-xr-xr-x 1 root root 1 Mar 21 15:36 tdh
# cd tdh
# ls -la
total 4
drwxr-xr-x 2 1066 staff 2 Mar 20 23:12 .
drwxr-xr-x 6 root sys 6 Mar 20 23:12 ..
dr-xr-xr-x 3 root root 3 Mar 21 15:36 .zfs
Obviously, the userid of 1066 has not been added to sandman. Notice that this is
a NFSv4 mount and the ID mapping must be working correctly. I.e., if it were not, we would
see nobody. A NFSv3 mount would show 1066 regardless of the ID domain settings.
Technorati Tags:
OpenSolaris Kanigix Open Source NFSv4 ZFS
Kanigix - Some more experiments I'm going to preface this with I should have been testing with my Sparc systems and not the x86. Can we say the default installation is hosed with respect to headless systems?
So I decided the next experiment was going to:
- Replace the grub splash screen with my own.
- Try and force the installation to automatically install the Software Companion packages.
I also decided to remove the Sparc packages from the DVD - eventually I am going to want that space for something else.
Well, I decided to not do my installation testing on wont. I want to do it on an old laptop I have, but I can't find its modular DVD reader and it won't boot from USB connected drives. I don't want another PC in the house, so I decided to use my w2100z.
It is hooked up via a serial console. This is a very important fact. And that is foreshadowing again - the sign of good journalism.
Pop in the new DVD, reboot and stare at the following for 5 minutes:
# sync; sync; sync; halt
Mar 21 00:49:07 warlock halt: initiated by root on /dev/console
syncing file systems... done
Press any key to reboot.
Resetting...
Add in the lack of a decent power reset switch on the w2100z, and the frustration mounts. I got it back up on Nevada b34 and then popped the DVD back into wont. And the reason
why I didn't get anything on the earlier console was really evident. The default grub selection is to boot to the attached monitor and not the console. If you are on the box, you can use the attached keyboard to get the selection to go to the console.
Headless boxes are pretty common with Solaris. I'll have to look to see how this is being handled in the labs.
I'm sure I could easily make headed and headless versions of the DVDs. By why waste a DVD?
Oh, by the way, the DVD boots on wont. I didn't see the splash screen, perhaps when the system reboots from the HD? Also, when the system prompted me for where to search for software to install, the DVD was greyed out. That might have nuked my second goal.
As an added bonus, I have some ZFS filesystems on the disk I didn't nuke. It will be interesting to see how they come up.
Ugh, I have my test system back - the modular DVD on the old laptop does not go in the modular bay - it is integrated. :-> I can swap the CD writer out for a floppy drive. Oh well, the first DVD version is installing on it.
Okay, I changed too many things at once on wont - I also added the extra software - which messed the initial boot up.
I'm not getting my splash screen.
The laptop is hung on:
Discovering additional network configuration...
It has a cardbus based network card.
So the good news is that I can repeat my orginal experiment to create a bootable x86 DVD. The bad news is that nothing new worked.
Strike that, I was able to modify the string name which appeared on the Gnome display of the DVD:
I did that with this entry in my ~/.mkisofsrc: Doh!, I nuked that area on wont :->
I really need to get that laptop bootable - even without the NIC it does the same thing.
I'll try to tie up all the loose ends sometime in the next day or so. Oh, I also can't see my ZFS filesystem, but I'll recover it as well.
Technorati Tags:
OpenSolaris Kanigix Open Source
Added links to www.alesti.org I added some links to allow the RSS feeds for both Kool Aid Served Daily and NFSv4 in Action to be added to user accounts of the Alesti RSS Reader.
They will look like this:
