Kool Aid Served Daily

How NFSv4 should work when crossing filesystems

In Some fun with NFSv4 and automount across a ssh tunnel, I revealed the work going on in Solaris for Mirror Mounts. The example was a desire to automount across a ssh tunnel. Well, I dusted off wont, the box from hell (being used by my son for video games) and created some zfs filesystems on it:

# zfs list
NAME               USED  AVAIL  REFER  MOUNTPOINT
zoo                398K   118G  24.5K  /zoo
zoo/home           256K   118G  35.5K  /export/zfs
zoo/home/braves   24.5K   118G  24.5K  /export/zfs/braves
zoo/home/kanigix  24.5K   118G  24.5K  /export/zfs/kanigix
zoo/home/loghyr   24.5K   118G  24.5K  /export/zfs/loghyr
zoo/home/mrx      24.5K   118G  24.5K  /export/zfs/mrx
zoo/home/nfsv2    24.5K   118G  24.5K  /export/zfs/nfsv2
zoo/home/nfsv3    24.5K   118G  24.5K  /export/zfs/nfsv3
zoo/home/nfsv4    24.5K   118G  24.5K  /export/zfs/nfsv4
zoo/home/spud     24.5K   118G  24.5K  /export/zfs/spud
zoo/home/tdh      24.5K   118G  24.5K  /export/zfs/tdh
# uname -a
SunOS wont 5.11 snv_55 i86pc i386 i86pc

I then opened a ssh tunnel to it on my Fedora Core 4 box and did a little bit of exploring:

[tdh@adept tdh]> uname -a
Linux adept 2.6.15-1.1833_FC4 #1 Wed Mar 1 23:41:37 EST 2006 i686 i686 i386 GNU/Linux
[tdh@adept ~/usenix]> ssh -fN -L "5049:wont:2049" wont
Password:
[tdh@adept ~/usenix]> sudo mount -o port=5049 -t nfs4 localhost:/ /nfs4/wont
[tdh@adept ~/usenix]> cd /nfs4/wont
[tdh@adept wont]> ls -la
total 6
drwxr-xr-x  38 root root 1024 Dec 31 17:49 .
drwxr-xr-x   4 root root 4096 Dec 31 18:17 ..
drwxr-xr-x   4 root sys   512 Dec 31 17:50 export
[tdh@adept wont]> cd export
[tdh@adept export]> ls -la
total 4
drwxr-xr-x   4 root sys   512 Dec 31 17:50 .
drwxr-xr-x  38 root root 1024 Dec 31 17:49 ..
drwxr-xr-x  11 root sys    11 Dec 31 17:50 zfs
[tdh@adept export]> cd zfs
[tdh@adept zfs]> ls -la
total 16
drwxr-xr-x  11 root sys  11 Dec 31 17:50 .
drwxr-xr-x   4 root sys 512 Dec 31 17:50 ..
drwxr-xr-x   2 root sys   2 Dec 31 17:50 braves
drwxr-xr-x   2 root sys   2 Dec 31 17:50 kanigix
drwxr-xr-x   2 root sys   2 Dec 31 17:50 loghyr
drwxr-xr-x   2 root sys   2 Dec 31 17:50 mrx
drwxr-xr-x   2 root sys   2 Dec 31 17:50 nfsv2
drwxr-xr-x   2 root sys   2 Dec 31 17:50 nfsv3
drwxr-xr-x   2 root sys   2 Dec 31 17:50 nfsv4
drwxr-xr-x   2 root sys   2 Dec 31 17:50 spud
drwxr-xr-x   2 root sys   2 Dec 31 17:50 tdh
[tdh@adept zfs]> cd tdh
[tdh@adept tdh]> ls -la
total 3
drwxr-xr-x   2 root sys  2 Dec 31 17:50 .
drwxr-xr-x  11 root sys 11 Dec 31 17:50 ..

Notice that I only did one mount command. As I crossed down into the exported filesystems, the Linux 2.16 implementation of NFSv4 did the mounts automatically for me in the background. Also, note that since '/' is not exported from wont, this must be a pseudo-fs:

[tdh@adept tdh]> showmount -e wont
Export list for wont:
/export/zfs         (everyone)
/export/zfs/nfsv2   (everyone)
/export/zfs/nfsv3   (everyone)
/export/zfs/nfsv4   (everyone)
/export/zfs/tdh     (everyone)
/export/zfs/loghyr  (everyone)
/export/zfs/kanigix (everyone)
/export/zfs/mrx     (everyone)
/export/zfs/spud    (everyone)
/export/zfs/braves  (everyone)

Let's export '/' and see what happens:

# share -F nfs -o rw -d "root" /

And on the Linux box:

[tdh@adept tdh]> cd /nfs4/wont
[tdh@adept wont]> ls -la
total 6
drwxr-xr-x  38 root root 1024 Dec 31 17:49 .
drwxr-xr-x   4 root root 4096 Dec 31 18:17 ..
drwxr-xr-x   4 root sys   512 Dec 31 17:50 export

What happened? Why didn't we see the root directory on wont? Well, when we did the mount command earlier, we basically got a reference to a file handle in the pseudo-fs. We need to flush this by umounting and remounting:

[tdh@adept wont]> cd
[tdh@adept ~]> sudo umount /nfs4/wont/
[tdh@adept ~]> sudo mount -o port=5049 -t nfs4 localhost:/ /nfs4/wont
[tdh@adept ~]> cd /nfs4/wont
[tdh@adept wont]> ls -la
total 67
drwxr-xr-x  38 root root 1024 Dec 31 17:49 .
drwxr-xr-x   4 root root 4096 Dec 31 18:17 ..
lrwxrwxrwx   1 root root    9 Dec 31 13:17 bin -> ./usr/bin
drwxr-xr-x   5 root sys   512 Dec 31 14:12 boot
drwxr-xr-x   2 root root  512 Dec 31 14:51 Desktop
drwxr-xr-x  24 root sys  4096 Dec 31 14:42 dev
drwxr-xr-x  10 root sys   512 Dec 31 14:42 devices
drwxr-xr-x   2 root root  512 Dec 31 14:51 Documents
drwxr-xr-x   9 root root  512 Dec 31 17:31 .dt
-rwxr-xr-x   1 root root 5111 Dec 31 14:51 .dtprofile
-rw-------   1 root root   16 Dec 31 17:31 .esd_auth
drwxr-xr-x  87 root sys  4608 Dec 31 17:52 etc
drwxr-xr-x   4 root sys   512 Dec 31 17:50 export
...

Let's walk down the paths again and see what happens:

[tdh@adept wont]> cd export [tdh@adept export]> ls -la total 5 drwxr-xr-x 4 root sys 512 Dec 31 17:50 . drwxr-xr-x 38 root root 1024 Dec 31 17:49 .. drwxr-xr-x 2 root root 512 Dec 31 13:17 home drwxr-xr-x 11 root sys 11 Dec 31 17:50 zfs [tdh@adept export]> cd zfs [tdh@adept zfs]> ls -la total 16 drwxr-xr-x 11 root sys 11 Dec 31 17:50 . drwxr-xr-x 4 root sys 512 Dec 31 17:50 .. drwxr-xr-x 2 root sys 2 Dec 31 17:50 braves drwxr-xr-x 2 root sys 2 Dec 31 17:50 kanigix drwxr-xr-x 2 root sys 2 Dec 31 17:50 loghyr drwxr-xr-x 2 root sys 2 Dec 31 17:50 mrx drwxr-xr-x 2 root sys 2 Dec 31 17:50 nfsv2 drwxr-xr-x 2 root sys 2 Dec 31 17:50 nfsv3 drwxr-xr-x 2 root sys 2 Dec 31 17:50 nfsv4 drwxr-xr-x 2 root sys 2 Dec 31 17:50 spud drwxr-xr-x 2 root sys 2 Dec 31 17:50 tdh [tdh@adept zfs]> cd tdh [tdh@adept tdh]> ls -la total 3 drwxr-xr-x 2 root sys 2 Dec 31 17:50 . drwxr-xr-x 11 root sys 11 Dec 31 17:50 ..

Let's make sure we are in the right place:

# scp sandman:/export/home/tdh/.tcshrc .
Password:
.tcshrc              100% |*******************************************************************|  5417       00:00
# chown tdh:staff .tcshrc
# ls -la
total 18
drwxr-xr-x   2 root     sys            3 Dec 31 18:10 .
drwxr-xr-x  11 root     sys           11 Dec 31 17:50 ..
-rw-------   1 tdh      staff       5417 Dec 31 18:10 .tcshrc

And on the client:

[tdh@adept tdh]> ls -la
total 9
drwxr-xr-x   2 root sys       3 Dec 31 18:10 .
drwxr-xr-x  11 root sys      11 Dec 31 17:50 ..
-rw-------   1 tdh  nobody 5417 Dec 31 18:10 .tcshrc
[tdh@adept tdh]> grep 10 /etc/group
wheel:x:10:root

The nobody shows up for the group because there is no mapping between the string "staff" and "wheel". In NFSv3, the numeric 10 would have gone across the wire and the ls command would have spit out "wheel".

Okay, let's check to see what the Solaris client would have done:

[tdh@sandman ~]> ssh -fN -L "5049:wont:2049" wont
Password:
[tdh@sandman ~]> su -
Password:
Sun Microsystems Inc.   SunOS 5.11      snv_54  October 2007
# mkdir -p /nfs4/wont
# mount -o port=5049 localhost:/ /nfs4/wont
# exit
[tdh@sandman ~]> cd /nfs4/wont
[tdh@sandman wont]> ls -la
total 134
drwxr-xr-x  38 root     root        1024 Dec 31 17:49 .
drwxr-xr-x   3 root     root         512 Dec 31 18:17 ..
...
drwxr-xr-x   2 root     root         512 Dec 31 14:51 Desktop
drwxr-xr-x   2 root     root         512 Dec 31 14:51 Documents
lrwxrwxrwx   1 root     root           9 Dec 31 13:17 bin -> ./usr/bin
drwxr-xr-x   5 root     sys          512 Dec 31 14:12 boot
drwxr-xr-x  24 root     sys         4096 Dec 31 14:42 dev
drwxr-xr-x  10 root     sys          512 Dec 31 14:42 devices
drwxr-xr-x  87 root     sys         4608 Dec 31 17:52 etc
drwxr-xr-x   4 root     sys          512 Dec 31 17:50 export
...
[tdh@sandman wont]> cd export
[tdh@sandman export]> ls -la
total 9
drwxr-xr-x   4 root     sys          512 Dec 31 17:50 .
drwxr-xr-x  38 root     root        1024 Dec 31 17:49 ..
drwxr-xr-x   2 root     root         512 Dec 31 13:17 home
drwxr-xr-x  11 root     sys           11 Dec 31 17:50 zfs
[tdh@sandman export]> cd zfs
[tdh@sandman zfs]> ls -la
total 5
drwxr-xr-x  11 root     sys           11 Dec 31 17:50 .
drwxr-xr-x   4 root     sys          512 Dec 31 17:50 ..

Okay, we have hit the crux of the problem for Mirror Mounts. We have a filesystem crossing on the server which needs to be mirrored on the client. We have to do this manually (or with an automounter if the ports are open):

[tdh@sandman zfs]> cd
[tdh@sandman ~]> su -
Password:
Sun Microsystems Inc.   SunOS 5.11      snv_54  October 2007
# mount -o port=5049 localhost:/export/zfs /nfs4/wont/export/zfs
# ls -la /nfs4/wont/export/zfs
total 32
drwxr-xr-x  11 root     sys           11 Dec 31 17:50 .
drwxr-xr-x   4 root     sys          512 Dec 31 17:50 ..
drwxr-xr-x   2 root     sys            2 Dec 31 17:50 braves
drwxr-xr-x   2 root     sys            2 Dec 31 17:50 kanigix
drwxr-xr-x   2 root     sys            2 Dec 31 17:50 loghyr
drwxr-xr-x   2 root     sys            2 Dec 31 17:50 mrx
drwxr-xr-x   2 root     sys            2 Dec 31 17:50 nfsv2
drwxr-xr-x   2 root     sys            2 Dec 31 17:50 nfsv3
drwxr-xr-x   2 root     sys            2 Dec 31 17:50 nfsv4
drwxr-xr-x   2 root     sys            2 Dec 31 17:50 spud
drwxr-xr-x   2 root     sys            3 Dec 31 18:10 tdh
# tcsh
# ls -la /nfs4/wont/export/zfs/tdh
total 6
drwxr-xr-x   2 root     sys            3 Dec 31 18:10 .
drwxr-xr-x  11 root     sys           11 Dec 31 17:50 ..
# mount -o port=5049 localhost:/export/zfs/tdh /nfs4/wont/export/zfs/tdh
# ls -la  /nfs4/wont/export/zfs/tdh
total 18
drwxr-xr-x   2 root     sys            3 Dec 31 18:10 .
drwxr-xr-x  11 root     sys           11 Dec 31 17:50 ..
-rw-------   1 tdh      staff       5417 Dec 31 18:10 .tcshrc

Notice how the '/export/zfs' gave information about the child filesystems whereas '/' did not. Also, note how we get the correct group name because the '/etc/group' is the same on the two Solaris hosts. Finally, even with zfs presenting up the child filesystems, we did have to manually mount the child in order to peer into it.

So the Mirror Mounts project in the NFSv4 development team is going to fix all of this. Under the hood, the client is going to understand it is about to traverse to a different filesystem and do the equivalent of a NFSv3 mount.

Technorati Tags: OpenSolaris NFSv4 ssh

---