Normally I don't summarize what I'm about to write about, however, I think this entry is all over the place. But there is useful information in here, So, I'm trying to get first kerberos and then NFSv4 working on a NSLU2 running OpenSlug. In order to validate my results, I also try to get a Linux NFSv4 server up and running on one of my Shuttle SS51G boxes. I finally get that to work, but I have no luck on getting the NSLU2 working correctly as either a server or client.

I decided to try another Linux client to see if I could get the process streamlined:

[tdh@sandman ~]> kadmin -p tdh/admin
Couldn't open log file /var/krb5/kdc.log: Permission denied
Authenticating as principal tdh/admin with password.
Password for tdh/admin@INTERNAL.EXCFB.COM:
kadmin:  addprinc -randkey nfs/mrbill.internal.excfb.com
WARNING: no policy specified for nfs/mrbill.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy
Principal "nfs/mrbill.internal.excfb.com@INTERNAL.EXCFB.COM" created.
kadmin:  addprinc -randkey host/mrbill.internal.excfb.com
WARNING: no policy specified for host/mrbill.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy
Principal "host/mrbill.internal.excfb.com@INTERNAL.EXCFB.COM" created.
kadmin:  ktadd -k /export/keytabs/mrbill.keytab -e des-cbc-crc:normal nfs/mrbill.internal.excfb.com
kadmin: No such file or directory while adding key to keytab

Okay, not only do I need to fix the above, I also need to fix not being able to add to /var/krb5/kdc.log. We can get the keytab generated with:

[tdh@sandman /export]> sudo chown tdh:staff keytabs/

And we see:

kadmin:  ktadd -k /export/keytabs/mrbill.keytab -e des-cbc-crc:normal nfs/mrbill.internal.excfb.com
Entry for principal nfs/mrbill.internal.excfb.com with kvno 4, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/export/keytabs/mrbill.keytab.
kadmin:  ktadd -k /export/keytabs/mrbill.keytab -e des-cbc-crc:normal host/mrbill.internal.excfb.com
Entry for principal host/mrbill.internal.excfb.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/export/keytabs/mrbill.keytab.

Okay, the first thing to note is that mrbill is running OpenSlug:

root@mrbill:~# uname -a
Linux mrbill 2.6.16 #1 PREEMPT Fri Jun 9 07:34:31 PDT 2006 armv5teb unknown unknown GNU/Linux

We try to get the keytab:

root@mrbill:~# mount sandman:/export/keytabs /mnt/sandman/keytabs
mount: can't get address for sandman
root@mrbill:~# host sandman
-sh: host: not found

Why? Well it turns out that:

root@mrbill:~# cat /etc/resolv.conf
search mshome
nameserver 192.168.2.108
nameserver 182.168.2.1

I thought that the domain entered in the turnup init was for the CIFS domain. Easy enough to fix...

root@mrbill:~# cat /etc/resolv.conf
search internal.excfb.com
nameserver 192.168.2.108
nameserver 182.168.2.1
root@mrbill:~#  mount sandman:/export/keytabs /mnt/sandman/keytabs
root@mrbill:~# cd /etc
root@mrbill:/etc# cp /mnt/sandman/keytabs/mrbill.keytab krb5.keytab
cp: cannot open `/mnt/sandman/keytabs/mrbill.keytab' for reading: Permission denied

What now? (Permissions)

root@mrbill:/etc# ls -la /mnt/sandman/keytabs
total 9
drwxr-xr-x  2 tdh  uucp  512 Feb 12  2007 .
drwxr-xr-x  5 root root 4096 Feb 12 08:22 ..
-rw-r--r--  1 root root 1968 Feb 12 06:50 krb5.conf
-rw-------  1 tdh  uucp  161 Feb 12  2007 mrbill.keytab
-rw-r--r--  1 root root  155 Feb 12 06:48 mrx.keytab

Fix them up on the server and:

root@mrbill:/etc# cp /mnt/sandman/keytabs/mrbill.keytab krb5.keytab

We need to get a good copy of krb5.conf, idmapd.conf, and sysconfig/nfs. For now, we will leave idmapd.conf alone, to illustrate the NFSv4 mapid issue.

root@mrbill:/etc# scp mrx:/etc/krb5.conf .
root@mrbill:/etc# scp mrx:/etc/sysconfig/nfs sysconfig

Now this time I know kerberos is not installed:

root@mrbill:/# ls -la ./usr/kerberos/bin/kinit
ls: ./usr/kerberos/bin/kinit: No such file or directory

And we can easily add it:

root@mrbill:/# ipkg list | grep krb5
kernel-module-rpcsec-gss-krb5 - 2.6.16-r6.6 - rpcsec-gss-krb5 kernel module
root@mrbill:/# ipkg install kernel-module-rpcsec-gss-krb5
Installing kernel-module-rpcsec-gss-krb5 (2.6.16-r6.6) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/slugos-bag/cross/3.10-beta/kernel-module-rpcsec-gss-krb5_2.6.16-r6.6_ixp4xxbe.ipk
Installing kernel-module-auth-rpcgss (2.6.16-r6.6) to root...
Downloading http://ipkg.nslu2-linux.org/feeds/slugos-bag/cross/3.10-beta/kernel-module-auth-rpcgss_2.6.16-r6.6_ixp4xxbe.ipk
Configuring kernel-module-auth-rpcgss
Configuring kernel-module-rpcsec-gss-krb5

Still not there for me:

root@mrbill:/# ls -la ./usr/kerberos/bin/kinit
ls: ./usr/kerberos/bin/kinit: No such file or directory
root@mrbill:/# find . -name kinit

My guess is that you can export with kerberos, you just can't mount it.

We should confirm that!

root@mrbill:~# mkdir /home/nfs4
root@mrbill:~# chmod 777 /home/nfs4
root@mrbill:~# cd /home/nfs4
root@mrbill:/home/nfs4# touch see_me
root@mrbill:/home/nfs4# chown tdh:10 see_me
root@mrbill:/home/nfs4# ls -la
total 8
drwxrwxrwx  2 root root 4096 Feb 12 09:00 .
drwxrwxr-x  8 root root 4096 Feb 12 09:00 ..
-rw-r--r--  1 tdh  uucp    0 Feb 12 09:00 see_me

And I try to add the export:

root@mrbill:/home/nfs4# more /etc/exports
/home/NFS4 172.16.0.0/16(rw,fsid=0,insecure,no_subtree_check,sync,anonuid=65534,anongid=65534)
root@mrbill:/home/nfs4# cd ..
root@mrbill:/home# ls -la
total 32
drwxrwxr-x   8 root root  4096 Feb 12 09:00 .
drwxr-xr-x  18 root root  4096 Feb  5 22:44 ..
drwxrwxrwx   2 tdh  uucp  4096 Feb  5 23:03 NFS4
drwxrwxrwx   2 root root  4096 Feb 12 09:00 nfs4
drwxr-xr-x   2 root root  4096 Feb  5 22:53 nfsv2
drwxr-xr-x   2 root root  4096 Feb  5 22:53 nfsv3
drwxr-xr-x   2 root root  4096 Feb  5 22:53 nfsv4
lrwxrwxrwx   1 root root     7 Feb  5 22:26 root -> ../root
drwxr-xr-x   2 tdh  staff 4096 Feb  7 21:21 tdh
root@mrbill:/home#

Looks like /home/NFS4 was created for me, or I'm suffering from severe memory loss...

I could have done this last week, note the time stamp.

root@mrbill:/home# ls -la NFS4
total 8
drwxrwxrwx  2 tdh    uucp 4096 Feb  5 23:03 .
drwxrwxr-x  8 root   root 4096 Feb 12 09:00 ..
-rw-r--r--  1 200096 uucp    0 Feb  5 23:03 ut

Must be memory loss!

root@mrbill:/home# cd NFS4/
root@mrbill:/home/NFS4# touch see_me
root@mrbill:/home/NFS4# chown tdh:10 see_me
root@mrbill:/home/NFS4# ls -la
total 8
drwxrwxrwx  2 tdh    uucp 4096 Feb 12 09:03 .
drwxrwxr-x  8 root   root 4096 Feb 12 09:00 ..
-rw-r--r--  1 tdh    uucp    0 Feb 12 09:03 see_me
-rw-r--r--  1 200096 uucp    0 Feb  5 23:03 ut

And yes:

[tdh@mrx ipk]> showmount -e mrbill
Export list for mrbill:
/home/NFS4 172.16.0.0/16

I was in 172.16.0.0/16 space last week. Touch up the export and:

[tdh@mrx ipk]> showmount -e mrbill
Export list for mrbill:
/home/NFS4 192.168.2.0/24

Okay, I do the mount and I'll claim it gets done as nfsv3:

[tdh@mrx ipk]> sudo mount mrbill:/home/NFS4 /mnt/mrbill/NFS4
[tdh@mrx ipk]> ls -la /mnt/mrbill/NFS4
total 8
drwxrwxrwx 2 tdh    wheel 4096 Feb 12 03:03 .
drwxr-xr-x 3 root   root  4096 Feb 12 11:08 ..
-rw-r--r-- 1 tdh    wheel    0 Feb 12 03:03 see_me
-rw-r--r-- 1 200096 wheel    0 Feb  5 17:03 ut

Why do I claim it is nfsv3? Because I suspect that the idmapping should be hosed. Can we verify this? Yes:

[tdh@mrx ipk]> sudo umount /mnt/mrbill/NFS4
[tdh@mrx ipk]> sudo mount -o vers=3 mrbill:/home/NFS4 /mnt/mrbill/NFS4
[tdh@mrx ipk]> ls -la /mnt/mrbill/NFS4
total 8
drwxrwxrwx 2 tdh    wheel 4096 Feb 12 03:03 .
drwxr-xr-x 3 root   root  4096 Feb 12 11:08 ..
-rw-r--r-- 1 tdh    wheel    0 Feb 12 03:03 see_me
-rw-r--r-- 1 200096 wheel    0 Feb  5 17:03 ut
[tdh@mrx ipk]> sudo umount /mnt/mrbill/NFS4
[tdh@mrx ipk]> sudo mount -o vers=4 mrbill:/home/NFS4 /mnt/mrbill/NFS4
'vers=4' is not supported.  Use '-t nfs4' instead.
[tdh@mrx ipk]> sudo mount -t nfs4 mrbill:/home/NFS4 /mnt/mrbill/NFS4
mount.nfs4: mount point /mnt/mrbill/NFS4 does not exist

Okay, mrbill knows nothing about NFSv4 as far as I can tell:

root@mrbill:/home/NFS4# mount -t nfs4 sandman:/export/home /mnt/sandman/home
mount: unknown filesystem type 'nfs4'

I'm sensing protocol discrimination here:

root@mrbill:/home/NFS4# ipkg list | grep -i nfs
kernel-module-lockd - 2.6.16-r6.6 - lockd kernel module; NFS file locking service version 0.5.
kernel-module-nfs - 2.6.16-r6.6 - nfs kernel module
kernel-module-nfs - 2.6.16-r6.4 -
kernel-module-nfsd - 2.6.16-r6.6 - nfsd kernel module
nfs-utils - 1.0.6-r7 - userspace utilities for kernel nfs
nfs-utils-doc - 1.0.6-r7 - userspace utilities for kernel nfs

Time to check the log file:

Feb 12 09:08:29 (none) user.warn kernel: nfsd: nfsv4 idmapping failing: has idmapd not been started?

Okay, configure idmapping and reboot:

Feb 12 09:16:37 (none) user.info kernel: Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Feb 12 09:16:37 (none) user.warn kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
Feb 12 09:16:37 (none) user.warn kernel: NFSD: unable to find recovery directory /var/lib/nfs/v4recovery
Feb 12 09:16:37 (none) user.warn kernel: NFSD: starting 90-second grace period

Try the mount again:

[tdh@mrx ipk]> sudo mount -t nfs4 mrbill:/home/NFS4 /mnt/mrbill/NFS4
mount.nfs4: Permission denied

And try it from a Solaris client:

[tdh@sandman keytabs]> sudo mount mrbill:/home/NFS4 /mnt/mrbill/NFS4
[tdh@sandman keytabs]> sudo mount mrbill:/home/NFS4 /mnt/mrbill/NFS4
NFS compound failed for server mrbill: error 7 (RPC: Authentication error)
NFS compound failed for server mrbill: error 7 (RPC: Authentication error)
NFS compound failed for server mrbill: error 7 (RPC: Authentication error)
NFS compound failed for server mrbill: error 7 (RPC: Authentication error)
NFS compound failed for server mrbill: error 7 (RPC: Authentication error)
NFS compound failed for server mrbill: error 7 (RPC: Authentication error)
nfs mount: mount: /mnt/mrbill/NFS4: Permission denied

Okay, can we get Kerberos working at all on the NSLU2?

root@mrbill:~# more /etc/exports
/home/NFS4 192.168.2.0/24(rw,fsid=0,sec=krb5,insecure,no_subtree_check,sync,anonuid=65534,anongid=65534)
root@mrbill:~# exportfs -rv
exportfs: /etc/exports:1: unknown keyword "sec=krb5"
unexporting sandman.internal.excfb.com:/home/NFS4 from kernel

The keyword is not correct? Time to try on a known good linux config:

[tdh@mrx ipk]> cat /etc/exports
/home/tdh 192.168.2.0/24(rw,fsid=0,sec=krb5,insecure,no_subtree_check,sync,anonuid=65534,anongid=65534)
[tdh@mrx ipk]> sudo exportfs -rv
exportfs: /etc/exports:1: unknown keyword "sec=krb5"

Okay, here is what we are supposed to do:

[tdh@mrx ipk]> cat /etc/exports
/home/tdh gss/krb5(rw,fsid=0,insecure,no_subtree_check,sync,anonuid=65534,anongid=65534)
[tdh@mrx ipk]> sudo exportfs -rv
exporting gss/krb5:/home/tdh
exporting gss/krb5:/home/tdh to kernel
gss/krb5:/home/tdh: Cannot allocate memory

By sheer effort of will, I determined that the firewall was on.

root@mrbill:~# showmount -e mrx
Export list for mrx:
/home/tdh gss/krb5

First lets see what happens without kerberos:

[tdh@sandman ~]> sudo mount -o vers=3 mrx:/home/tdh /mnt/mrx/tdh
[tdh@sandman ~]> ls -la /mnt/mrx/tdh
total 230394
drwxr-xr-x   7 tdh      staff       4096 Feb 12 02:01 .
drwxr-xr-x   3 root     root         512 Feb 12 11:49 ..

And NFSv4:

[tdh@sandman ~]> sudo mount mrx:/home/tdh /mnt/mrx/tdh
nfs mount: mrx:/home/tdh: No such file or directory

Okay, I knew about this, but forgot it. I think I heard Bruce complaining about still having it:

[tdh@sandman ~]> sudo mount mrx:/ /mnt/mrx/tdh
[tdh@sandman ~]> ls -al /mnt/mrx/tdh
total 230394
drwxr-xr-x   7 tdh      nobody      4096 Feb 12 02:01 .
drwxr-xr-x   3 root     root         512 Feb 12 11:49 ..
-rw-------   1 tdh      nobody        68 Feb 12 01:51 .Xauthority
-rw-------   1 tdh      nobody        96 Feb 12 11:31 .lesshst

And now we turn on kerberos:

[tdh@sandman ~]> sudo mount mrx:/ /mnt/mrx/tdh
NFS compound failed for server mrx: error 7 (RPC: Authentication error)
NFS compound failed for server mrx: error 7 (RPC: Authentication error)
NFS compound failed for server mrx: error 7 (RPC: Authentication error)
nfs mount: mount: /mnt/mrx/tdh: Permission denied

We can be very specific about what security flavor we want to use:

[tdh@sandman ~]> sudo mount -o sec=krb5 mrx:/ /mnt/mrx/tdh
nfs mount: mount: /mnt/mrx/tdh: Permission denied

Note that the compound fails messages must have been about AUTH_NONE, AUTH_SYS, and AUTH_DH.

I think I've found the answer in Mike Eisler's blog Real Authentication in NFS, scroll down into the comments:

> Also, does NetApp require a root principle like Solaris did prior to 10?

Actually even prior to Solaris 10, the Solaris NFS server would allow
an NFSv3 mount if root didn't have Kerberos credentials. ONTAP is the
same way. However, if using NFSv4, because NFSv4 has no separate mount
protocol, an NFSv4 server cannot distinguish a mount from a LOOKUP. If
a volume is exported with sec=krb5, then the NFSv4 requests need to be
using Kerberos. Since UNIX clients usually require one to be superuser
to do an NFS mount, superuser (root) needs to have credentials. Root
credentials aren't required, but whatever uid the credentials map to
has to have search permissions for the path name.

And we can try that here:

kadmin:  addprinc root
WARNING: no policy specified for root@INTERNAL.EXCFB.COM; defaulting to no policy
Enter password for principal "root@INTERNAL.EXCFB.COM":
Re-enter password for principal "root@INTERNAL.EXCFB.COM":
Principal "root@INTERNAL.EXCFB.COM" created.

And then we grab a ticket:

[tdh@sandman ~]> sudo kinit root
Password for root@INTERNAL.EXCFB.COM:
[tdh@sandman ~]> sudo mount -o sec=krb5 mrx:/ /mnt/mrx/tdh

Aargh!

[tdh@sandman ~]> ls -la /mnt/mrx/tdh
total 230394
drwxr-xr-x   7 tdh      nobody      4096 Feb 12 02:01 .
drwxr-xr-x   3 root     root         512 Feb 12 11:49 ..
-rw-------   1 tdh      nobody        68 Feb 12 01:51 .Xauthority
-rw-------   1 tdh      nobody        96 Feb 12 11:31 .lesshst

Since we can't even get the export shared without kerberos on mrbill, that does not explain the issue on that machine.

This works:

[tdh@sandman ~]> sudo mount -o vers=3 mrbill:/home/NFS4 /mnt/mrbill/NFS4

And this does not:

[tdh@sandman ~]> sudo mount -o vers=4 mrbill:/ /mnt/mrbill/NFS4
nfs mount: mount: /mnt/mrbill/NFS4: Resource temporarily unavailable

I'll come back to this later...