Neat blogs
Navigation
Editing
Powered by Roller Weblogger.
statcounter.com
clustrmaps.com
technorati.com
Monday February 12, 2007 Trying to get a Kerberized NFSv4 server/client on a NSLU2
Normally I don't summarize what I'm about to write about, however, I think this entry is all over the place. But there is useful information in here, So, I'm trying to get first kerberos and then NFSv4 working on a NSLU2 running OpenSlug. In order to validate my results, I also try to get a Linux NFSv4 server up and running on one of my Shuttle SS51G boxes. I finally get that to work, but I have no luck on getting the NSLU2 working correctly as either a server or client.
I decided to try another Linux client to see if I could get the process streamlined:
[tdh@sandman ~]> kadmin -p tdh/admin Couldn't open log file /var/krb5/kdc.log: Permission denied Authenticating as principal tdh/admin with password. Password for tdh/admin@INTERNAL.EXCFB.COM: kadmin: addprinc -randkey nfs/mrbill.internal.excfb.com WARNING: no policy specified for nfs/mrbill.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy Principal "nfs/mrbill.internal.excfb.com@INTERNAL.EXCFB.COM" created. kadmin: addprinc -randkey host/mrbill.internal.excfb.com WARNING: no policy specified for host/mrbill.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy Principal "host/mrbill.internal.excfb.com@INTERNAL.EXCFB.COM" created. kadmin: ktadd -k /export/keytabs/mrbill.keytab -e des-cbc-crc:normal nfs/mrbill.internal.excfb.com kadmin: No such file or directory while adding key to keytab
Okay, not only do I need to fix the above, I also need to fix not being able to add to /var/krb5/kdc.log. We can get the keytab generated with:
[tdh@sandman /export]> sudo chown tdh:staff keytabs/
And we see:
kadmin: ktadd -k /export/keytabs/mrbill.keytab -e des-cbc-crc:normal nfs/mrbill.internal.excfb.com Entry for principal nfs/mrbill.internal.excfb.com with kvno 4, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/export/keytabs/mrbill.keytab. kadmin: ktadd -k /export/keytabs/mrbill.keytab -e des-cbc-crc:normal host/mrbill.internal.excfb.com Entry for principal host/mrbill.internal.excfb.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/export/keytabs/mrbill.keytab.
Okay, the first thing to note is that mrbill is running OpenSlug:
root@mrbill:~# uname -a Linux mrbill 2.6.16 #1 PREEMPT Fri Jun 9 07:34:31 PDT 2006 armv5teb unknown unknown GNU/Linux
We try to get the keytab:
root@mrbill:~# mount sandman:/export/keytabs /mnt/sandman/keytabs mount: can't get address for sandman root@mrbill:~# host sandman -sh: host: not found
Why? Well it turns out that:
root@mrbill:~# cat /etc/resolv.conf search mshome nameserver 192.168.2.108 nameserver 182.168.2.1
I thought that the domain entered in the turnup init was for the CIFS domain. Easy enough to fix...
root@mrbill:~# cat /etc/resolv.conf search internal.excfb.com nameserver 192.168.2.108 nameserver 182.168.2.1 root@mrbill:~# mount sandman:/export/keytabs /mnt/sandman/keytabs root@mrbill:~# cd /etc root@mrbill:/etc# cp /mnt/sandman/keytabs/mrbill.keytab krb5.keytab cp: cannot open `/mnt/sandman/keytabs/mrbill.keytab' for reading: Permission denied
What now? (Permissions)
root@mrbill:/etc# ls -la /mnt/sandman/keytabs total 9 drwxr-xr-x 2 tdh uucp 512 Feb 12 2007 . drwxr-xr-x 5 root root 4096 Feb 12 08:22 .. -rw-r--r-- 1 root root 1968 Feb 12 06:50 krb5.conf -rw------- 1 tdh uucp 161 Feb 12 2007 mrbill.keytab -rw-r--r-- 1 root root 155 Feb 12 06:48 mrx.keytab
Fix them up on the server and:
root@mrbill:/etc# cp /mnt/sandman/keytabs/mrbill.keytab krb5.keytab
We need to get a good copy of krb5.conf, idmapd.conf, and sysconfig/nfs. For now, we will leave idmapd.conf alone, to illustrate the NFSv4 mapid issue.
root@mrbill:/etc# scp mrx:/etc/krb5.conf . root@mrbill:/etc# scp mrx:/etc/sysconfig/nfs sysconfig
Now this time I know kerberos is not installed:
root@mrbill:/# ls -la ./usr/kerberos/bin/kinit ls: ./usr/kerberos/bin/kinit: No such file or directory
And we can easily add it:
root@mrbill:/# ipkg list | grep krb5 kernel-module-rpcsec-gss-krb5 - 2.6.16-r6.6 - rpcsec-gss-krb5 kernel module root@mrbill:/# ipkg install kernel-module-rpcsec-gss-krb5 Installing kernel-module-rpcsec-gss-krb5 (2.6.16-r6.6) to root... Downloading http://ipkg.nslu2-linux.org/feeds/slugos-bag/cross/3.10-beta/kernel-module-rpcsec-gss-krb5_2.6.16-r6.6_ixp4xxbe.ipk Installing kernel-module-auth-rpcgss (2.6.16-r6.6) to root... Downloading http://ipkg.nslu2-linux.org/feeds/slugos-bag/cross/3.10-beta/kernel-module-auth-rpcgss_2.6.16-r6.6_ixp4xxbe.ipk Configuring kernel-module-auth-rpcgss Configuring kernel-module-rpcsec-gss-krb5
Still not there for me:
root@mrbill:/# ls -la ./usr/kerberos/bin/kinit ls: ./usr/kerberos/bin/kinit: No such file or directory root@mrbill:/# find . -name kinit
My guess is that you can export with kerberos, you just can't mount it.
We should confirm that!
root@mrbill:~# mkdir /home/nfs4 root@mrbill:~# chmod 777 /home/nfs4 root@mrbill:~# cd /home/nfs4 root@mrbill:/home/nfs4# touch see_me root@mrbill:/home/nfs4# chown tdh:10 see_me root@mrbill:/home/nfs4# ls -la total 8 drwxrwxrwx 2 root root 4096 Feb 12 09:00 . drwxrwxr-x 8 root root 4096 Feb 12 09:00 .. -rw-r--r-- 1 tdh uucp 0 Feb 12 09:00 see_me
And I try to add the export:
root@mrbill:/home/nfs4# more /etc/exports /home/NFS4 172.16.0.0/16(rw,fsid=0,insecure,no_subtree_check,sync,anonuid=65534,anongid=65534) root@mrbill:/home/nfs4# cd .. root@mrbill:/home# ls -la total 32 drwxrwxr-x 8 root root 4096 Feb 12 09:00 . drwxr-xr-x 18 root root 4096 Feb 5 22:44 .. drwxrwxrwx 2 tdh uucp 4096 Feb 5 23:03 NFS4 drwxrwxrwx 2 root root 4096 Feb 12 09:00 nfs4 drwxr-xr-x 2 root root 4096 Feb 5 22:53 nfsv2 drwxr-xr-x 2 root root 4096 Feb 5 22:53 nfsv3 drwxr-xr-x 2 root root 4096 Feb 5 22:53 nfsv4 lrwxrwxrwx 1 root root 7 Feb 5 22:26 root -> ../root drwxr-xr-x 2 tdh staff 4096 Feb 7 21:21 tdh root@mrbill:/home#
Looks like /home/NFS4 was created for me, or I'm suffering from severe memory loss...
I could have done this last week, note the time stamp.
root@mrbill:/home# ls -la NFS4 total 8 drwxrwxrwx 2 tdh uucp 4096 Feb 5 23:03 . drwxrwxr-x 8 root root 4096 Feb 12 09:00 .. -rw-r--r-- 1 200096 uucp 0 Feb 5 23:03 ut
Must be memory loss!
root@mrbill:/home# cd NFS4/ root@mrbill:/home/NFS4# touch see_me root@mrbill:/home/NFS4# chown tdh:10 see_me root@mrbill:/home/NFS4# ls -la total 8 drwxrwxrwx 2 tdh uucp 4096 Feb 12 09:03 . drwxrwxr-x 8 root root 4096 Feb 12 09:00 .. -rw-r--r-- 1 tdh uucp 0 Feb 12 09:03 see_me -rw-r--r-- 1 200096 uucp 0 Feb 5 23:03 ut
And yes:
[tdh@mrx ipk]> showmount -e mrbill Export list for mrbill: /home/NFS4 172.16.0.0/16
I was in 172.16.0.0/16 space last week. Touch up the export and:
[tdh@mrx ipk]> showmount -e mrbill Export list for mrbill: /home/NFS4 192.168.2.0/24
Okay, I do the mount and I'll claim it gets done as nfsv3:
[tdh@mrx ipk]> sudo mount mrbill:/home/NFS4 /mnt/mrbill/NFS4 [tdh@mrx ipk]> ls -la /mnt/mrbill/NFS4 total 8 drwxrwxrwx 2 tdh wheel 4096 Feb 12 03:03 . drwxr-xr-x 3 root root 4096 Feb 12 11:08 .. -rw-r--r-- 1 tdh wheel 0 Feb 12 03:03 see_me -rw-r--r-- 1 200096 wheel 0 Feb 5 17:03 ut
Why do I claim it is nfsv3? Because I suspect that the idmapping should be hosed. Can we verify this? Yes:
[tdh@mrx ipk]> sudo umount /mnt/mrbill/NFS4 [tdh@mrx ipk]> sudo mount -o vers=3 mrbill:/home/NFS4 /mnt/mrbill/NFS4 [tdh@mrx ipk]> ls -la /mnt/mrbill/NFS4 total 8 drwxrwxrwx 2 tdh wheel 4096 Feb 12 03:03 . drwxr-xr-x 3 root root 4096 Feb 12 11:08 .. -rw-r--r-- 1 tdh wheel 0 Feb 12 03:03 see_me -rw-r--r-- 1 200096 wheel 0 Feb 5 17:03 ut [tdh@mrx ipk]> sudo umount /mnt/mrbill/NFS4 [tdh@mrx ipk]> sudo mount -o vers=4 mrbill:/home/NFS4 /mnt/mrbill/NFS4 'vers=4' is not supported. Use '-t nfs4' instead. [tdh@mrx ipk]> sudo mount -t nfs4 mrbill:/home/NFS4 /mnt/mrbill/NFS4 mount.nfs4: mount point /mnt/mrbill/NFS4 does not exist
Okay, mrbill knows nothing about NFSv4 as far as I can tell:
root@mrbill:/home/NFS4# mount -t nfs4 sandman:/export/home /mnt/sandman/home mount: unknown filesystem type 'nfs4'
I'm sensing protocol discrimination here:
root@mrbill:/home/NFS4# ipkg list | grep -i nfs kernel-module-lockd - 2.6.16-r6.6 - lockd kernel module; NFS file locking service version 0.5. kernel-module-nfs - 2.6.16-r6.6 - nfs kernel module kernel-module-nfs - 2.6.16-r6.4 - kernel-module-nfsd - 2.6.16-r6.6 - nfsd kernel module nfs-utils - 1.0.6-r7 - userspace utilities for kernel nfs nfs-utils-doc - 1.0.6-r7 - userspace utilities for kernel nfs
Time to check the log file:
Feb 12 09:08:29 (none) user.warn kernel: nfsd: nfsv4 idmapping failing: has idmapd not been started?
Okay, configure idmapping and reboot:
Feb 12 09:16:37 (none) user.info kernel: Installing knfsd (copyright (C) 1996 okir@monad.swb.de). Feb 12 09:16:37 (none) user.warn kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory Feb 12 09:16:37 (none) user.warn kernel: NFSD: unable to find recovery directory /var/lib/nfs/v4recovery Feb 12 09:16:37 (none) user.warn kernel: NFSD: starting 90-second grace period
Try the mount again:
[tdh@mrx ipk]> sudo mount -t nfs4 mrbill:/home/NFS4 /mnt/mrbill/NFS4 mount.nfs4: Permission denied
And try it from a Solaris client:
[tdh@sandman keytabs]> sudo mount mrbill:/home/NFS4 /mnt/mrbill/NFS4 [tdh@sandman keytabs]> sudo mount mrbill:/home/NFS4 /mnt/mrbill/NFS4 NFS compound failed for server mrbill: error 7 (RPC: Authentication error) NFS compound failed for server mrbill: error 7 (RPC: Authentication error) NFS compound failed for server mrbill: error 7 (RPC: Authentication error) NFS compound failed for server mrbill: error 7 (RPC: Authentication error) NFS compound failed for server mrbill: error 7 (RPC: Authentication error) NFS compound failed for server mrbill: error 7 (RPC: Authentication error) nfs mount: mount: /mnt/mrbill/NFS4: Permission denied
Okay, can we get Kerberos working at all on the NSLU2?
root@mrbill:~# more /etc/exports /home/NFS4 192.168.2.0/24(rw,fsid=0,sec=krb5,insecure,no_subtree_check,sync,anonuid=65534,anongid=65534) root@mrbill:~# exportfs -rv exportfs: /etc/exports:1: unknown keyword "sec=krb5" unexporting sandman.internal.excfb.com:/home/NFS4 from kernel
The keyword is not correct? Time to try on a known good linux config:
[tdh@mrx ipk]> cat /etc/exports /home/tdh 192.168.2.0/24(rw,fsid=0,sec=krb5,insecure,no_subtree_check,sync,anonuid=65534,anongid=65534) [tdh@mrx ipk]> sudo exportfs -rv exportfs: /etc/exports:1: unknown keyword "sec=krb5"
Okay, here is what we are supposed to do:
[tdh@mrx ipk]> cat /etc/exports /home/tdh gss/krb5(rw,fsid=0,insecure,no_subtree_check,sync,anonuid=65534,anongid=65534) [tdh@mrx ipk]> sudo exportfs -rv exporting gss/krb5:/home/tdh exporting gss/krb5:/home/tdh to kernel gss/krb5:/home/tdh: Cannot allocate memory
By sheer effort of will, I determined that the firewall was on.
root@mrbill:~# showmount -e mrx Export list for mrx: /home/tdh gss/krb5
First lets see what happens without kerberos:
[tdh@sandman ~]> sudo mount -o vers=3 mrx:/home/tdh /mnt/mrx/tdh [tdh@sandman ~]> ls -la /mnt/mrx/tdh total 230394 drwxr-xr-x 7 tdh staff 4096 Feb 12 02:01 . drwxr-xr-x 3 root root 512 Feb 12 11:49 ..
And NFSv4:
[tdh@sandman ~]> sudo mount mrx:/home/tdh /mnt/mrx/tdh nfs mount: mrx:/home/tdh: No such file or directory
Okay, I knew about this, but forgot it. I think I heard Bruce complaining about still having it:
[tdh@sandman ~]> sudo mount mrx:/ /mnt/mrx/tdh [tdh@sandman ~]> ls -al /mnt/mrx/tdh total 230394 drwxr-xr-x 7 tdh nobody 4096 Feb 12 02:01 . drwxr-xr-x 3 root root 512 Feb 12 11:49 .. -rw------- 1 tdh nobody 68 Feb 12 01:51 .Xauthority -rw------- 1 tdh nobody 96 Feb 12 11:31 .lesshst
And now we turn on kerberos:
[tdh@sandman ~]> sudo mount mrx:/ /mnt/mrx/tdh NFS compound failed for server mrx: error 7 (RPC: Authentication error) NFS compound failed for server mrx: error 7 (RPC: Authentication error) NFS compound failed for server mrx: error 7 (RPC: Authentication error) nfs mount: mount: /mnt/mrx/tdh: Permission denied
We can be very specific about what security flavor we want to use:
[tdh@sandman ~]> sudo mount -o sec=krb5 mrx:/ /mnt/mrx/tdh nfs mount: mount: /mnt/mrx/tdh: Permission denied
Note that the compound fails messages must have been about AUTH_NONE, AUTH_SYS, and AUTH_DH.
I think I've found the answer in Mike Eisler's blog Real Authentication in NFS, scroll down into the comments:
> Also, does NetApp require a root principle like Solaris did prior to 10? Actually even prior to Solaris 10, the Solaris NFS server would allow an NFSv3 mount if root didn't have Kerberos credentials. ONTAP is the same way. However, if using NFSv4, because NFSv4 has no separate mount protocol, an NFSv4 server cannot distinguish a mount from a LOOKUP. If a volume is exported with sec=krb5, then the NFSv4 requests need to be using Kerberos. Since UNIX clients usually require one to be superuser to do an NFS mount, superuser (root) needs to have credentials. Root credentials aren't required, but whatever uid the credentials map to has to have search permissions for the path name.
And we can try that here:
kadmin: addprinc root WARNING: no policy specified for root@INTERNAL.EXCFB.COM; defaulting to no policy Enter password for principal "root@INTERNAL.EXCFB.COM": Re-enter password for principal "root@INTERNAL.EXCFB.COM": Principal "root@INTERNAL.EXCFB.COM" created.
And then we grab a ticket:
[tdh@sandman ~]> sudo kinit root Password for root@INTERNAL.EXCFB.COM: [tdh@sandman ~]> sudo mount -o sec=krb5 mrx:/ /mnt/mrx/tdh
Aargh!
[tdh@sandman ~]> ls -la /mnt/mrx/tdh total 230394 drwxr-xr-x 7 tdh nobody 4096 Feb 12 02:01 . drwxr-xr-x 3 root root 512 Feb 12 11:49 .. -rw------- 1 tdh nobody 68 Feb 12 01:51 .Xauthority -rw------- 1 tdh nobody 96 Feb 12 11:31 .lesshst
Since we can't even get the export shared without kerberos on mrbill, that does not explain the issue on that machine.
This works:
[tdh@sandman ~]> sudo mount -o vers=3 mrbill:/home/NFS4 /mnt/mrbill/NFS4
And this does not:
[tdh@sandman ~]> sudo mount -o vers=4 mrbill:/ /mnt/mrbill/NFS4 nfs mount: mount: /mnt/mrbill/NFS4: Resource temporarily unavailable
I'll come back to this later...
Installing a Kerberos KDC and setting up NFS mounts
We always seem to have problems at Connectathon setting up Kerberos. So I decided to take the cookbook we use there and get kerberos working on my home systems. Please note that I could easily clean up the notes to not show some errors I make. But then, where is the love?
Also, as with any first foray into a new tool, I have no clue what I am doing. I kinda understand tickets and the ideas behind Kerberos, but I'm really in the dark as to what I'm supposed to do.
First edit /etc/krb5/krb5.conf:
# diff krb5.conf stock/krb5.conf
35c35
< default_realm = INTERNAL.EXCFB.COM
---
> default_realm = ___default_realm___
38,41c38,43
< INTERNAL.EXCFB.COM = {
< kdc = sandman.internal.excfb.com
< kdc = ultralord.internal.excfb.com
< admin_server = sandman.internal.excfb.com
---
> ___default_realm___ = {
> kdc = ___master_kdc___
> kdc = ___slave_kdc1___
> kdc = ___slave_kdc2___
> kdc = ___slave_kdcN___
> admin_server = ___master_kdc___
Then edit /etc/krb5/kdc.conf:
# diff kdc.conf stock/kdc.conf
32c32
< INTERNAL.EXCFB.COM = {
---
> ___default_realm___ = {
41,42d40
< sunw_dbprob_enable = true
< sunw_dbprop_master_ulogsize = 1000
Make sure you can get at the kdcs via DNS (or whatever name service in /etc/resolv.conf)
# host sandman sandman.internal.excfb.com has address 192.168.2.109 # host sandman.internal.excfb.com sandman.internal.excfb.com has address 192.168.2.109
Create the kerberos database
# /usr/sbin/kdb5_util create -r INTERNAL.EXCFB.COM -s Initializing database '/var/krb5/principal' for realm 'INTERNAL.EXCFB.COM', master key name 'K/M@INTERNAL.EXCFB.COM' You will be prompted for the database Master Password. It is important that you NOT FORGET this password. Enter KDC database master key: Re-enter KDC database master key to verify:
Start getting some principals:
# /usr/sbin/kadmin.local Authenticating as principal root/admin@INTERNAL.EXCFB.COM with password. kadmin.local: addprinc tdh/admin WARNING: no policy specified for tdh/admin@INTERNAL.EXCFB.COM; defaulting to no policy Enter password for principal "tdh/admin@INTERNAL.EXCFB.COM": Re-enter password for principal "tdh/admin@INTERNAL.EXCFB.COM": Principal "tdh/admin@INTERNAL.EXCFB.COM" created.
Get some kiprop installed:
kadmin.local: addprinc -randkey kiprop/sandman.internal.excfb.com WARNING: no policy specified for kiprop/sandman.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy add_principal: Principal or policy already exists while creating "kiprop/sandman.internal.excfb.com@INTERNAL.EXCFB.COM". kadmin.local: addprinc -randkey kiprop/ultralord.internal.excfb.com WARNING: no policy specified for kiprop/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy Principal "kiprop/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM" created.
Enable kadmin and changepw:
kadmin.local: ktadd -k /etc/krb5/kadm.keytab kadmin/sandman.internal.excfb.com Entry for principal kadmin/sandman.internal.excfb.com with kvno 3, encryption type AES-128 CTS mode with 96-bit SHA-1 HMAC added to keytab WRFILE:/etc/krb5/kadm.keytab. Entry for principal kadmin/sandman.internal.excfb.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5/kadm.keytab. Entry for principal kadmin/sandman.internal.excfb.com with kvno 3, encryption type ArcFour with HMAC/md5 added to keytab WRFILE:/etc/krb5/kadm.keytab. Entry for principal kadmin/sandman.internal.excfb.com with kvno 3, encryption type DES cbc mode with RSA-MD5 added to keytab WRFILE:/etc/krb5/kadm.keytab. kadmin.local: ktadd -k /etc/krb5/kadm.keytab changepw/sandman.internal.excfb.com Entry for principal changepw/sandman.internal.excfb.com with kvno 3, encryption type AES-128 CTS mode with 96-bit SHA-1 HMAC added to keytab WRFILE:/etc/krb5/kadm.keytab. Entry for principal changepw/sandman.internal.excfb.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5/kadm.keytab. Entry for principal changepw/sandman.internal.excfb.com with kvno 3, encryption type ArcFour with HMAC/md5 added to keytab WRFILE:/etc/krb5/kadm.keytab. Entry for principal changepw/sandman.internal.excfb.com with kvno 3, encryption type DES cbc mode with RSA-MD5 added to keytab WRFILE:/etc/krb5/kadm.keytab.
Enable kiprop:
kadmin.local: ktadd -k /etc/krb5/kadm.keytab kiprop/sandman.internal.excfb.com Entry for principal kiprop/sandman.internal.excfb.com with kvno 3, encryption type AES-128 CTS mode with 96-bit SHA-1 HMAC added to keytab WRFILE:/etc/krb5/kadm.keytab. Entry for principal kiprop/sandman.internal.excfb.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5/kadm.keytab. Entry for principal kiprop/sandman.internal.excfb.com with kvno 3, encryption type ArcFour with HMAC/md5 added to keytab WRFILE:/etc/krb5/kadm.keytab. Entry for principal kiprop/sandman.internal.excfb.com with kvno 3, encryption type DES cbc mode with RSA-MD5 added to keytab WRFILE:/etc/krb5/kadm.keytab.
Quit:
kadmin.local: quit
Enable the services:
# svcadm enable -r network/security/krb5kdc # svcadm enable -r network/security/kadmin
Authenticate the admin account:
# /usr/sbin/kadmin -p tdh/admin Authenticating as principal tdh/admin with password. Password for tdh/admin@INTERNAL.EXCFB.COM: kadmin: Communication failure with server while initializing kadmin interface
Hmm, I got the right password. I can see what happens when it is wrong:
# /usr/sbin/kadmin -p tdh/admin Authenticating as principal tdh/admin with password. Password for tdh/admin@INTERNAL.EXCFB.COM: kadmin: Incorrect password while initializing kadmin interface
Ahh, lets see if kerberos is up and running:
# grep kadmin /var/adm/messages Feb 11 23:31:19 sandman svc.startd[7]: [ID 748625 daemon.error] network/security/kadmin:default failed repeatedly: transitioned to maintenance (see 'svcs -xv' for details) Feb 11 23:31:57 sandman kadmin[4143]: [ID 737709 user.error] unable to open connection to ADMIN server (t_error 9) Feb 11 23:33:56 sandman kadmin[4146]: [ID 737709 user.error] unable to open connection to ADMIN server (t_error 9)
No, it is not.
# svcs -xv svc:/network/security/kadmin:default (Kerberos administration daemon) State: maintenance since Sun Feb 11 23:31:19 2007 Reason: Restarting too quickly. See: http://sun.com/msg/SMF-8000-L5 See: man -M /usr/share/man -s 1M kadmind See: /var/svc/log/network-security-kadmin:default.log Impact: This service is not running.
Clear the maintenance state:
# svcadm clear /network/security/kadmin:default
Restart:
# svcadm enable -r network/security/kadminCheck:
# svcs -xv #
And try again:
# /usr/sbin/kadmin -p tdh/admin Authenticating as principal tdh/admin with password. Password for tdh/admin@INTERNAL.EXCFB.COM: kadmin: Communication failure with server while initializing kadmin interface
If we look at kadm5.acl:
*/admin@___default_realm___ *
Hmm, touch that up:
*/admin@INTERNAL.EXCFB.COM *
And for sanity:
# grep default * kdc.conf:[kdcdefaults] kdc.conf: default_principal_flags = +preauth krb5.conf:[libdefaults] krb5.conf: default_realm = INTERNAL.EXCFB.COM krb5.conf: ___domainname___ = ___default_realm___ krb5.conf: default = FILE:/var/krb5/kdc.log krb5.conf:[appdefaults]
Okay, time to fix up krb5.conf as well:
[domain_realm]
___domainname___ = INTERNAL.EXCFB.COM
And restart:
# svcadm restart network/security/krb5kdc # svcadm restart network/security/kadmin
And try again:
# /usr/sbin/kadmin -p tdh/admin Authenticating as principal tdh/admin with password. Password for tdh/admin@INTERNAL.EXCFB.COM: kadmin: Communication failure with server while initializing kadmin interface
Okay, we know it is talking to something, i.e., it understands a bad password.
Lets try something else:
# kadmin.local Authenticating as principal root/admin@INTERNAL.EXCFB.COM with password. kadmin.local: addprinc admin/admin@INTERNAL.EXCFB.COM WARNING: no policy specified for admin/admin@INTERNAL.EXCFB.COM; defaulting to no policy Enter password for principal "admin/admin@INTERNAL.EXCFB.COM": Re-enter password for principal "admin/admin@INTERNAL.EXCFB.COM": Principal "admin/admin@INTERNAL.EXCFB.COM" created. kadmin.local: quit
Okay, time to search. If we look at System Administration Guide: Security Services :
Communication failure with server while initializing kadmin interface
Cause: The host that was entered for the admin server, also called the master KDC,
did not have the kadmind daemon running.
Solution: Make sure that you specified the correct host name for the master KDC.
If you specified the correct host name, make sure that kadmind is running on
the master KDC that you specified.
But wait:
# svcs | grep krb online 23:43:04 svc:/network/security/krb5kdc:default # svcs | grep kad maintenance 23:42:54 svc:/network/security/kadmin:default # svcs -vx svc:/network/security/kadmin:default (Kerberos administration daemon) State: maintenance since Sun Feb 11 23:42:54 2007 Reason: Restarting too quickly. See: http://sun.com/msg/SMF-8000-L5 See: man -M /usr/share/man -s 1M kadmind See: /var/svc/log/network-security-kadmin:default.log Impact: This service is not running.
Lets look at the log file:
Feb 11 23:42:53 sandman kadmind[4275](Error): Keytab file "/etc/krb5/kadm5.keytab" does not exist
Feb 11 23:42:53 sandman kadmind[4275](Error): Keytab file "/etc/krb5/kadm5.keytab" does not exist
Feb 11 23:42:53 sandman kadmind[4275](info): No dictionary file specified, continuing without one.
Feb 11 23:42:53 sandman kadmind[4275](Error): Unable to set RPCSEC_GSS service names ('kadmin@sandman.internal.excfb.com,changepw@sandman.internal.excfb.com')
krb5kdc: Interrupted system call - while selecting for network input(1)
Feb 11 23:43:03 sandman krb5kdc[4105](info): shutting down
Hmm, we need to create a keytab:
# ls -la /etc/krb5/kadm5.keytab /etc/krb5/kadm5.keytab: No such file or directory
Ack, why do I have a kadm.keytab and not a kadm5.keytab?
# mv kadm.keytab kadm5.keytab
Because that is what I frigging entered in my session!
# /usr/sbin/kadmin -p tdh/admin Authenticating as principal tdh/admin with password. Password for tdh/admin@INTERNAL.EXCFB.COM: kadmin:
The correct incantations should have been:
kadmin.local: ktadd -k /etc/krb5/kadm5.keytab kadmin/sandman.internal.excfb.com kadmin.local: ktadd -k /etc/krb5/kadm5.keytab changepw/sandman.internal.excfb.com kadmin.local: ktadd -k /etc/krb5/kadm5.keytab kiprop/sandman.internal.excfb.com
Okay, back to our regularly scheduled programming:
What principals exist?
kadmin: listprincs K/M@INTERNAL.EXCFB.COM admin/admin@INTERNAL.EXCFB.COM changepw/sandman.internal.excfb.com@INTERNAL.EXCFB.COM kadmin/changepw@INTERNAL.EXCFB.COM kadmin/history@INTERNAL.EXCFB.COM kadmin/sandman.internal.excfb.com@INTERNAL.EXCFB.COM kiprop/sandman.internal.excfb.com@INTERNAL.EXCFB.COM kiprop/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM krbtgt/INTERNAL.EXCFB.COM@INTERNAL.EXCFB.COM tdh/admin@INTERNAL.EXCFB.COM
To kerberize NFS, we need to touch up /etc/nfssec.conf:
# diff nfssec.conf nfssec.conf.stock 48,50c48,50 < krb5 390003 kerberos_v5 default - # RPCSEC_GSS < krb5i 390004 kerberos_v5 default integrity # RPCSEC_GSS < krb5p 390005 kerberos_v5 default privacy # RPCSEC_GSS --- > #krb5 390003 kerberos_v5 default - # RPCSEC_GSS > #krb5i 390004 kerberos_v5 default integrity # RPCSEC_GSS > #krb5p 390005 kerberos_v5 default privacy # RPCSEC_GSS
We need to add a nfs principal:
kadmin: addprinc -randkey nfs/sandman.internal.excfb.com WARNING: no policy specified for nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy Principal "nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM" created. kadmin: ktadd nfs/sandman.internal.excfb.com Entry for principal nfs/sandman.internal.excfb.com with kvno 3, encryption type AES-128 CTS mode with 96-bit SHA-1 HMAC added to keytab WRFILE:/etc/krb5/krb5.keytab. Entry for principal nfs/sandman.internal.excfb.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5/krb5.keytab. Entry for principal nfs/sandman.internal.excfb.com with kvno 3, encryption type ArcFour with HMAC/md5 added to keytab WRFILE:/etc/krb5/krb5.keytab. Entry for principal nfs/sandman.internal.excfb.com with kvno 3, encryption type DES cbc mode with RSA-MD5 added to keytab WRFILE:/etc/krb5/krb5.keytab.
Verify that is does indeed exist:
# klist -k Keytab name: FILE:/etc/krb5/krb5.keytab KVNO Principal ---- -------------------------------------------------------------------------- 3 nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM 3 nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM 3 nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM 3 nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM
And now we are going to have to make a share that is kerberized and setup a client to access it:
# /usr/sbin/kclient
Starting client setup
---------------------------------------------------
Do you want to use DNS for kerberos lookups ? [y/n]: n
No action performed.
Enter the Kerberos realm: INTERNAL.EXCFB.COM
Specify the KDC hostname for the above realm: sandman.internal.excfb.com
sandman.internal.excfb.com
Note, this system and the KDC's time must be within 5 minutes of each other for Kerberos to function. Both systems should run some form of time
synchronization system like Network Time Protocol (NTP).
Setting up /etc/krb5/krb5.conf.
Enter the krb5 administrative principal to be used: tdh/admin
Obtaining TGT for tdh/admin ...
Password for tdh/admin@INTERNAL.EXCFB.COM:
Do you have multiple DNS domains spanning the Kerberos realm INTERNAL.EXCFB.COM ? [y/n]: n
No action performed.
Do you plan on doing Kerberized nfs ? [y/n]: y
nfs/ultralord.internal.excfb.com entry ADDED to KDC database.
nfs/ultralord.internal.excfb.com entry ADDED to keytab.
host/ultralord.internal.excfb.com entry ADDED to KDC database.
host/ultralord.internal.excfb.com entry ADDED to keytab.
Do you want to copy over the master krb5.conf file ? [y/n]: y
Enter the pathname of the file to be copied: /etc/krb5/krb5.conf
cp: /etc/krb5/krb5.conf and /etc/krb5/krb5.conf are identical
Copy of /etc/krb5/krb5.conf failed, exiting.
---------------------------------------------------
Setup FAILED.
Hmm, how are we supposed to enter that? I bet we need to use /net. Which I don't have configured right now. Okay, the hard way:
# scp sandman:/etc/krb5/krb5.conf /etc/krb5/krb5.conf
Now, lets set up a test share:
# cd /export # mkdir kerberos # cd kerberos # touch see_me # chown tdh:staff see_me # ls -la total 4 drwxr-xr-x 2 root root 512 Feb 12 00:23 . drwxr-xr-x 4 root sys 512 Feb 12 00:23 .. -rw-r--r-- 1 tdh staff 0 Feb 12 00:23 see_me # share -F nfs -o sec=krb5:krb5i:krb5p -d "Kerberos" /export/kerberos # share -F nfs -d "Home dirs" /export/home # share - /export/kerberos sec=krb5,sec=krb5i,sec=krb5p "Kerberos" - /export/home rw "Home dirs"
Now try to get some access:
[tdh@ultralord ~]> kinit kinit(v5): Client not found in Kerberos database while getting initial credentials [tdh@ultralord ~]> sudo klist -k Keytab name: FILE:/etc/krb5/krb5.keytab KVNO Principal ---- -------------------------------------------------------------------------- 4 nfs/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM 4 nfs/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM 4 nfs/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM 4 nfs/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM 4 host/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM 4 host/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM 4 host/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM 4 host/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM
Okay, I think I need to add user principals for tdh:
kadmin: addprinc tdh WARNING: no policy specified for tdh@INTERNAL.EXCFB.COM; defaulting to no policy Enter password for principal "tdh@INTERNAL.EXCFB.COM": Re-enter password for principal "tdh@INTERNAL.EXCFB.COM": Principal "tdh@INTERNAL.EXCFB.COM" created. [tdh@ultralord ~]> kinit Password for tdh@INTERNAL.EXCFB.COM:
And now I want to get a mount:
[tdh@ultralord ~]> sudo mkdir -p /mnt/sandman/home [tdh@ultralord ~]> sudo mkdir -p /mnt/sandman/kerberos [tdh@ultralord ~]> sudo showmount -e sandman export list for sandman: /export/kerberos (everyone) /export/home (everyone) [tdh@ultralord ~]> sudo mount sandman:/export/kerberos /mnt/sandman/kerberos [tdh@ultralord ~]> sudo mount sandman:/export/home /mnt/sandman/home [tdh@ultralord ~]> ls -al /mnt/sandman/kerberos total 4 drwxr-xr-x 2 root root 512 Feb 12 00:23 . drwxr-xr-x 4 root root 512 Feb 12 00:36 .. -rw-r--r-- 1 tdh staff 0 Feb 12 00:23 see_me [tdh@ultralord ~]> ls -la /mnt/sandman/home total 22 drwxr-xr-x 4 root root 512 Dec 30 15:01 . drwxr-xr-x 4 root root 512 Feb 12 00:36 .. drwx------ 2 root root 8192 Dec 20 11:28 lost+found drwxr-xr-x 4 tdh staff 512 Jan 21 20:48 tdh
Success!
But wait, we need to show that a client without kerberos enabled will be denied access to sandman:/export/kerberos:
[tdh@kanigix ~]> sudo mkdir -p /mnt/sandman/home [tdh@kanigix ~]> sudo mkdir -p /mnt/sandman/kerberos [tdh@kanigix ~]> sudo mount sandman:/export/kerberos /mnt/sandman/kerberos nfs mount: mount: /mnt/sandman/kerberos: Permission denied
Some other things to do would be to setup /etc/pam.conf to allow single signon - i.e., use ssh without a password. We also need to setup ultralord as a slave.
But before I tune this out, we need to get a Linux client up and running. Why? Because we need to show we can interoperate.
Some systems only support single DES, so we need to create special keytabs for them:
kadmin: addprinc -randkey nfs/mrx.internal.excfb.com WARNING: no policy specified for nfs/mrx.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy Principal "nfs/mrx.internal.excfb.com@INTERNAL.EXCFB.COM" created. kadmin: addprinc -randkey host/mrx.internal.excfb.com WARNING: no policy specified for host/mrx.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy Principal "host/mrx.internal.excfb.com@INTERNAL.EXCFB.COM" created.
Now, I've created /export/keytabs to store the keytab files we will need:
# cd /export # mkdir keytabs # share -F nfs -o ro /export/keytabs
And we can create the keytab:
kadmin: ktadd -k /export/keytabs/mrx.keytab -e des-cbc-crc:normal nfs/mrx.internal.excfb.com Entry for principal nfs/mrx.internal.excfb.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/export/keytabs/mrx.keytab. kadmin: ktadd -k /export/keytabs/mrx.keytab -e des-cbc-crc:normal host/mrx.internal.excfb.com Entry for principal host/mrx.internal.excfb.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/export/keytabs/mrx.keytab.
We see we are in business:
# cp /etc/krb5/krb5.conf /export/keytabs/ # ls -la total 10 drwxr-xr-x 2 root root 512 Feb 12 00:50 . drwxr-xr-x 5 root sys 512 Feb 12 00:46 .. -rw-r--r-- 1 root root 1968 Feb 12 00:50 krb5.conf -rw------- 1 root root 155 Feb 12 00:48 mrx.keytab # chmod +r mrx.keytab
And now we setup the Linux machine:
[root@mrx ~]# mkdir -p /mnt/sandman/keytabs [root@mrx ~]# showmount -e sandman Export list for sandman: /export/kerberos (everyone) /export/home (everyone) /export/keytabs (everyone) [root@mrx ~]# mount sandman:/export/keytabs /mnt/sandman/keytabs
We should make sure we do not have access to sandman:/export/kerberos:
[root@mrx ~]# mkdir -p /mnt/sandman/kerberos [root@mrx ~]# mkdir -p /mnt/sandman/home [root@mrx ~]# mount sandman:/export/kerberos /mnt/sandman/kerberos mount: sandman:/export/kerberos failed, security flavor not supported
What do we need to change:
[root@mrx ~]# cd /etc [root@mrx etc]# ls -la k* -rw-r--r-- 1 root root 657 Jan 9 14:03 krb5.conf -rw-r--r-- 1 root root 2241 Jul 13 2006 krb.conf -rw-r--r-- 1 root root 1296 Jul 13 2006 krb.realms [root@mrx etc]# mkdir stock [root@mrx etc]# cp k* stock [root@mrx etc]# cp /mnt/sandman/keytabs/krb5.conf . cp: overwrite `./krb5.conf'? y [root@mrx etc]# cp /mnt/sandman/keytabs/mrx.keytab krb5.keytab
And we try to authenticate:
[tdh@mrx ~]> kinit kinit: Command not found.
Okay, we need to install the kerberos packages:
[tdh@mrx /]> sudo yum install krb5-workstation Loading "installonlyn" plugin Setting up Install Process Setting up repositories Reading repository metadata in from local files Parsing package install arguments Nothing to do
No, we don't. Where is that rascally rabbit?
[tdh@mrx /]> sudo find . -name kinit ./usr/kerberos/bin/kinit [tdh@mrx /]> ./usr/kerberos/bin/kinit Password for tdh@INTERNAL.EXCFB.COM:
And we try the mount:
[tdh@mrx /]> sudo mount sandman:/export/kerberos /mnt/sandman/kerberos
mount: sandman:/export/kerberos failed, security flavor not supported
[tdh@mrx /]> ./usr/kerberos/bin/klist
Ticket cache: FILE:/tmp/krb5cc_1066
Default principal: tdh@INTERNAL.EXCFB.COM
Valid starting Expires Service principal
02/12/07 01:01:42 02/12/07 09:01:42 krbtgt/INTERNAL.EXCFB.COM@INTERNAL.EXCFB.COM
renew until 02/13/07 00:59:17
Kerberos 4 ticket cache: /tmp/tkt1066
klist: You have no tickets cached
What is up here?
# snoop -x 0,2000 -o /tmp/m2s.snoop sandman mrx Using device /dev/hme (promiscuous mode) 33 ^C
Note: I used -x 0,2000 to get payload data. I knew I would want to look at most of the packet.
And
[tdh@mrx ~]> sudo mount -t nfs4 sandman:/export/kerberos /mnt/sandman/kerberos mount.nfs4: Operation not permitted 26 0.00034 mrx.internal.excfb.com -> sandman NFS C 4 () PUTFH FH=324D LOOKUP export GETFH GETATTR 10011a 30a23a 27 0.00030 sandman -> mrx.internal.excfb.com NFS R 4 () NFS4_OK PUTFH NFS4_OK LOOKUP NFS4_OK GETFH NFS4_OK FH=30E6 GETATTR NFS4_OK 28 0.00033 mrx.internal.excfb.com -> sandman NFS C 4 () PUTFH FH=30E6 LOOKUP kerberos GETFH GETATTR 10011a 30a23a 29 0.00021 sandman -> mrx.internal.excfb.com NFS R 4 () NFS4ERR_WRONGSEC PUTFH NFS4_OK LOOKUP NFS4ERR_WRONGSEC
I popped into wireshark and I found out that mrx is only sending AUTH_SYS and AUTH_NULL.
Note: I used wireshark because it will parse the payload data for me. I didn't want to be doing byte conversions and consulting some specs!
In NetApp Filer, NFSv4, and Linux, we find using -o sec=krb5. We can try that:
[tdh@mrx ~]> sudo mount -t nfs4 -o sec=krb5 sandman:/export/kerberos /mnt/sandman/kerberos Warning: rpc.gssd appears not to be running. mount.nfs4: Invalid argument
Which is strange, since it is running:
[tdh@mrx ~]> sudo chkconfig --list | grep rpcgssd rpcgssd 0:off 1:off 2:off 3:on 4:on 5:on 6:off [tdh@mrx ~]> sudo chkconfig --list | grep rpcidmapd rpcidmapd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
What does the log state:
RPC: Couldn't create auth handle (flavor 390003)
I've copied the stock krb5.conf back and now the diffs are:
[tdh@mrx /etc]> diff krb5.conf stock/krb5.conf
7c7
< default_realm = INTERNAL.EXCFB.COM
---
> default_realm = EXAMPLE.COM
14,17c14,17
< INTERNAL.EXCFB.COM = {
< kdc = sandman.internal.excfb.com:88
< admin_server = sandman.internal.excfb.com:749
< default_domain = internal.excfb.com
---
> EXAMPLE.COM = {
> kdc = kerberos.example.com:88
> admin_server = kerberos.example.com:749
> default_domain = example.com
21,22c21,22
< .internal.excfb.com = INTERNAL.EXCFB.COM
< internal.excfb.com = INTERNAL.EXCFB.COM
---
> .example.com = EXAMPLE.COM
> example.com = EXAMPLE.COM
You know what, rpc.gssd is not running!
[tdh@mrx /etc]> ps -ef | grep rpc rpc 1877 1 0 01:49 ? 00:00:00 portmap root 1898 1 0 01:49 ? 00:00:00 rpc.statd root 1931 1 0 01:49 ? 00:00:00 rpc.idmapd tdh 2697 2519 0 02:04 pts/0 00:00:00 grep rpc [tdh@mrx /etc]> sudo sh -c "ulimit -c unlimited;/usr/sbin/rpc.gssd -f -vvv" Using keytab file '/etc/krb5.keytab' Processing keytab entry for principal 'nfs/mrx.internal.excfb.com@INTERNAL.EXCFB.COM' We will use this entry (nfs/mrx.internal.excfb.com@INTERNAL.EXCFB.COM) Processing keytab entry for principal 'host/mrx.internal.excfb.com@INTERNAL.EXCFB.COM' We will NOT use this entry (host/mrx.internal.excfb.com@INTERNAL.EXCFB.COM) Using (machine) credentials cache: 'MEMORY:/tmp/krb5cc_machine_INTERNAL.EXCFB.COM'
And I put it in the background. Hmm, why doesn't it like the host entry?
Alright, I went back to why isn't rpc.gssd starting up at boot:
[ -f /etc/sysconfig/nfs ] && . /etc/sysconfig/nfs
[ "${SECURE_NFS}" != "yes" ] && exit 0
# ls -la /etc/sysconfig/nfs
#
Time to create it (look at Learning NFSv4 with Fedora Core 2 (Linux 2.6. 5 kernel))
# This entry should be "yes" if you are using RPCSEC_GSS_KRB5 (auth=krb5,krb5i, or krb5p) SECURE_NFS="yes" # This entry sets the number of NFS server processes. 8 is the default RPCNFSDCOUNT=8 [tdh@mrx sysconfig]> sudo /etc/init.d/rpcgssd start Starting RPC gssd: [ OK ]
God I'm totally hacked about this:
[tdh@mrx sysconfig]> sudo mount -o sec=krb5 sandman:/export/kerberos /mnt/sandman/kerberos [tdh@mrx sysconfig]> ls -la /mnt/sandman/kerberos total 5 drwxr-xr-x 2 root root 512 Feb 12 00:23 . drwxr-xr-x 5 root root 4096 Feb 12 00:49 .. -rw-r--r-- 1 tdh wheel 0 Feb 12 00:23 see_me
Copyright (C) 2009, Kool Aid Served Daily