We always seem to have problems at Connectathon setting up Kerberos. So I decided to take the cookbook we use there and get kerberos working on my home systems. Please note that I could easily clean up the notes to not show some errors I make. But then, where is the love?

Also, as with any first foray into a new tool, I have no clue what I am doing. I kinda understand tickets and the ideas behind Kerberos, but I'm really in the dark as to what I'm supposed to do.

First edit /etc/krb5/krb5.conf:

# diff krb5.conf stock/krb5.conf
35c35
<         default_realm = INTERNAL.EXCFB.COM
---
>         default_realm = ___default_realm___
38,41c38,43
<         INTERNAL.EXCFB.COM = {
<                 kdc = sandman.internal.excfb.com
<                 kdc = ultralord.internal.excfb.com
<                 admin_server = sandman.internal.excfb.com
---
>         ___default_realm___ = {
>                 kdc = ___master_kdc___
>                 kdc = ___slave_kdc1___
>                 kdc = ___slave_kdc2___
>                 kdc = ___slave_kdcN___
>                 admin_server = ___master_kdc___

Then edit /etc/krb5/kdc.conf:

# diff kdc.conf stock/kdc.conf
32c32
<       INTERNAL.EXCFB.COM = {
---
>       ___default_realm___ = {
41,42d40
<               sunw_dbprob_enable = true
<               sunw_dbprop_master_ulogsize = 1000

Make sure you can get at the kdcs via DNS (or whatever name service in /etc/resolv.conf)

# host sandman
sandman.internal.excfb.com has address 192.168.2.109
# host sandman.internal.excfb.com
sandman.internal.excfb.com has address 192.168.2.109

Create the kerberos database

# /usr/sbin/kdb5_util create -r INTERNAL.EXCFB.COM -s
Initializing database '/var/krb5/principal' for realm 'INTERNAL.EXCFB.COM',
master key name 'K/M@INTERNAL.EXCFB.COM'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify:

Start getting some principals:

# /usr/sbin/kadmin.local
Authenticating as principal root/admin@INTERNAL.EXCFB.COM with password.
kadmin.local:  addprinc tdh/admin
WARNING: no policy specified for tdh/admin@INTERNAL.EXCFB.COM; defaulting to no policy
Enter password for principal "tdh/admin@INTERNAL.EXCFB.COM":
Re-enter password for principal "tdh/admin@INTERNAL.EXCFB.COM":
Principal "tdh/admin@INTERNAL.EXCFB.COM" created.

Get some kiprop installed:

kadmin.local:  addprinc -randkey kiprop/sandman.internal.excfb.com
WARNING: no policy specified for kiprop/sandman.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kiprop/sandman.internal.excfb.com@INTERNAL.EXCFB.COM".
kadmin.local:  addprinc -randkey kiprop/ultralord.internal.excfb.com
WARNING: no policy specified for kiprop/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy
Principal "kiprop/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM" created.

Enable kadmin and changepw:

kadmin.local:  ktadd -k /etc/krb5/kadm.keytab kadmin/sandman.internal.excfb.com
Entry for principal kadmin/sandman.internal.excfb.com with kvno 3, encryption type AES-128 CTS mode with 96-bit SHA-1 HMAC added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal kadmin/sandman.internal.excfb.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal kadmin/sandman.internal.excfb.com with kvno 3, encryption type ArcFour with HMAC/md5 added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal kadmin/sandman.internal.excfb.com with kvno 3, encryption type DES cbc mode with RSA-MD5 added to keytab WRFILE:/etc/krb5/kadm.keytab.
kadmin.local:  ktadd -k /etc/krb5/kadm.keytab changepw/sandman.internal.excfb.com
Entry for principal changepw/sandman.internal.excfb.com with kvno 3, encryption type AES-128 CTS mode with 96-bit SHA-1 HMAC added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal changepw/sandman.internal.excfb.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal changepw/sandman.internal.excfb.com with kvno 3, encryption type ArcFour with HMAC/md5 added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal changepw/sandman.internal.excfb.com with kvno 3, encryption type DES cbc mode with RSA-MD5 added to keytab WRFILE:/etc/krb5/kadm.keytab.

Enable kiprop:

kadmin.local:  ktadd -k /etc/krb5/kadm.keytab kiprop/sandman.internal.excfb.com
Entry for principal kiprop/sandman.internal.excfb.com with kvno 3, encryption type AES-128 CTS mode with 96-bit SHA-1 HMAC added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal kiprop/sandman.internal.excfb.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal kiprop/sandman.internal.excfb.com with kvno 3, encryption type ArcFour with HMAC/md5 added to keytab WRFILE:/etc/krb5/kadm.keytab.
Entry for principal kiprop/sandman.internal.excfb.com with kvno 3, encryption type DES cbc mode with RSA-MD5 added to keytab WRFILE:/etc/krb5/kadm.keytab.

Quit:

kadmin.local:  quit

Enable the services:

# svcadm enable -r network/security/krb5kdc
# svcadm enable -r network/security/kadmin

Authenticate the admin account:

# /usr/sbin/kadmin -p tdh/admin
Authenticating as principal tdh/admin with password.
Password for tdh/admin@INTERNAL.EXCFB.COM:
kadmin: Communication failure with server while initializing kadmin interface

Hmm, I got the right password. I can see what happens when it is wrong:

# /usr/sbin/kadmin -p tdh/admin
Authenticating as principal tdh/admin with password.
Password for tdh/admin@INTERNAL.EXCFB.COM:
kadmin: Incorrect password while initializing kadmin interface

Ahh, lets see if kerberos is up and running:

# grep kadmin /var/adm/messages
Feb 11 23:31:19 sandman svc.startd[7]: [ID 748625 daemon.error] network/security/kadmin:default failed repeatedly: transitioned to maintenance (see 'svcs -xv' for details)
Feb 11 23:31:57 sandman kadmin[4143]: [ID 737709 user.error] unable to open connection to ADMIN server (t_error 9)
Feb 11 23:33:56 sandman kadmin[4146]: [ID 737709 user.error] unable to open connection to ADMIN server (t_error 9)

No, it is not.

# svcs -xv
svc:/network/security/kadmin:default (Kerberos administration daemon)
 State: maintenance since Sun Feb 11 23:31:19 2007
Reason: Restarting too quickly.
   See: http://sun.com/msg/SMF-8000-L5
   See: man -M /usr/share/man -s 1M kadmind
   See: /var/svc/log/network-security-kadmin:default.log
Impact: This service is not running.

Clear the maintenance state:

# svcadm clear /network/security/kadmin:default

Restart:

# svcadm enable -r network/security/kadmin

Check:
# svcs -xv
#

And try again:

# /usr/sbin/kadmin -p tdh/admin
Authenticating as principal tdh/admin with password.
Password for tdh/admin@INTERNAL.EXCFB.COM:
kadmin: Communication failure with server while initializing kadmin interface

If we look at kadm5.acl:

*/admin@___default_realm___ *

Hmm, touch that up:

*/admin@INTERNAL.EXCFB.COM *

And for sanity:

# grep default *
kdc.conf:[kdcdefaults]
kdc.conf:               default_principal_flags = +preauth
krb5.conf:[libdefaults]
krb5.conf:        default_realm = INTERNAL.EXCFB.COM
krb5.conf:      ___domainname___ = ___default_realm___
krb5.conf:        default = FILE:/var/krb5/kdc.log
krb5.conf:[appdefaults]

Okay, time to fix up krb5.conf as well:

[domain_realm]
        ___domainname___ = INTERNAL.EXCFB.COM

And restart:

# svcadm restart network/security/krb5kdc
# svcadm restart network/security/kadmin

And try again:

# /usr/sbin/kadmin -p tdh/admin
Authenticating as principal tdh/admin with password.
Password for tdh/admin@INTERNAL.EXCFB.COM:
kadmin: Communication failure with server while initializing kadmin interface

Okay, we know it is talking to something, i.e., it understands a bad password.

Lets try something else:

# kadmin.local
Authenticating as principal root/admin@INTERNAL.EXCFB.COM with password.
kadmin.local:  addprinc admin/admin@INTERNAL.EXCFB.COM
WARNING: no policy specified for admin/admin@INTERNAL.EXCFB.COM; defaulting to no policy
Enter password for principal "admin/admin@INTERNAL.EXCFB.COM":
Re-enter password for principal "admin/admin@INTERNAL.EXCFB.COM":
Principal "admin/admin@INTERNAL.EXCFB.COM" created.
kadmin.local:  quit

Okay, time to search. If we look at System Administration Guide: Security Services :

Communication failure with server while initializing kadmin interface

    Cause: The host that was entered for the admin server, also called the master KDC,
    did not have the kadmind daemon running.

    Solution: Make sure that you specified the correct host name for the master KDC.
    If you specified the correct host name, make sure that kadmind is running on
    the master KDC that you specified.

But wait:

# svcs | grep krb
online         23:43:04 svc:/network/security/krb5kdc:default
# svcs | grep kad
maintenance    23:42:54 svc:/network/security/kadmin:default
# svcs -vx
svc:/network/security/kadmin:default (Kerberos administration daemon)
 State: maintenance since Sun Feb 11 23:42:54 2007
Reason: Restarting too quickly.
   See: http://sun.com/msg/SMF-8000-L5
   See: man -M /usr/share/man -s 1M kadmind
   See: /var/svc/log/network-security-kadmin:default.log
Impact: This service is not running.

Lets look at the log file:

Feb 11 23:42:53 sandman kadmind[4275](Error): Keytab file "/etc/krb5/kadm5.keytab" does not exist
Feb 11 23:42:53 sandman kadmind[4275](Error): Keytab file "/etc/krb5/kadm5.keytab" does not exist
Feb 11 23:42:53 sandman kadmind[4275](info): No dictionary file specified, continuing without one.
Feb 11 23:42:53 sandman kadmind[4275](Error): Unable to set RPCSEC_GSS service names ('kadmin@sandman.internal.excfb.com,changepw@sandman.internal.excfb.com')
krb5kdc: Interrupted system call - while selecting for network input(1)
Feb 11 23:43:03 sandman krb5kdc[4105](info): shutting down

Hmm, we need to create a keytab:

# ls -la /etc/krb5/kadm5.keytab
/etc/krb5/kadm5.keytab: No such file or directory

Ack, why do I have a kadm.keytab and not a kadm5.keytab?

# mv kadm.keytab kadm5.keytab

Because that is what I frigging entered in my session!

# /usr/sbin/kadmin -p tdh/admin
Authenticating as principal tdh/admin with password.
Password for tdh/admin@INTERNAL.EXCFB.COM:
kadmin:

The correct incantations should have been:

kadmin.local:  ktadd -k /etc/krb5/kadm5.keytab kadmin/sandman.internal.excfb.com
kadmin.local:  ktadd -k /etc/krb5/kadm5.keytab changepw/sandman.internal.excfb.com
kadmin.local:  ktadd -k /etc/krb5/kadm5.keytab kiprop/sandman.internal.excfb.com

Okay, back to our regularly scheduled programming:

What principals exist?

kadmin:  listprincs
K/M@INTERNAL.EXCFB.COM
admin/admin@INTERNAL.EXCFB.COM
changepw/sandman.internal.excfb.com@INTERNAL.EXCFB.COM
kadmin/changepw@INTERNAL.EXCFB.COM
kadmin/history@INTERNAL.EXCFB.COM
kadmin/sandman.internal.excfb.com@INTERNAL.EXCFB.COM
kiprop/sandman.internal.excfb.com@INTERNAL.EXCFB.COM
kiprop/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM
krbtgt/INTERNAL.EXCFB.COM@INTERNAL.EXCFB.COM
tdh/admin@INTERNAL.EXCFB.COM

To kerberize NFS, we need to touch up /etc/nfssec.conf:

# diff nfssec.conf nfssec.conf.stock
48,50c48,50
< krb5          390003  kerberos_v5     default -               # RPCSEC_GSS
< krb5i         390004  kerberos_v5     default integrity       # RPCSEC_GSS
< krb5p         390005  kerberos_v5     default privacy         # RPCSEC_GSS
---
> #krb5         390003  kerberos_v5     default -               # RPCSEC_GSS
> #krb5i                390004  kerberos_v5     default integrity       # RPCSEC_GSS
> #krb5p                390005  kerberos_v5     default privacy         # RPCSEC_GSS

We need to add a nfs principal:

kadmin:  addprinc -randkey nfs/sandman.internal.excfb.com
WARNING: no policy specified for nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy
Principal "nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM" created.
kadmin:  ktadd nfs/sandman.internal.excfb.com
Entry for principal nfs/sandman.internal.excfb.com with kvno 3, encryption type AES-128 CTS mode with 96-bit SHA-1 HMAC added to keytab WRFILE:/etc/krb5/krb5.keytab.
Entry for principal nfs/sandman.internal.excfb.com with kvno 3, encryption type Triple DES cbc mode with HMAC/sha1 added to keytab WRFILE:/etc/krb5/krb5.keytab.
Entry for principal nfs/sandman.internal.excfb.com with kvno 3, encryption type ArcFour with HMAC/md5 added to keytab WRFILE:/etc/krb5/krb5.keytab.
Entry for principal nfs/sandman.internal.excfb.com with kvno 3, encryption type DES cbc mode with RSA-MD5 added to keytab WRFILE:/etc/krb5/krb5.keytab.

Verify that is does indeed exist:

# klist -k
Keytab name: FILE:/etc/krb5/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   3 nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM
   3 nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM
   3 nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM
   3 nfs/sandman.internal.excfb.com@INTERNAL.EXCFB.COM

And now we are going to have to make a share that is kerberized and setup a client to access it:

# /usr/sbin/kclient

Starting client setup

---------------------------------------------------
Do you want to use DNS for kerberos lookups ? [y/n]: n
        No action performed.
Enter the Kerberos realm: INTERNAL.EXCFB.COM
Specify the KDC hostname for the above realm: sandman.internal.excfb.com
sandman.internal.excfb.com

Note, this system and the KDC's time must be within 5 minutes of each other for Kerberos to function.  Both systems should run some form of time
 synchronization system like Network Time Protocol (NTP).

Setting up /etc/krb5/krb5.conf.

Enter the krb5 administrative principal to be used: tdh/admin
Obtaining TGT for tdh/admin ...
Password for tdh/admin@INTERNAL.EXCFB.COM:

Do you have multiple DNS domains spanning the Kerberos realm INTERNAL.EXCFB.COM ? [y/n]: n
        No action performed.

Do you plan on doing Kerberized nfs ? [y/n]: y

nfs/ultralord.internal.excfb.com entry ADDED to KDC database.
nfs/ultralord.internal.excfb.com entry ADDED to keytab.

host/ultralord.internal.excfb.com entry ADDED to KDC database.
host/ultralord.internal.excfb.com entry ADDED to keytab.

Do you want to copy over the master krb5.conf file ? [y/n]: y
Enter the pathname of the file to be copied: /etc/krb5/krb5.conf
cp: /etc/krb5/krb5.conf and /etc/krb5/krb5.conf are identical

Copy of /etc/krb5/krb5.conf failed, exiting.
---------------------------------------------------
Setup FAILED.

Hmm, how are we supposed to enter that? I bet we need to use /net. Which I don't have configured right now. Okay, the hard way:

# scp sandman:/etc/krb5/krb5.conf /etc/krb5/krb5.conf

Now, lets set up a test share:

# cd /export
# mkdir kerberos
# cd kerberos
# touch see_me
# chown tdh:staff see_me
# ls -la
total 4
drwxr-xr-x   2 root     root         512 Feb 12 00:23 .
drwxr-xr-x   4 root     sys          512 Feb 12 00:23 ..
-rw-r--r--   1 tdh      staff          0 Feb 12 00:23 see_me
# share -F nfs -o sec=krb5:krb5i:krb5p -d "Kerberos" /export/kerberos
# share -F nfs -d "Home dirs" /export/home
# share
-               /export/kerberos   sec=krb5,sec=krb5i,sec=krb5p   "Kerberos"
-               /export/home   rw   "Home dirs"

Now try to get some access:

[tdh@ultralord ~]> kinit
kinit(v5): Client not found in Kerberos database while getting initial credentials
[tdh@ultralord ~]> sudo klist -k
Keytab name: FILE:/etc/krb5/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   4 nfs/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM
   4 nfs/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM
   4 nfs/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM
   4 nfs/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM
   4 host/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM
   4 host/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM
   4 host/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM
   4 host/ultralord.internal.excfb.com@INTERNAL.EXCFB.COM

Okay, I think I need to add user principals for tdh:

kadmin:  addprinc tdh
WARNING: no policy specified for tdh@INTERNAL.EXCFB.COM; defaulting to no policy
Enter password for principal "tdh@INTERNAL.EXCFB.COM":
Re-enter password for principal "tdh@INTERNAL.EXCFB.COM":
Principal "tdh@INTERNAL.EXCFB.COM" created.

[tdh@ultralord ~]> kinit
Password for tdh@INTERNAL.EXCFB.COM:

And now I want to get a mount:

[tdh@ultralord ~]> sudo mkdir -p /mnt/sandman/home
[tdh@ultralord ~]> sudo mkdir -p /mnt/sandman/kerberos
[tdh@ultralord ~]> sudo showmount -e sandman
export list for sandman:
/export/kerberos (everyone)
/export/home     (everyone)
[tdh@ultralord ~]> sudo mount sandman:/export/kerberos /mnt/sandman/kerberos
[tdh@ultralord ~]> sudo mount sandman:/export/home /mnt/sandman/home
[tdh@ultralord ~]> ls -al /mnt/sandman/kerberos
total 4
drwxr-xr-x   2 root     root         512 Feb 12 00:23 .
drwxr-xr-x   4 root     root         512 Feb 12 00:36 ..
-rw-r--r--   1 tdh      staff          0 Feb 12 00:23 see_me
[tdh@ultralord ~]> ls -la /mnt/sandman/home
total 22
drwxr-xr-x   4 root     root         512 Dec 30 15:01 .
drwxr-xr-x   4 root     root         512 Feb 12 00:36 ..
drwx------   2 root     root        8192 Dec 20 11:28 lost+found
drwxr-xr-x   4 tdh      staff        512 Jan 21 20:48 tdh

Success!

But wait, we need to show that a client without kerberos enabled will be denied access to sandman:/export/kerberos:

[tdh@kanigix ~]> sudo mkdir -p /mnt/sandman/home
[tdh@kanigix ~]> sudo mkdir -p /mnt/sandman/kerberos
[tdh@kanigix ~]> sudo mount sandman:/export/kerberos /mnt/sandman/kerberos
nfs mount: mount: /mnt/sandman/kerberos: Permission denied

Some other things to do would be to setup /etc/pam.conf to allow single signon - i.e., use ssh without a password. We also need to setup ultralord as a slave.

But before I tune this out, we need to get a Linux client up and running. Why? Because we need to show we can interoperate.

Some systems only support single DES, so we need to create special keytabs for them:

kadmin:  addprinc -randkey nfs/mrx.internal.excfb.com
WARNING: no policy specified for nfs/mrx.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy
Principal "nfs/mrx.internal.excfb.com@INTERNAL.EXCFB.COM" created.
kadmin:  addprinc -randkey host/mrx.internal.excfb.com
WARNING: no policy specified for host/mrx.internal.excfb.com@INTERNAL.EXCFB.COM; defaulting to no policy
Principal "host/mrx.internal.excfb.com@INTERNAL.EXCFB.COM" created.

Now, I've created /export/keytabs to store the keytab files we will need:

# cd /export
# mkdir keytabs
# share -F nfs -o ro /export/keytabs

And we can create the keytab:

kadmin:  ktadd -k /export/keytabs/mrx.keytab -e des-cbc-crc:normal nfs/mrx.internal.excfb.com
Entry for principal nfs/mrx.internal.excfb.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/export/keytabs/mrx.keytab.
kadmin:  ktadd -k /export/keytabs/mrx.keytab -e des-cbc-crc:normal host/mrx.internal.excfb.com
Entry for principal host/mrx.internal.excfb.com with kvno 3, encryption type DES cbc mode with CRC-32 added to keytab WRFILE:/export/keytabs/mrx.keytab.

We see we are in business:

# cp /etc/krb5/krb5.conf /export/keytabs/
# ls -la
total 10
drwxr-xr-x   2 root     root         512 Feb 12 00:50 .
drwxr-xr-x   5 root     sys          512 Feb 12 00:46 ..
-rw-r--r--   1 root     root        1968 Feb 12 00:50 krb5.conf
-rw-------   1 root     root         155 Feb 12 00:48 mrx.keytab
# chmod +r mrx.keytab

And now we setup the Linux machine:

[root@mrx ~]# mkdir -p /mnt/sandman/keytabs
[root@mrx ~]# showmount -e sandman
Export list for sandman:
/export/kerberos (everyone)
/export/home     (everyone)
/export/keytabs  (everyone)
[root@mrx ~]# mount sandman:/export/keytabs /mnt/sandman/keytabs

We should make sure we do not have access to sandman:/export/kerberos:

[root@mrx ~]# mkdir -p /mnt/sandman/kerberos
[root@mrx ~]# mkdir -p /mnt/sandman/home
[root@mrx ~]# mount sandman:/export/kerberos /mnt/sandman/kerberos
mount: sandman:/export/kerberos failed, security flavor not supported

What do we need to change:

[root@mrx ~]# cd /etc
[root@mrx etc]# ls -la k*
-rw-r--r-- 1 root root  657 Jan  9 14:03 krb5.conf
-rw-r--r-- 1 root root 2241 Jul 13  2006 krb.conf
-rw-r--r-- 1 root root 1296 Jul 13  2006 krb.realms
[root@mrx etc]# mkdir stock
[root@mrx etc]# cp k* stock
[root@mrx etc]# cp /mnt/sandman/keytabs/krb5.conf .
cp: overwrite `./krb5.conf'? y
[root@mrx etc]# cp /mnt/sandman/keytabs/mrx.keytab krb5.keytab

And we try to authenticate:

[tdh@mrx ~]> kinit
kinit: Command not found.

Okay, we need to install the kerberos packages:

[tdh@mrx /]> sudo yum install krb5-workstation
Loading "installonlyn" plugin
Setting up Install Process
Setting up repositories
Reading repository metadata in from local files
Parsing package install arguments
Nothing to do

No, we don't. Where is that rascally rabbit?

[tdh@mrx /]> sudo find . -name kinit
./usr/kerberos/bin/kinit
[tdh@mrx /]> ./usr/kerberos/bin/kinit
Password for tdh@INTERNAL.EXCFB.COM:

And we try the mount:

[tdh@mrx /]> sudo mount sandman:/export/kerberos /mnt/sandman/kerberos
mount: sandman:/export/kerberos failed, security flavor not supported
[tdh@mrx /]> ./usr/kerberos/bin/klist
Ticket cache: FILE:/tmp/krb5cc_1066
Default principal: tdh@INTERNAL.EXCFB.COM

Valid starting     Expires            Service principal
02/12/07 01:01:42  02/12/07 09:01:42  krbtgt/INTERNAL.EXCFB.COM@INTERNAL.EXCFB.COM
        renew until 02/13/07 00:59:17



Kerberos 4 ticket cache: /tmp/tkt1066
klist: You have no tickets cached

What is up here?

# snoop -x 0,2000 -o /tmp/m2s.snoop sandman mrx
Using device /dev/hme (promiscuous mode)
33 ^C

Note: I used -x 0,2000 to get payload data. I knew I would want to look at most of the packet.

And

[tdh@mrx ~]> sudo mount -t nfs4 sandman:/export/kerberos /mnt/sandman/kerberos
mount.nfs4: Operation not permitted

 26   0.00034 mrx.internal.excfb.com -> sandman      NFS C 4 () PUTFH FH=324D LOOKUP export GETFH GETATTR 10011a 30a23a
 27   0.00030      sandman -> mrx.internal.excfb.com NFS R 4 () NFS4_OK PUTFH NFS4_OK LOOKUP NFS4_OK GETFH NFS4_OK FH=30E6 GETATTR NFS4_OK
 28   0.00033 mrx.internal.excfb.com -> sandman      NFS C 4 () PUTFH FH=30E6 LOOKUP kerberos GETFH GETATTR 10011a 30a23a
 29   0.00021      sandman -> mrx.internal.excfb.com NFS R 4 () NFS4ERR_WRONGSEC PUTFH NFS4_OK LOOKUP NFS4ERR_WRONGSEC

I popped into wireshark and I found out that mrx is only sending AUTH_SYS and AUTH_NULL.

Note: I used wireshark because it will parse the payload data for me. I didn't want to be doing byte conversions and consulting some specs!

In NetApp Filer, NFSv4, and Linux, we find using -o sec=krb5. We can try that:

[tdh@mrx ~]> sudo mount -t nfs4 -o sec=krb5 sandman:/export/kerberos /mnt/sandman/kerberos
Warning: rpc.gssd appears not to be running.
mount.nfs4: Invalid argument

Which is strange, since it is running:

[tdh@mrx ~]> sudo chkconfig --list | grep rpcgssd
rpcgssd         0:off   1:off   2:off   3:on    4:on    5:on    6:off
[tdh@mrx ~]> sudo chkconfig --list | grep rpcidmapd
rpcidmapd       0:off   1:off   2:off   3:on    4:on    5:on    6:off

What does the log state:

RPC: Couldn't create auth handle (flavor 390003)

I've copied the stock krb5.conf back and now the diffs are:

[tdh@mrx /etc]> diff krb5.conf stock/krb5.conf
7c7
<  default_realm = INTERNAL.EXCFB.COM
---
>  default_realm = EXAMPLE.COM
14,17c14,17
<  INTERNAL.EXCFB.COM = {
<   kdc = sandman.internal.excfb.com:88
<   admin_server = sandman.internal.excfb.com:749
<   default_domain = internal.excfb.com
---
>  EXAMPLE.COM = {
>   kdc = kerberos.example.com:88
>   admin_server = kerberos.example.com:749
>   default_domain = example.com
21,22c21,22
<  .internal.excfb.com = INTERNAL.EXCFB.COM
<  internal.excfb.com = INTERNAL.EXCFB.COM
---
>  .example.com = EXAMPLE.COM
>  example.com = EXAMPLE.COM

You know what, rpc.gssd is not running!

[tdh@mrx /etc]> ps -ef | grep rpc
rpc       1877     1  0 01:49 ?        00:00:00 portmap
root      1898     1  0 01:49 ?        00:00:00 rpc.statd
root      1931     1  0 01:49 ?        00:00:00 rpc.idmapd
tdh       2697  2519  0 02:04 pts/0    00:00:00 grep rpc

[tdh@mrx /etc]> sudo sh -c "ulimit -c unlimited;/usr/sbin/rpc.gssd -f -vvv"
Using keytab file '/etc/krb5.keytab'
Processing keytab entry for principal 'nfs/mrx.internal.excfb.com@INTERNAL.EXCFB.COM'
We will use this entry (nfs/mrx.internal.excfb.com@INTERNAL.EXCFB.COM)
Processing keytab entry for principal 'host/mrx.internal.excfb.com@INTERNAL.EXCFB.COM'
We will NOT use this entry (host/mrx.internal.excfb.com@INTERNAL.EXCFB.COM)
Using (machine) credentials cache: 'MEMORY:/tmp/krb5cc_machine_INTERNAL.EXCFB.COM'

And I put it in the background. Hmm, why doesn't it like the host entry?

Alright, I went back to why isn't rpc.gssd starting up at boot:

[ -f /etc/sysconfig/nfs ] && . /etc/sysconfig/nfs
[ "${SECURE_NFS}" != "yes" ] && exit 0

# ls -la /etc/sysconfig/nfs
#

Time to create it (look at Learning NFSv4 with Fedora Core 2 (Linux 2.6. 5 kernel))

# This entry should be "yes" if you are using RPCSEC_GSS_KRB5 (auth=krb5,krb5i, or krb5p)
SECURE_NFS="yes"
# This entry sets the number of NFS server processes.  8 is the default
RPCNFSDCOUNT=8

[tdh@mrx sysconfig]> sudo /etc/init.d/rpcgssd start
Starting RPC gssd:                                         [  OK  ]

God I'm totally hacked about this:

[tdh@mrx sysconfig]> sudo mount -o sec=krb5 sandman:/export/kerberos /mnt/sandman/kerberos
[tdh@mrx sysconfig]> ls -la /mnt/sandman/kerberos
total 5
drwxr-xr-x 2 root root   512 Feb 12 00:23 .
drwxr-xr-x 5 root root  4096 Feb 12 00:49 ..
-rw-r--r-- 1 tdh  wheel    0 Feb 12 00:23 see_me