In no particular order...

• Do a banner of mds, client, and ds (or ds1, ds2, etc) and place it in /etc/motd for the respective hosts. It will help you more than you think. Especially after a core dump and logging in to the machine.
• The first clue as to whether your community is setup correctly is whether or not you have an auth record on the mds.

  Here is a bad result

  mds # echo '::walk Device_entry_cache | ::print struct rfs4_dbe data | ::print mds_device_t' | mdb -k
  mds #
  

  And here is a good result:

  # echo '::walk Device_entry_cache | ::print struct rfs4_dbe data | ::print mds_device_t' | mdb -k
  {
      dbe = 0xffffff02fbb21f08
      dev_addr = {
          na_r_netid = 0xffffff02f6d17b20 "tcp"
          na_r_addr = 0xffffff02ec14c000 "10.1.233.119.217.62"
      }
      dev_flags = 0x3
      dev_infop = 0xffffff02fbb22f78
      dev_list_next = {
          list_next = 0xffffff02fbb22fb8
          list_prev = 0xffffff02fbb22fb8
      }
  }
  

  Unfortunately, many problems have this symptom.

• The only way to tell that your client is talking to the ds is to use snoop. I.e., you may have the client and mds telling you that pNFS is enabled, but until you see traffic to the ds, it is not working!

  client # mount pnfs-3-14:/ /mnt
  Mar 19 11:29:21 pnfs-3-12 nfs: NOTICE: enabling pNFS on pnfs-3-14
  client # cd /mnt
  client # cd
  client # snoop pnfs-3-14 pnfs-3-13
  Using device bge0 (promiscuous mode)
  ^Cclient # cd  
  

  I typically open another window and do simple copies from /etc:

  [pnfs-3-12 zippy]> cp /etc/motd .
  [pnfs-3-12 zippy]> cp /etc/motd kkk
  

  It is only when I see valid traffic that I know I am set.

  # snoop pnfs-3-12 pnfs-3-13
  Using device bge0 (promiscuous mode)
  Mar 19 11:39:07 pnfs-3-12 nfs: NOTICE: enabling pNFS on pnfs-3-14
     pnfs-3-12 -> pnfs-3-13    TCP D=55614 S=1014 Ack=93767047 Seq=74796989 Len=0 Win=49640
     pnfs-3-12 -> pnfs-3-13    NFS C 4 (exchange_id ) EXCHANGE_ID Verf=11E050B247E141B2 COID=F6C0 PNFS_DS  NONE 
  

• Make sure that you share your pool on the DS.

  ds # zpool create -f dpool /dev/dsk/c1t0d0s7
  ds # dservadm addmds 10.1.233.120.8.1
  ds # dservadm addpool dpool
  ds # dservadm enable
  

  If you don't, then you won't see traffic going from the client to the DS.

  client # snoop pnfs-3-12 pnfs-3-13
  Using device bge0 (promiscuous mode)
  Mar 19 11:19:26 pnfs-3-12 last message repeated 2 times
  

  So go back and do:

  ds # zfs set sharenfs=on dpool
  

  Note you will also not see a record for the auth in the mds:

  mds # echo '::walk Device_entry_cache | ::print struct rfs4_dbe data | ::print mds_device_t' | mdb -k
  

• You need to add the mds auth record every time the mds reboots.
  • Make sure that you disable the ds before you enable the mds:

    ds # dservadm disable
    

    And

    # mdsadm -o add -t auth -a ip=10.1.233.119
    adding: IP Addr - 10.1.233.119
    

    ds # dservadm enable
    

  • You might see two auth records if the ds is not disabled:

    mds # mdsadm -o add -t auth -a ip=10.1.233.119
    adding: IP Addr - 10.1.233.119
    mds # echo '::walk Device_entry_cache | ::print struct rfs4_dbe data | ::print mds_device_t' | mdb -k
    

    Whoops, forgot to disable the ds!

    ds # dservadm disable
    

    Still nothing!

    mds # echo '::walk Device_entry_cache | ::print struct rfs4_dbe data | ::print mds_device_t' | mdb -k
    

    Okay, issues again:

    mds # mdsadm -o add -t auth -a ip=10.1.233.119
    adding: IP Addr - 10.1.233.119
    nfssys:: File exists
    mds # echo '::walk Device_entry_cache | ::print struct rfs4_dbe data | ::print mds_device_t' | mdb -k
    

    And finally:

    ds # dservadm enable
    

    Which yields:

    # echo '::walk Device_entry_cache | ::print struct rfs4_dbe data | ::print mds_device_t' | mdb -k
    {
        dbe = 0xffffff02f4493f08
        dev_addr = {
            na_r_netid = 0xffffff02f76f28e0 "tcp"
            na_r_addr = 0xffffff02f51ec820 "10.1.233.119.254.189"
        }
        dev_flags = 0x3
        dev_infop = 0xffffff02f4494f78
        dev_list_next = {
            list_next = 0xffffff02f4494fb8
            list_prev = 0xffffff02f4494fb8
        }
    }
    

    Hmm, I wasn't able to reproduce the double record thing... Oh yes I was!

    ds # dservadm disable
    ds # dservadm enable
    

    And

    mds # echo '::walk Device_entry_cache | ::print struct rfs4_dbe data | ::print mds_device_t' | mdb -k
    {
        dbe = 0xffffff02f4493e18
        dev_addr = {
            na_r_netid = 0xffffff02f76f2fc0 "tcp"
            na_r_addr = 0xffffff02f51ecb20 "10.1.233.119.212.217"
        }
        dev_flags = 0x3
        dev_infop = 0xffffff02f4494f78
        dev_list_next = {
            list_next = 0xffffff02f4494fb8
            list_prev = 0xffffff02f4494fb8
        }
    }
    {
        dbe = 0xffffff02f4493f08
        dev_addr = {
            na_r_netid = 0xffffff02f76f28e0 "tcp"
            na_r_addr = 0xffffff02f51ec820 "10.1.233.119.254.189"
        }
        dev_flags = 0x3
        dev_infop = 0xffffff02f4494f78
        dev_list_next = {
            list_next = 0
            list_prev = 0
        }
    }
    

    The point is with multiple auth records you may not get the results you want. I'd reboot them both at this point. :->

• When testing connectathon as root, make sure your share on the mds has either root= or anon=0 set:

  client # ./server -p /nippy -m /mnt/pnfs-3-14 pnfs-3-14
  Start tests on path /mnt/pnfs-3-14/pnfs-3-12.test [y/n]? y
  
  sh ./runtests  -a -t /mnt/pnfs-3-14/pnfs-3-12.test
  
  Starting BASIC tests: test directory /mnt/pnfs-3-14/pnfs-3-12.test (arg: -t)
  mkdir: Failed to make directory "/mnt/pnfs-3-14/pnfs-3-12.test"; Permission denied
  Can't make directory /mnt/pnfs-3-14/pnfs-3-12.test
  basic tests failed
  Tests failed, leaving /mnt/pnfs-3-14 mounted
  

  So:

  mds #share
  -@nippy         /nippy   rw   ""  
  mds # zfs set sharenfs=rw,anon=0 nippy
  

---

Originally posted on Kool Aid Served Daily
Copyright (C) 2008, Kool Aid Served Daily