Some behind the scenes tips for getting a community working In no particular order...
- Do a banner of mds, client, and ds (or ds1, ds2, etc) and place it in /etc/motd for the respective hosts. It will help you more than you think. Especially after a core dump and logging in to the machine.
- The first clue as to whether your community is setup correctly is whether or not you have
an auth record on the mds.
Here is a bad result
mds # echo '::walk Device_entry_cache | ::print struct rfs4_dbe data | ::print mds_device_t' | mdb -k
mds #
And here is a good result:
# echo '::walk Device_entry_cache | ::print struct rfs4_dbe data | ::print mds_device_t' | mdb -k
{
dbe = 0xffffff02fbb21f08
dev_addr = {
na_r_netid = 0xffffff02f6d17b20 "tcp"
na_r_addr = 0xffffff02ec14c000 "10.1.233.119.217.62"
}
dev_flags = 0x3
dev_infop = 0xffffff02fbb22f78
dev_list_next = {
list_next = 0xffffff02fbb22fb8
list_prev = 0xffffff02fbb22fb8
}
}
Unfortunately, many problems have this symptom.
- The only way to tell that your client is talking to the ds is to use snoop. I.e., you
may have the client and mds telling you that pNFS is enabled, but until you see traffic to
the ds, it is not working!
client # mount pnfs-3-14:/ /mnt
Mar 19 11:29:21 pnfs-3-12 nfs: NOTICE: enabling pNFS on pnfs-3-14
client # cd /mnt
client # cd
client # snoop pnfs-3-14 pnfs-3-13
Using device bge0 (promiscuous mode)
^Cclient # cd
I typically open another window and do simple copies from /etc:
[pnfs-3-12 zippy]> cp /etc/motd .
[pnfs-3-12 zippy]> cp /etc/motd kkk
It is only when I see valid traffic that I know I am set.
# snoop pnfs-3-12 pnfs-3-13
Using device bge0 (promiscuous mode)
Mar 19 11:39:07 pnfs-3-12 nfs: NOTICE: enabling pNFS on pnfs-3-14
pnfs-3-12 -> pnfs-3-13 TCP D=55614 S=1014 Ack=93767047 Seq=74796989 Len=0 Win=49640
pnfs-3-12 -> pnfs-3-13 NFS C 4 (exchange_id ) EXCHANGE_ID Verf=11E050B247E141B2 COID=F6C0 PNFS_DS NONE
- Make sure that you share your pool on the DS.
ds # zpool create -f dpool /dev/dsk/c1t0d0s7
ds # dservadm addmds 10.1.233.120.8.1
ds # dservadm addpool dpool
ds # dservadm enable
If you don't, then you won't see traffic going from the client to the DS.
client # snoop pnfs-3-12 pnfs-3-13
Using device bge0 (promiscuous mode)
Mar 19 11:19:26 pnfs-3-12 last message repeated 2 times
So go back and do:
ds # zfs set sharenfs=on dpool
Note you will also not see a record for the auth in the mds:
mds # echo '::walk Device_entry_cache | ::print struct rfs4_dbe data | ::print mds_device_t' | mdb -k
- You need to add the mds auth record every time the mds reboots.
- Make sure that you disable the ds before you enable the mds:
ds # dservadm disable
And
# mdsadm -o add -t auth -a ip=10.1.233.119
adding: IP Addr - 10.1.233.119
ds # dservadm enable
- You might see two auth records if the ds is not disabled:
mds # mdsadm -o add -t auth -a ip=10.1.233.119
adding: IP Addr - 10.1.233.119
mds # echo '::walk Device_entry_cache | ::print struct rfs4_dbe data | ::print mds_device_t' | mdb -k
Whoops, forgot to disable the ds!
ds # dservadm disable
Still nothing!
mds # echo '::walk Device_entry_cache | ::print struct rfs4_dbe data | ::print mds_device_t' | mdb -k
Okay, issues again:
mds # mdsadm -o add -t auth -a ip=10.1.233.119
adding: IP Addr - 10.1.233.119
nfssys:: File exists
mds # echo '::walk Device_entry_cache | ::print struct rfs4_dbe data | ::print mds_device_t' | mdb -k
And finally:
ds # dservadm enable
Which yields:
# echo '::walk Device_entry_cache | ::print struct rfs4_dbe data | ::print mds_device_t' | mdb -k
{
dbe = 0xffffff02f4493f08
dev_addr = {
na_r_netid = 0xffffff02f76f28e0 "tcp"
na_r_addr = 0xffffff02f51ec820 "10.1.233.119.254.189"
}
dev_flags = 0x3
dev_infop = 0xffffff02f4494f78
dev_list_next = {
list_next = 0xffffff02f4494fb8
list_prev = 0xffffff02f4494fb8
}
}
Hmm, I wasn't able to reproduce the double record thing... Oh yes I was!
ds # dservadm disable
ds # dservadm enable
And
mds # echo '::walk Device_entry_cache | ::print struct rfs4_dbe data | ::print mds_device_t' | mdb -k
{
dbe = 0xffffff02f4493e18
dev_addr = {
na_r_netid = 0xffffff02f76f2fc0 "tcp"
na_r_addr = 0xffffff02f51ecb20 "10.1.233.119.212.217"
}
dev_flags = 0x3
dev_infop = 0xffffff02f4494f78
dev_list_next = {
list_next = 0xffffff02f4494fb8
list_prev = 0xffffff02f4494fb8
}
}
{
dbe = 0xffffff02f4493f08
dev_addr = {
na_r_netid = 0xffffff02f76f28e0 "tcp"
na_r_addr = 0xffffff02f51ec820 "10.1.233.119.254.189"
}
dev_flags = 0x3
dev_infop = 0xffffff02f4494f78
dev_list_next = {
list_next = 0
list_prev = 0
}
}
The point is with multiple auth records you may not get the results
you want. I'd reboot them both at this point. :->
- When testing connectathon as root, make sure your share on the mds has either root= or anon=0 set:
client # ./server -p /nippy -m /mnt/pnfs-3-14 pnfs-3-14
Start tests on path /mnt/pnfs-3-14/pnfs-3-12.test [y/n]? y
sh ./runtests -a -t /mnt/pnfs-3-14/pnfs-3-12.test
Starting BASIC tests: test directory /mnt/pnfs-3-14/pnfs-3-12.test (arg: -t)
mkdir: Failed to make directory "/mnt/pnfs-3-14/pnfs-3-12.test"; Permission denied
Can't make directory /mnt/pnfs-3-14/pnfs-3-12.test
basic tests failed
Tests failed, leaving /mnt/pnfs-3-14 mounted
So:
mds #share
-@nippy /nippy rw ""
mds # zfs set sharenfs=rw,anon=0 nippy
