Helen Chao, a colleague who had never really used Linux, asked me to help configure a kernel. I asked why and she said she needed to test RDMA over NFSv4. It turns out that the stock 2.6.25 kernel with Fedora Core 9 already had the support in it. We followed the directions at the nfs-rdma.txt and were not able to get it running.

Helen (a great test engineer) proceeded to investigate from there and couldn't get a simple loopback or NFS mount to succeed.

So I exported the root to all hosts and went to work debugging this issue. A 'rpcinfo -p' on the server showed the expected registered services. The same call from a client failed, but a ping worked:

[th199096@jhereg ~]> rpcinfo -p pnfs-9-30
^C
[th199096@jhereg ~]> rpcinfo -p pnfs-9-30
^C
[th199096@jhereg ~]> sudo mount -o vers=3 pnfs-9-30:/ /mnt
^C
[th199096@jhereg ~]> sudo mount -o vers=3 pnfs-9-30:/ /mnt
nfs mount: pnfs-9-30: : RPC: Rpcbind failure - RPC: Timed out
nfs mount: retrying: /mnt
nfs mount: pnfs-9-30: : RPC: Rpcbind failure - RPC: Timed out
^C
[th199096@jhereg ~]> ping pnfs-9-30
pnfs-9-30 is alive

I thought that perhaps it was a firewall issue and disabled IPTABLES.

No luck and I knew the mount should succeed - I tried it with my home Core 8 box and an OpenSolaris server. It worked, but then again, that Linux box has been configured for ages. Long story short, I asked Chuck Lever for help.

His only suggestion was to turn off selinux or as he puts it:

Also disable selinux, just so your systems behave like normal Unix.

So I followed the directions I found here: How to Disable SELinux and now the mount works:

# mount -o vers=3 pnfs-9-30:/ /mnt
nfs mount: pnfs-9-30: : RPC: Rpcbind failure - RPC: Timed out
nfs mount: retrying: /mnt
nfs mount: pnfs-9-30: : RPC: Rpcbind failure - RPC: Timed out
nfs mount: pnfs-9-30: : RPC: Rpcbind failure - RPC: Timed out
nfs mount: /mnt: mounted OK
# 

Most of the help I found with google on the RPC messages wasn't informative. Either the suggestion was to turn off IPTABLES or there was no reply.