OpenSolaris 20050909 - Build 22 - NFS port binding default.

In Solaris Nevada build number 22; aka OpenSolaris-20050909 the default for the NFS client is to use a non reserved port for the connection to the server.

If the server should require the use of a reserved port the default may be changed by setting the value of clnt_cots_do_bindresvport (for TCP) to 1 and clnt_clts_do_bindresvport for UDP.

Great, now how exactly do you do that ?

For a quick fix on a running system (as the root user) we can use MDB like this:

# echo "clnt_cots_do_bindresvport/W 1" | mdb -kw 

if this proves to work and you want to make this permanent then we need to add an entry to /etc/system. When adding to /etc/system you will need to know in which module clnt_cots_do_bindresvport resides. Again using mdb we can do this:

# echo "clnt_cots_do_bindresvport::whatis" | mdb -k
7024045c is clnt_cots_do_bindresvport+0 in rpcmod's data segment

so adding the following to /etc/system will set the value on reboot:

set rpcmod:clnt_cots_do_bindresvport=1

See also: This NFS thread

Technorati Tags:
OpenSolaris
Solaris
nfs

[posted with ecto]

Bake off and Connectathons..

Are a good way to shake out those interoperability issues and sometimes it
turns out that those same issues happen on sun-on-sun too! shock horror!

At a recent bake-off we discovered what appeared to be a specific problem
for a Solaris NFS client to HPUX Server ( tracked as bug id 6289595 ).
Within a week or so, Bill Baker discovered the same symptom on his office
solaris box, occurring to one of our home directory servers. A little investigation
with MDB and it was apparent we had reproduced good 'ol 6289595. (why do things
happen in clusters like that ?)

During the AUTH_REFRESH() for RPCSEC the RPC GSS code will hi-jack the CLIENT
handle. For this failure we needed the connection to not be established and the
connection to be marked dead (jim); when the RPC GSS code used the CLIENT handle
and called down 'lower' it was noticed that the connection had been marked dead, and
so we attempted some clean up; unfortunately during the clean up we mistakenly wait
for all outstanding references to be released... not accounting that the remaining
reference was in-fact "us" (the current thread)

More on NFS and Dtrace.

Mike also asked how to figure out what static trace points exist, of course dtrace
will tell us (and make a jolly good pot of tea as well :) )

root@zmaxdp : dtrace -m sdt:nfs* -l
   ID   PROVIDER            MODULE                          FUNCTION NAME
29166        sdt               nfs                validate_idmapping nfs4-acl-nobody
29167        sdt               nfs                    errs_to_action nfs4-expired
29168        sdt               nfs                         nfs4renew nfs4-renew-end
29169        sdt               nfs                         nfs4renew nfs4-renew-start
29170        sdt               nfs                      nfs4_rfscall nfs4-rfscall_debug
29171        sdt               nfs                 nfs_idmap_str_gid nfs4-str-gid
29172        sdt               nfs                 nfs_idmap_str_uid nfs4-str-uid

The other portion of the question of how we figure out what data is
collected at these points is (unfortunately) a little harder to find out
and for now we'll need to refer to the source code.

common_dispatch() in the NFS server and ma Buddies Dtrace and MDB.

Over here Mike Eisler asked a couple of interesting questions with respect to the NFS server and dtrace. I thought i'd expand on Eric's answer. So, Eric's Dtrace script is using the FBT provider which is function boundary tracing. Not the old 'vtrace point' that Mike added back in S8 time (which *I* just ripped out :) ). We can figure out what function calls happen in the NFS Server by using Dtrace like this :-

# dtrace -m nfssrv       
dtrace: description 'nfssrv' matched 1392 probes
CPU     ID                    FUNCTION:NAME
0  36317                    nfs_svc:entry 
0  35475    nfs_srv_set_sc_versions:entry 
0  35476   nfs_srv_set_sc_versions:return 
0  35477          rfs4_server_start:entry 
0  35478         rfs4_server_start:return 
0  36318                   nfs_svc:return 
0  35489               rfs_dispatch:entry 
0  35485            common_dispatch:entry 

That's nice but what are the arguments to common_dispatch() ? -- there are probably other ways to obtain the information (ugh... look at code ? who has time for that) but i like mdb :-

# mdb -k
Loading modules: [ unix krtld genunix specfs dtrace ufs ip sctp usba uhci s1394 random fctl nca lofs nfs audiosup sppp crypto ptm ]
<MDB> common_dispatch::nm
Value      Size       Type  Bind  Other Shndx    Name
0xf763063d|0x00000774|FUNC |LOCL |0x0  |1       |common_dispatch

<MDB> common_dispatch::nm -f ctype
C Type                                            
void (*)(struct svc_req *, SVCXPRT *, rpcvers_t, rpcvers_t, char *, struct rpc_disptable *)

so now we know what the arguments are.. but what on earth is in a svc_req ?

<MDB> ::print -a -t struct svc_req
{

0 rpcprog_t rq_prog 
4 rpcvers_t rq_vers 
8 rpcproc_t rq_proc 
c struct opaque_auth rq_cred {

c enum_t oa_flavor 
10 caddr_t oa_base 
14 uint_t oa_length 

}
18 caddr_t rq_clntcred 
1c SVCXPRT *rq_xprt 

}

Hmmm what could we do with this info ? -- How about counting the number of NFS Program dispatches ?

# dtrace -n fbt:nfssrv:common_dispatch:entry'/args[0]->rq_prog == 100003/ {@nfsc["NFS Dispatch count"]=count()}'
dtrace: description 'fbt:nfssrv:common_dispatch:entry' matched 1 probe
^C
NFS Dispatch count                                             6584