Spencer's notes on NFS theShepler

Where-'o-where have my ACKs gone. I am using port 623 and my ACKs are no where to be found. My NFS client is attempting to mount and the server is not responding. The server is there and it likes other applications; why not my friendly NFS client. The ACKs have been pulled into a black hole; they are on the network but not in my networking stack.

What is going on?

Someone reported a problem, intermittent of course, where the NFS client would be happy for awhile and then attempt to mount from a server and it would just hang. Nothing from the client; no message, no error, nothing. A little network tracing with snoop showed that the client was in the middle of establishing a TCP connection with the server. However, the server never responded. Must be a problem with the NFS server. Go over to the server and do a little snooping there and WHAT! The server is receiving the SYN and dutifully responding with its ACK. Well, where is it going? Back to the client?

Snoop at the client should be showing me everything that is coming in; used a little Dtrace to poke around to double check that packets were not being dropped further down the stack.... nothing. Someone happened to have a port set up on the switch to which the client was attached and was able to capture the network traffic going to the client's port and the ACKs were in fact making it to the client. How weird.

Along the way, connectivity to the server was tested via other means. Ping, telnet, ssh. Everything else was working except this silly NFS client mount.

Well, along they way, after my mind had blown a fuse as to why the client wasn't seeing a response, I had asked for help from others. Someone more perceptive than I took notice of the port number the NFS client had chosen for its connection attempt. Port 623. A quick google search for "NFS port 623" and this blog entry pops up. Interesting. Intel network interfaces and port 623. Well, this is the source of the problem I have been looking at.

Turns out that NFS client is in fact a Sun Fire V65x which uses an Intel system board and network interface. If one looks closely at the spec sheet for this system, it says "Remote Management: IPMI and DMI compliant Management Service Processor" Well, that's interesting.

Let's go look for port 623 over at IANA and see what they have to say about this port. The assignments says:

asf-rmcp        623/tcp    ASF Remote Management and Control Protocol
asf-rmcp        623/udp    ASF Remote Management and Control Protocol

So it seems that ASF expands to "Alert Standard Forum" and, while I don't know the lineage, ASF has transformed itself into IPMI or "Intelligent Platform Management Interface". In any case, port 623 has been assigned for this purpose and there must be something going on.

A quick check with the user that reported the problem and sure enough, the IPMI feature is configured on the NFS client in question and a quick check with the ipmiadm tool confirms it. So, the ACKs being sent from the server are being sucked into the black hole of the client's IPMI service processor and the client is none the wiser.

So now that we know that the client has been configured to use IPMI and that anything destined for that port on the client will never make it into the client's networking stack, what can be done?

The first thing that can be done is to disable the IPMI feature on the system. In this case, that was not possible; the feature was being used for remote power management. Second would be to convince the client to avoid using that port number. There are two ways of doing that in the Solaris client. The first is to tell the NFS client to avoid using "reserved" or low port numbers. This is done by setting a kernel variable via

/etc/system

with an entry something like this:

set rpcmod:clnt_cots_do_bindresvport=0x0 

Upon reboot, the NFS client will not use reserved ports. Of course, there may be NFS server that require reserved port usage (how lame) and this is not a good workaround.

The second way to avoid using port 623 is to set up a dummy inetd service that will hold onto the port and therefore preventing the NFS client from using it. The way this can be done is to add the following to

/etc/services

rmcp            623/udp
rmcp            623/tcp

and then to place the following in /etc/inetd.conf.

rmcp    dgram   udp     wait    nobody  /bin/false
rmcp    stream  tcp     wait    nobody  /bin/false

After this is done, and inetd is restarted/refreshed. Port 623 on the client will be bound and the NFS client will be able to function once again.

Are there other conditions under which this can happen? Yes, one of the other scenarios is if there is an IPSEC configuration where a port has been configured for restricted access; while the IPSEC configuration is still in place, the application is no longer configured. This allows the NFS client to bind to the IPSEC restricted port and the same behavior ensues.