Trond Norbye's Weblog

Socket connection timeout

I have been struggling to get a patch sent to the libmemcached mailing list to work as expected on my system. The first thing I normally do when I receive a new patch is to apply it and run the test suite, and if that doesn't work I normally notifies the author of the patch without digging too deep into the patch. This time I didn't get a test failure caused by one of the assert() statements in the patch, but the test program terminated unexpectedly. In order to give the author some more information on where it failed, I had to at least figure out how the program terminated (calling exit()?? a signal??? etc). truss is a good tool for that:

trond@opensolaris> truss -o /tmp/truss -E ./tests/testapp consistent_not
[... cut ...]
trond@opensolaris> tail -10 /tmp/truss
 0.0000 so_socket(PF_INET, SOCK_STREAM, IPPROTO_IP, "", SOV_DEFAULT) = 9
 0.0000 setsockopt(9, SOL_SOCKET, SO_LINGER, 0xFFBFF570, 8, SOV_DEFAULT) = 0
 0.0000 fcntl(9, F_GETFL)                               = 2
 0.0000 fcntl(9, F_SETFL, FWRITE|FNONBLOCK)             = 0
 0.0001 connect(9, 0x0004D3F0, 16, SOV_DEFAULT)         Err#150 EINPROGRESS
 0.0001 pollsys(0xFFBFF5E8, 1, 0xFFBFF578, 0x00000000)  = 1
 0.0000 fcntl(9, F_GETFL)                               = 130
 0.0000 fcntl(9, F_SETFL, FWRITE)                       = 0
 0.0000 write(9, " g e t   t H m Z u B 7 T".., 107)     Err#32 EPIPE
 0.0000     Received signal #13, SIGPIPE [default]

Hmm.. So we see that the program terminates when it receives the SIGPIPE signal, and we don't have a signal handler for this. If we look in the man page for write, you will see that write will generate SIGPIPE if the stream isn't connected to the peer.

It took me some time to figure out the bug here (I created a scaled down example and posted to a mailing list, and a friendly soul out there pointed me to the error!!). If you look closely in the truss output above, I only call connect once. The libmemcached took it for granted that if poll returned with a POLLOUT event, the connection was established. The correct thing would however be to call connect once more and check the return value (and possibly errno). The connect man page could have spent a paragraph describing this behavior ;-)

To end this blog, I'll create a new version of the connect call taking a timeout value as a parameter:

int my_connect(int s, const struct sockaddr *name, int namelen, int timeout)
{
   /* We may need to toggle on non-blocking mode */
   int flags = fcntl(s, F_GETFL, 0);
   if (flags == -1) {
      flags = 0;
   } else if (!(flags & O_NONBLOCK)) {
      (void)fcntl(s, F_SETFL, flags | O_NONBLOCK);
   }
   
   int ret;
   while (connect(s, name, namelen) < 0) {
      if (errno == EISCONN) {
         ret = 0;
         break;
      } else if (errno == EINPROGRESS || errno == EALREADY) {
         struct pollfd fds[1] = {[0].fd = s,
                                 [0].events = POLLOUT};         
         if (poll(fds, 1, timeout) == 0) {
            /* poll timed out, so let's set that in errno ;-) */
            errno = ETIMEDOUT;
            ret = -1;
            break;
         } else if (fds[0].revents & POLLERR) {
            ret = -1;
            break;
         }
      } else if (errno != EINTR) {
         ret = 1;
         break;
      }
   }
   
   /* Restore the old setting on the socket if we updated it above */
   int error = errno;
   if (!(flags & O_NONBLOCK)) {
      (void)fcntl(s, F_SETFL, flags);
   }
   errno = error;
   return ret;
}

Posted at 09:37PM Mar 01, 2009 by trond in OpenSolaris  |  Comments[0]