Marcos Ares Troubleshooting Weblog

Problem description

The Sun Premium Plus Software Support team (P+) detects a huge amount of opened "ESTABLISHED" connections in the DS6.3 masters inside the topology and an immediate analysis is opened to study the nature of such opened connections, some of them dating from 5-6 months ago.

Analysis

Analysis shows that the connections that remain opened are connections which were reseted from the client application side without an UNBIND operation. Further investigation shows the load balancers in front of the replicas are responsible for such improper closing. Nevertheless, such connections should still be closed on the DS side thanks to one of the two available mechanisms:

• at the TCP level (via the TCP SO_KEEPALIVE socket settings) => unfortunately, this is not happenning due to bug #5087249, a very old bug which affects all of the 5.x and 6.x series for which DS does not set the SO_KEEPALIVE socket option when opening connections. By not setting the socket option, the DS process is not activating the TCP keepalive mechanism and hence the operating system / TCP stack will not be made responsible of any connection cleaning procedures
  
• or at the DS application level (via the nsslapd-idletimeout setting) => unfortunately, this is not happening either because of the fact that the nsslapd-idletimeout attribute's value is set to "0" (i.e., infinite)

Resolution

P+ proposes to set nsslapd-idletimeout to 30 minutes instead of infinite. A nightly task is scheduled for June 8th 2009 to execute the following action without the need to restart the replicas:

dsconf set-server-prop -p 1389 -e -D "cn=Directory Manager" -w /export/home/dsuser/Documents/password.txt idle-timeout:1800

The task was succesfully executed at 01:30am, with the following instant evolution on "established" connections immediately after:

[09/06/2009 01:36:01] INSTANCE rep1     LDAP: currentconnections: 7383     NETSTAT: 7382
[09/06/2009 01:36:01] INSTANCE rep2     LDAP: currentconnections: 4331     NETSTAT: 4330
...
[09/06/2009 01:48:11] INSTANCE rep1     LDAP: currentconnections: 37     NETSTAT: 1600
[09/06/2009 01:48:11] INSTANCE rep2     LDAP: currentconnections: 16     NETSTAT: 526
...
[09/06/2009 02:00:44] INSTANCE rep1     LDAP: currentconnections: 37     NETSTAT: 36
[09/06/2009 02:00:44] INSTANCE rep2     LDAP: currentconnections: 16     NETSTAT: 14