Kshitiz Saxena

Main | Converged Load-Balan... »

http://blogs.sun.com/kshitiz/date/20070410

Tuesday April 10, 2007

Connection Leak Tracing

Does your application suffer from connection starvation? Worked long and hard on your application to get rid of such issues but in vain

. Ever wondered what is eating up all the connections. Unable to figure out whether it is your application or someone-else's

. Now you have an option in glassfish v2 to trace out all such cases of connection leaks

. The new connection leak tracing feature not only provides information of connection leaks, it also allows a user to reclaim leaking connections to make it available for other applications.

Now on a more technical front a brief description of the feature is provided, and how to configure, monitor and view the connection leaks.

Brief Description

Lets start with the definition of connection leak. As of now the definition is kept really simple. An application is said to be leaking connection, if it acquires a connection and does not close it within specified time period. If this feature is enabled, the application server detects these potential connection leaks and dumps the leak tracing logs to server logs. Looking at the logs the user can figure out which application is leaking connection and fix the issues with application, if exists.

An additional feature of connection reclaim is also provided with connection leak tracing. This can only be enabled when connection leak tracing is turned on. If enabled, the leaking connection with be reclaimed by application server and made available to other applications accessing the connection pool. This feature should be used responsibly. The feature should not be enabled if user's application caches connection. In such cases, enabling this feature will result in unexpected behavior.

For more detailed description, please refer to the one pager.

This feature is not enabled by default. The user need to configure his/her connection pools in order to enable connection leak tracing. The steps to achieve this is provided in the next section.

How to Configure

The connection leak tracing is an advanced feature. Additional steps are needed to configure the connection leak tracing after creating a connection pool. These settings are picked up by application server dynamically. Hence no server restart is needed.

Using GUI

1. Browse to the pool. This can be jdbc(or connector) connection pool.
2. Click on Advanced Tab of the pool
3. Provide appropriate inputs Provide non-zero value for Leak Timeout to enable connection leak tracing Click the check-box of Leak Reclaim if you want connection reclaim to be enabled
4. Save the changes

Using CLI

1. Set a non-zero connection leak timeout value for connection pool

For jdbc connection pool

asadmin>set domain.resources.jdbc-connection-pool.<pool-name>.connection_leak_timeout_in_seconds = 300

For connector connection pool

asadmin>set domain.resources.connector-connection-pool.<pool-name>.connection_leak_timeout_in_seconds = 300

2. Enable connection leak reclaim if desired

For jdbc connection pool

asadmin>set domain.resources.jdbc-connection-pool.<pool-name>.connection-leak-reclaim = true

For connector connection pool

asadmin>set domain.resources.connector-connection-pool.<pool-name>.connection-leak-reclaim = true

Monitoring stats for connection leak tracing

The connection leak tracing data is available to the user when monitoring is enabled for that component. To enable monitoring please refer to monitoring manual. For getting connection leak stats of :

JDBC connection pool, set monitoring level for JDBC connection pool component to HIGH.
Connector connection pool, set monitoring level for JMS/Connector Service component to HIGH.

After configuring the monitoring level for component, the monitoring data can be viewed using GUI as well as CLI.

Using GUI

1. Browse to Monitor tab. Now click on Resources tab. Select your connection pool from the drop down box.
2. Scroll down the page and you will find connection leak stats

Using CLI

The command to get monitoring stats related to connection leak using CLI is:

For jdbc/connector connection pool

asadmin> get -m server.resources.<pool-name>.numpotentialconnleak*

The stats will look like

server.resources.<pool-name>.numpotentialconnleak-count = 3
server.resources.<pool-name>.numpotentialconnleak-description = Provides a count value reflecting the number of potential connection leaks
server.resources.<pool-name>.numpotentialconnleak-lastsampletime = 1175495050536
server.resources.<pool-name>.numpotentialconnleak-name = NumPotentialConnLeak
server.resources.<pool-name>.numpotentialconnleak-starttime = 1175494865525
server.resources.<pool-name>.numpotentialconnleak-unit = Count

Viewing potential connection leak traces

The connection leak traces are dumped to server log with WARNING as log level. So, the log level of Connector component should be at least WARNING. This is default log level for out of the box installation. To enable logging, please refer to logging manual.

It is cumbersome to search through server logs manually. A utility is already provided in application server GUI to search through logs. The steps are provided below to get connection leak traces using this utility.

1. Open the Log Search page by clicking View Log files button. It will open up a new window.
2. Click Advanced Search on this page
3. Provide inputs Select Log Level : WARNING Select Logger : javax.enterprise.resource.resourceadapter Provide connection pool name in Name-Value Pairs : ConnectionPoolName=<pool-name> Click Search
4. If connection leak records exist, they will be displayed
5. Click on individual record to view the record in detail
6. The complete stack trace is provided in record detail. User can browse through it and find out the application which is causing connection leaks