Kshitiz Saxena: Kshitiz's Blog

There are handful of features newly added for Converged Load-Balancer(CLB) in SailFin 2.0. These features are as follows :

1. DCR plugin in JAVA
2. CLB monitoring
   
3. Increase load on recovered instance in periodic manner
4. Reuse TCP connection between client to CLB front-end

DCR plugin in JAVA

In SailFin 1.5, user need to provide DCR plugin in propriety XML format. Developer found it difficult to write DCR as it was hard to understand propriety format. Also this format is very restrictive in nature, and difficult to enhance it to support new rules.

Hence in SailFin 2.0, we have introduced support for DCR plugin in JAVA. Please refer to my blog to get complete details on how to develop DCR plugin in JAVA.

CLB Monitoring

In SailFin 2.0, user can gather CLB monitoring data. CLB exposes both front-end and back-end statistics. Monitoring level is dynamically (re)configurable.

How to enable CLB monitoring

User can enable CLB monitoring by setting property converged-load-balancer under module-monitoring-levels to HIGH or
LOW. User can set it using admin CLI or GUI.

Front-end statistics

Front-end statistics are SIP and HTTP requests or responses
handled by CLB acting as front-end. The exposed statistics are as follows :

• Total number of http requests received
• Total number of http requests received to be serviced locally
• Total number of http requests proxied to remote back-end to be serviced
• Total number of http requests which resulted in error responses
• Total number of failed over http requests
• Total number of sip requests received
• Total number of sip requests received to be serviced locally
• Total number of sip requests proxied to remote back-end to be serviced
• Total number of failed over sip requests
• Total number of sip requests which resulted in error responses
• Total number of outgoing sip requests
• Total number of sip responses received
• Total number of sip responses received to be consumed locally
  
• Total number of sip responses proxied to remote back-end to be consumed
• Total number of sip responses discarded as they cannot be routed to its
  consumer

Back-end statistics

Back-end statistics are SIP and HTTP requests or responses
handled by CLB acting as back-end. The exposed statistics are as follows :

• Total number of incoming sip requests
• Total number of incoming http requests
• Total number of incoming sip responses
• Total number of outgoing sip responses
• Total number of outgoing sip requests
  

Increase load on recovered instance in periodic manner

In SailFin 2.0, new properties are added to increase load on recovered instance in periodic manner. This feature is mainly introduced for Sip Session Replication(SSR).

These new properties are namely load-increase-factor and load-factor-increase-period-in-seconds. These properties can be defined under converged-clb-config and are dynamically (re)configurable. Both of these properties will be used by consistent-hash algorithm in CLB. When an instance recovers, consistent-hash algorithm will not immediately include that instance for load distribution. It will increase load, from 0 to factor of 1, on recovered instance in a periodic manner. The load factor increases by load-increase-factor every load-factor-increase-period-in-seconds seconds.

The default value of load-increase-factor is 1 and load-factor-increase-period-in-seconds is 0. User can set load-increase-factor to any decimal value between 0 and 1, and load-factor-increase-period-in-seconds to positive integer.

User can set using following asadmin commands:

• asadmin set domain.converged-lb-configs.<converged-lb-config-name>.property.load-factor-increase-period-in-seconds=120
• asadmin set domain.converged-lb-configs.<converged-lb-config-name>.property.load-increase-factor=.05

All requests which are distributed using consistent hash algorithm will follow described load-distribution. Consistent hash algorithm is applicable for
1. SIP requests
2. HTTP requests for converged application, only if dcr xml is provided.

This does not impact round-robin algorithm at all. Thus HTTP requests belonging to pure web application(s) will have no impact in behavior what-so-ever.

This will not impact load-distribution in case of cluster startup. It will only come into effect in case of instance startup.

If load-factor-increase-period-in-seconds is set to 0, then load-increase-factor will not have any impact. However if load-factor-increase-period-in-seconds is set and load-increase-factor is not set, then recovered instance will not get any load for period specified by load-factor-increase-period-in-seconds. After period load-factor-increase-period-in-seconds  lapses, then load factor for recovered instance will immediately become 1.

Let me explain how it will work with an example. Lets start with some assumptions :
=> There are two instances, instance1 and instance2, in cluster
=> Total Call Per Second(CPS) on system is 200, so per instance it is 100 CPS
=> load-factor-increase-period-in-seconds is set to 60
=> load-increase-factor is set to .1

Now instance1 goes down and then recovers, then load on instance1 will increase in this manner :
0 - 60 : 0 CPS
60 - 120 : 10 CPS
120 - 180 : 20 CPS
180 - 240 : 30 CPS
240 - 300 : 40 CPS
300 - 360 : 50 CPS
360 - 420 : 60 CPS
420 - 480 : 70 CPS
480 - 540 : 80 CPS
540 - 600 : 90 CPS
600 - onward : 100 CPS

Reuse TCP connection between client to CLB front-end connection

In case client in behind a firewall, then SailFin cannot open new TCP connections to client. In such a scenario, SailFin must reuse existing TCP connections between client and SailFin instance.

New property defined to control this behavior is reuseClientFEConnection. The default value of this property is false. User need to set this property to true to enable this feature. User can set it using following asadmin command:

• asadmin set domain.converged-lb-configs.<converged-lb-config-name>.property.reuseClientFEConnection=true
  

To achieve it, client to CLB Front-End(FE) mapping is maintained with-in CLB. However there are conditions to store client to CLB FE mapping, which are as follows :

• Request has to come over TCP
• Contact is a sip-uri
• Response has a request associated with it, implies SailFin instance is either acting as UAS or B2BUA

A new mapping of client to CLB FE is created for following cases :

• For all Register requests which result in 2xx response, a mapping is created for contact to CLB FE address mapping
• For all dialog creating requests which result in 2xx response, CLB FE address is added as an attribute in session
  

Mapping created by REGISTER requests are cleaned up either by timeout or REGISTER request with expires set to 0.

For all outgoing requests, we check whether we have a mapping for contact address in map created by REGISTER request, or attribute set in session. If either is found, we push a ROUTE header for CLB FE and use that to send request out to client.

In SailFin 1.0, user can provide Data Centric Rules(DCR) as xml file. User found it is difficult to define rules in xml file. Also flexibility was a major concern.

To solve above issues, in SailFinv2 user can provide DCR plugin as a java implementation. User can define rules as per their requirement in a very flexible manner.

This blog provides detail on how to write a DCR plugin in Java.

Functional Specification

For deep-dive design details on DCR plugin as jar file, user can refer to functional specification. It provides introduction to DCR Plugin, configuration and design details.

Developing DCR Plugin jar

Writing DCR plugin implementation

User implementation for DCR plugin must implement interface org.glassfish.comms.api.datacentric.DcrPlugin. A sample implementation of DCR plugin is provided below :

package mydcr;

import org.glassfish.comms.api.datacentric.DcrPlugin;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.sip.SipFactory;
import javax.servlet.sip.SipServletRequest;
import javax.servlet.sip.SipURI;
import org.glassfish.comms.api.telurl.TelUrlResolver;
import org.glassfish.comms.api.uriutils.UriTools;

public class MyDcrPluginImpl implements DcrPlugin{

    public String getKey(SipServletRequest request,
            SipFactory sipFactory, UriTools uriTools,
            TelUrlResolver telUrlResolver) {
        String key = null;
        String method = request.getMethod().toUpperCase();
        if(method.equals( "PUBLISH" ) || method.equals( "INVITE" ) || method.equals( "CANCEL" )){
           key = SipURI.class.cast(request.getTo().getURI()).getUser();
        } else if (method.equals( "REGISTER" )
                || method.equals( "SUBSCRIBE" )){
            key = SipURI.class.cast(request.getFrom().getURI()).getUser();
        }
        System.out.println("MY DCR PLUGIN - Key extracted from SIP request : " + key);
        return key;
    }

    public String getKey(HttpServletRequest request,
            SipFactory sipFactory, UriTools uriTools,
            TelUrlResolver telUrlResolver) {
        String key =  request.getParameter( "j_username" );
        System.out.println("MY DCR PLUGIN - Key extracted from HTTP request : " + key);
        return key;
    }

}

Compiling DCR plugin

User need three jars to compile DCR plugin. These jars are ssa-api.jar, javaee.jar and comms-appserv-api.jar. They are available in SailFin installation. Below is command to compile DCR plugin :

javac -classpath <SAILFIN_INSTALL_PATH>/lib/ssa-api.jar:<SAILFIN_INSTALL_PATH>/lib/javaee.jar:<SAILFIN_INSTALL_PATH>/lib/comms-appserv-api.jar -d classes src/mydcr/MyDcrPluginImpl.java

Note: classes directory must exist. Compiled classes will reside in classes directory.

Writing Manifest file

Below is sample manifest file :

Manifest-Version: 1.0
Dcr-Plugin-Class: mydcr.MyDcrPluginImpl

Lets call this file as mymanifest

Creating DCR plugin jar

Command to create jar file using above manifest and compiled dcr plugin implementation class is provided below :

jar -cvfm mydcrplugin.jar mymanifest -C classes mydcr/MyDcrPluginImpl.class

Command to set this jar as DCR file

User can set DCR file using asadmin set-dcr-file command. One example of the command is provided below :

<SAILFIN_INSTALL_PATH>/bin/asadmin set-dcr-file --host <host-name> --port <port-number> --clbconfig  <clb-config-name> mydcrplugin.jar

User can set dcr file using admin GUI as well.

In this blog we will try and provide references to all information available for GlassFish load-balancer plugin. We will put in continuous efforts to keep this blog up to date.

Contents

• Latest version
• Official documentation
• Blogs
• Bugs fixed in a load-balancer plugin version

Latest Version

The latest version of load-balancer plugin is aslb-9.1.1-b11. User can download it from following download locations :

• Solaris-SPARC
• Solaris-x86
• Linux
• Windows
  

Official documentation

Latest documentation for load-balancer plugin is available under Sun GlassFish Enterprise Server 2.1's High Availability Administration Guide. The relevant information is in two chapters :

• Chapter 4 : Configuring HTTP Load Balancing
• Chapter 5 : Configuring Web Servers for HTTP Load Balancing

There is also an official white paper available on GlassFish load-balancer plugin.

Blogs

There are multiple blogs written on how to configure load-balancer plugin with various web-servers. Below are links for them :

• Sun Java System Web Server 7.x (SSL Setup)
• Apache 2.2
• IIS

Bugs fixed in a load-balancer plugin

• aslb- 9.1.1-b11.jar
• 7131
  
• 8449
• BT 6197760
• BT 6847314
• BT 6856096
• BT 6855712

Platform supported

• Windows 2003 Enterprise Edition

Currently the load-balancer plugin is well tested on above mentioned platform. It may work on other windows platform, even though it is not tested on those platform.

IIS Version supported

• IIS 6.0
• IIS 5.0

These are tested version of IIS with load-balancer plugin

Download Location

The load-balancer plugin library can be downloaded from http://download.java.net/javaee5/external/WINNT/aslb/jars

The latest version of load-balancer plugin available is aslb-9.1.1-b7.jar. User can download any other latest version compared to above version, if available.

Steps to configure load-balancer plugin on IIS

Let me first put down steps which is automatically taken care of by installer when using Sun GlassFish Enterprise Server :

1. User has already installed IIS and directory C:\Inetpub\wwwroot(or equivalent) exists. Now create a directory sun-passthrough under C:\Inetpub\wwwroot. From now on directory C:\Inetpub\wwwroot\sun-passthrough will be referred to as <sun-passthrough-dir>.
2. Download load-balancer plugin and unjar it. Then unzip SUNWaspx.zip and SUNWaslb.zip. This directory will be referred to as <lbplugin-unzip-dir>.
3. Create directory
1. <sun-passthrough-dir>\errorpages
2. <sun-passthrough-dir>\resource
3. <sun-passthrough-dir>\sec_db_files
4. Copy file - <lbplugin-unzip-dir>\lib\webserver-plugin\windows\iis\sun-passthrough.dll to <sun-passthrough-dir>. Also change permission of the file to executable for IIS run-as user(IIS_WPG).
   
5. Copy file - <lbplugin-unzip-dir>\lib\webserver-plugin\windows\iis\sun-passthrough.properties to <sun-passthrough-dir>. Append following entries to the file :
   
1. lb-config-file=<sun-passthrough-dir>\loadbalancer.xml
2. log-file=<sun-passthrough-dir>\lb.log
3. log-level=INFO
   
6. Copy files - <lbplugin-unzip-dir>\lib\webserver-plugin\windows\iis\errorpages\default-error.html and <lbplugin-unzip-dir>\lib\webserver-plugin\windows\iis\errorpages\sun-http-lberror.html to <sun-passthrough-dir>\errorpages directory.
7. Copy files - <lbplugin-unzip-dir>\lib\webserver-plugin\windows\iis\LBPluginDefault_root.res and <lbplugin-unzip-dir>\lib\webserver-plugin\windows\iis\LBPlugin_root.res to <sun-passthrough-dir>\resource directory.
   
8. Copy files - <lbplugin-unzip-dir>\lib\webserver-plugin\windows\iis\secmod.db, <lbplugin-unzip-dir>\lib\webserver-plugin\windows\iis\cert8.db and <lbplugin-unzip-dir>\lib\webserver-plugin\windows\iis\key3.db to <sun-passthrough-dir>\sec_db_files directory.
   
9. Copy file <lbplugin-unzip-dir>\lib\install\templates\loadbalancer.xml.example to <sun-passthrough-dir>\loadbalancer.xml directory.
   
10. Copy files - <lbplugin-unzip-dir>\lib\dtds\sun-loadbalancer_1_1.dtd and <lbplugin-unzip-dir>\lib\dtds\sun-loadbalancer_1_2.dtd to <sun-passthrough-dir> directory.

Manual steps that are needed even when using installer are well documented and are available at http://docs.sun.com/app/docs/doc/819-4740/abdhq?a=view
Note: In step 6, use <lbplugin-unzip-dir>\lib instead of <application-server-install-dir\bin and <application-server-install-dir\lib as suggested in linked documentation.

User need to edit load-balancer xml(<sun-passthrough-dir>\loadbalancer.xml) to reflect their cluster configuration. User can export load-balancer xml in DAS and then copy it to above mentioned location using following instructions.

Pankaj has written a whitepaper on Converged Load-Balancer(CLB). This whitepaper provides following details :

1. Overview of CLB
2. Deployment Topology
3. Load-balancing algorithms
4. Functional details of CLB
5. Cluster health monitoring

Support for Apache 2.2 will be coming in GlassFish Server v2.1. Users can download this version of GlassFish server from GlassFish download page. The enterprise edition will even contain installer support for installing load-balancer plugin on Apache 2.2.

For the benefit  of GlassFish user, steps to manually configure load-balancer plugin with Apache2.2 is provided in this blog.

 Platform supported

• Solaris SPARC
• Solaris x86
• Linux

For exact version of the platform supported, please refer to release notes of GlassFish Server 2.1

NOTE : Apache 2.2 in not supported on windows right now. However Apache 2.0.x is supported on windows.  

Download Location

The load-balancer plugin library can be downloaded from http://download.java.net/javaee5/external/<OS>/aslb/jars where OS can be SunOS, SunOS_X86 and Linux.

The latest version of load-balancer plugin available is aslb-9.1.1-b5.jar. User can download any other latest version compared to above version, if available.

Steps to configure load-balancer plugin on Apache2.2

1. Build Apache 2.2 and install it. If user wants to use auto-apply feature, then it should be build with openssl and server certificate must be installed. You can refer to Apache2.0 documentation for building Apache2.2 with ssl and installing certificate. Use --with-included-apr option when building Apache 2.2 to build bundled apr. Apache 2.2 install directory will be referred to as <apache2.2-install-dir>.

2. Download load-balancer plugin and unjar it. Then unzip SUNWaspx.zip and SUNWaslb.zip. This directory will be referred to as <lbplugin-unzip-dir>.

3. Create directory

1. <apache2.2-install-dir>/modules/errorpages
2. <apache2.2-install-dir>/modules/resource
3. <apache2.2-install-dir>/sec_db_files

4. Copy file - <lbplugin-unzip-dir>/lib/webserver-plugin/<OS>/apache2.2/mod_loadbalancer.so to <apache2.2-install-dir>/modules. Also change permission of the file to executable.

5. Copy files - <lbplugin-unzip-dir>/lib/webserver-plugin/<OS>/apache2.2/errorpages/default-error.html and <lbplugin-unzip-dir>/lib/webserver-plugin/<OS>/apache2.2/errorpages/sun-http-lberror.html to <apache2.2-install-dir>/modules/errorpages directory.
   

6. Copy files - <lbplugin-unzip-dir>/lib/webserver-plugin/<OS>/apache2.2/LBPluginDefault_root.res and <lbplugin-unzip-dir>/lib/webserver-plugin/<OS>/apache2.2/LBPlugin_root.res to <apache2.2-install-dir>/modules/resource directory.
   

7. Copy files - <lbplugin-unzip-dir>/lib/webserver-plugin/<OS>/apache2.2/secmod.db, <lbplugin-unzip-dir>/lib/webserver-plugin/<OS>/apache2.2/cert8.db and <lbplugin-unzip-dir>/lib/webserver-plugin/<OS>/apache2.2/key3.db to <apache2.2-install-dir>/sec_db_files directory.
   

8. Copy file <lbplugin-unzip-dir>/lib/install/templates/loadbalancer.xml.example to <apache2.2-install-dir>/conf directory.
   

9. Copy files - <lbplugin-unzip-dir>/lib/dtds/sun-loadbalancer_1_1.dtd and <lbplugin-unzip-dir>/lib/dtds/sun-loadbalancer_1_2.dtd to <apache2.2-install-dir>/conf directory.
   

10. Changes in file <apache2.2-install-dir>/conf/httpd.conf

1. Change
   FROM:
   ServerName www.example.com:80
   TO:
   ServerName <host-name>:80

2. Change
   FROM:
   #Include conf/extra/httpd-mpm.conf
   TO:
   Include conf/extra/httpd-mpm.conf

3. Change
   FROM:
   #Include conf/extra/httpd-vhosts.conf
   TO:
   Include conf/extra/httpd-vhosts.conf

4. Change
   FROM:
   #Include conf/extra/httpd-ssl.conf
   TO:
   Include conf/extra/httpd-ssl.conf

5. Append following entry
   ##BEGIN EE LB Plugin Parameters
   LoadFile /usr/lib/libCstd.so.1 (THIS LINE IS ONLY NEEDED ON SOLARIS SPARC)
   LoadModule apachelbplugin_module modules/mod_loadbalancer.so
   #AddModule apachelbplugin_module
   <IfModule apachelbplugin_module>
   config-file <apache2.2-install-dir>/conf/loadbalancer.xml
   locale en
   </IfModule>
   ##END EE LB Plugin Parameters

11. Changes in file <apache2.2-install-dir>/conf/extra/httpd-mpm.conf

    1. Change prefork MPM settings
       Set StartServers to 1
       Set MaxClients to 1

12. Changes in file <apache2.2-install-dir>/conf/extra/httpd-vhosts.conf

    1. Append
       ##BEGIN EE LB Plugin Parameters
       <VirtualHost <machine-ip-address>>
           ServerName <host-name>
           DocumentRoot "<apache2.2-install-dir>/htdocs"
       </VirtualHost>
       ##END EE LB Plugin Parameters

    2. User can remove other existing VirtualHost entries from this file

13. Changes in file <apache2.2-install-dir>/conf/extra/httpd-ssl.conf

    1. Change
       FROM:
       <VirtualHost _default_:443>
       TO:
       <VirtualHost <machine-ip-address>:443>
       b)Change
       FROM:
       ServerName www.example.com:443
       TO:
       ServerName <host-name>:443

14. If user want to use feature auto-apply feature

    1. Export DAS certificate using command : certutil -L -d <application-server-DAS-dir>/config -n s1as -a -o sjsas.crt

    2. Copy certificate file(sjsas.crt) to <apache2.2-install-dir>/conf/ssl.crt directory. If directory does not exist, create one.
       

    3. Append in file <apache2.2-install-dir>/conf/extra/httpd-ssl.conf
       ##BEGIN EE LB Plugin Parameters
       <Location /lbconfigupdate >
       SSLVerifyClient require
       SSLVerifyDepth 1
       SSLRequireSSL
       SSLCACertificateFile <apache2.2-install-dir>/conf/ssl.crt/sjsas.crt
       SSLRequire ( %{SSL_CIPHER} !~ m/^(EXP|NULL)-/ \
       and %{SSL_CLIENT_S_DN_O} eq "Sun Microsystems" \
       and %{SSL_CLIENT_S_DN_OU} eq "Sun Java System Application Server" \
       and %{SSL_CLIENT_M_SERIAL} eq "<Cert-Serial-Number>" )
       </Location>
       <Location /getmonitordata >
       SSLVerifyClient require
       SSLVerifyDepth 1
       SSLRequireSSL
       SSLCACertificateFile  <apache2.2-install-dir>/conf/ssl.crt/sjsas.crt
       SSLRequire ( %{SSL_CIPHER} !~ m/^(EXP|NULL)-/ \
       and %{SSL_CLIENT_S_DN_O} eq "Sun Microsystems" \
       and %{SSL_CLIENT_S_DN_OU} eq "Sun Java System Application Server" \
       and %{SSL_CLIENT_M_SERIAL} eq "<Cert-Serial-Number>" )
       </Location>
       ##END EE LB Plugin Parameters
       
       NOTE:
       

       1. Cert-Serial-Number need to be generated from provided DAS certificate file(sjsas.crt).  Command for that is : keytool -printcert -file sjsas.crt. Note Serial number from output of above command. Take its value and change all lower case characters to upper case.
       2. Application Server name is assumed to be Sun Java System Application Server. If it is different, then it needs to be reflected here as well. Above command will print that as well.
          

15. Changes in file <apache2.2-install-dir>/bin/envvars

    1. Suffix <lbplugin-unzip-dir>/lib and <apache2.2-install-dir>/modules directory to LD_LIBRARY_PATH on unix platform

16. For auto-apply feature to work, the run-as user for apache must have writable permission for <apache2.2-install-dir>/conf directory. If security is not an issue, user can even set <apache2.2-install-dir>/conf directory permission to 777.

Are you looking for load-balancer to handle SIP requests ? If yes, you have come to right place .

Brief history : Sun Communication Application Server, aka sailfin, is built over glassfish. Glassfish is  Java EE 5 based application server. And sailfin provides SIP Servlet Container based on JSR 289 over and above what is offered by Glassfish.

Sailfin not only provide containers to deploy SIP and JAVA EE 5 based applications. It is complete package with high-availability features like load-balancing and session-replication required by enterprise systems.

 To learn more about load-balancer available with sailfin, please refer to my blogs on Converged Load Balancer.

Does your application suffer from connection starvation? Worked long and hard on your application  to get rid of such issues but in vain . Ever wondered what is eating up all the connections. Unable to figure out whether it is your application or someone-else's . Now you have an option in glassfish v2 to trace out all such cases of connection leaks . The new connection leak tracing feature not only provides information of connection leaks, it also allows a user to reclaim leaking connections to make it available for other applications.

Now on a more technical front a brief description of the feature is provided, and how to configure, monitor and view the connection leaks.

Brief Description

How to Configure

Using GUI

1. Browse to the pool. This can be jdbc(or connector) connection pool.
2. Click on Advanced Tab of the pool
3. Provide appropriate inputs Provide non-zero value for Leak Timeout to enable connection leak tracing Click the check-box of Leak Reclaim if you want connection reclaim to be enabled
4. Save the changes

Using CLI

Monitoring stats for connection leak tracing

• JDBC connection pool, set monitoring level for JDBC connection pool component to HIGH.
  
• Connector connection pool, set monitoring level for JMS/Connector Service component to HIGH.

Using GUI

1. Browse to Monitor tab. Now click on Resources tab. Select your connection pool from the drop down box.
2. Scroll down the page and you will find connection leak stats

Using CLI

• For jdbc/connector connection pool
• asadmin> get -m server.resources.<pool-name>.numpotentialconnleak*
• The stats will look like
• server.resources.<pool-name>.numpotentialconnleak-count = 3
• server.resources.<pool-name>.numpotentialconnleak-description = Provides a count value reflecting the number of potential connection leaks
• server.resources.<pool-name>.numpotentialconnleak-lastsampletime = 1175495050536
• server.resources.<pool-name>.numpotentialconnleak-name = NumPotentialConnLeak
• server.resources.<pool-name>.numpotentialconnleak-starttime = 1175494865525
• server.resources.<pool-name>.numpotentialconnleak-unit = Count
  

Viewing potential connection leak traces

1. Open the Log Search page by clicking View Log files button. It will open up a new window.
2. Click Advanced Search on this page
3. Provide inputs Select Log Level : WARNING Select Logger : javax.enterprise.resource.resourceadapter Provide connection pool name in Name-Value Pairs : ConnectionPoolName=<pool-name> Click Search
4. If connection leak records exist, they will be displayed
5. Click on individual record to view the record in detail
6. The complete stack trace is provided in record detail. User can browse through it and find out the application which is causing connection leaks