Clustering Open ESB/GlassFish ESB Composite Applications (JBI Service Assemblies)

Is it possible to host Open ESB/GlassFish ESB Composite Applications (JBI Service Assemblies) in GlassFish (Sun Application Server) Clusters?

Well, it depends on which JBI components you are using in your composite application. Yes, clustering is possible if you use only the cluster aware subset of JBI components that are currently available in Open ESB/GlassFish ESB.

We have enough tooling available already in GlassFish Application Server for clustering JBI Service Assemblies (Composite Applications). Using the GlassFish Admin Console or the command line asadmin tool, one can deploy, configure, and administer composite applications in clusters.

But in order for the clustering to be fully functional and stable, all the components (Binding Components and Service Engines) that make up the composite application, the JBI runtime, and the underlying GlassFish clustering framework need to work together seamlessly. Not all the JBI components are cluster aware. From the functional standpoint, the implementation is almost complete in JBI runtime and some of the core JBI components, but we still need to iron out some of the issues before this feature can be officially supported.

But this shouldn't stop you from experimenting with the existing capability, should it?

I have tried clustering a simple echo composite app made up of HTTP BC and BPEL SE and found it to be working. I encountered some issues but there are workarounds for them.

Let me show you an installation setup I have created that automatically performs all the cumbersome installation steps for you, including the workaround steps you need to do to overcome some of the current clustering limitations, so you can fast forward directly to playing with the composite application in a GlassFish cluster on your Windows machine.

So, lets get to it, shall we? You should be able to install and complete this demo in about 30 - 60 minutes.

1. Installation

This demo is resource intensive. When the installation is complete and the demo is running (GlassFish DAS, a cluster with two instances, Derby Database Server, Sun Web Server Administrator Server, and a Sun Web Server Instance), it will take about 625 Meg disk space and will use about 750 Meg of memory. So, I highly recommend, you shutdown all the other unnecessary apps before we begin. Rebooting to start with a clean machine would be ideal, but not required.

Download the demo zip archive.

Instead of directing you to different urls at Sun to download various binaries, I have zipped up all of them and created a single download for your convenience. This zip file based installation is for your demo purpose only and all the licensing agreements of the different products in it are still enforced.

Download it form here: gfesbclusterdemo1windows

Extract the zip file under C:\Sun, and you will see everything get extracted under C:\Sun\gfesbcluster

To avoid the manual interaction, some of the config values are hard-coded to assume you are starting under C:\Sun. It also assumes you are accepting the default ports for the servers. However, it is possible to change these values by updating the configuration files before starting the install script. If you must configure different values, follow the instructions under the Configuration section.

Henceforth, this C:\Sun\gfesbcluster dir will be referred to as <gfesbcluster-dir>.

cd <gfesbcluster-dir>\resources\bin

Edit setenv.bat and specify the dir for the JDK_HOME and JAVA_HOME environment variables.

.\install.bat > install.log 2>&1

This will take about 10 minutes to complete. It installs the following:

Installs GlassFish Application Server with cluster profile.
Installs (updates the existing one) JBI Runtime in GlassFish.
Starts GlassFish Application Server and Derby Database Server.
Creates a sample cluster (Cluster1) in GlassFish with two instances (I1 and I2) and starts this cluster.
Creates the necessary resources in GlassFish Application Server for the Composite Application.
Installs the necessary JBI Components (HTTP BC, BPEL SE, and Shared Libraries) in the cluster.
Deploys and configures a sample echo composite application in the cluster.
Installs Sun Webserver and configures it to do round-robin load balancing into the two GlassFish cluster instances.
Starts Sun Web Admin Server and the Web Server Instance.

2. Installation Verification

Now the installation is complete, let's first verify it went well.

2.1 Inspect install.log

View the install.log in a text editor and inspect for any failures. Do case sensitive search for "CLI" to see if there were any errors in running the asadmin commands in GlassFish.

The main install.bat calls various other .bat files that accomplish different steps of the installation. So, if a step fails, you can run the respective uninstall/install bat files to reinstall that particular component.

2.2 Verify BPEL SE

Check the database table of the BPEL SE to verify it was installed correctly onto the cluster nodes.

In the Netbeans of GlassFish ESB, under the services section, create a connection to bpelseDB Derby database of BPEL SE using the following values. Specify "pass2" as the password for user "usr2".

Now lets open the ENGINE table and verify BPEL SE Engine IDs of the two cluster instances I1 and I2 are listed there.

2.3 Verify HTTP BC

In the browser, make sure you can access the wsdls of the composite application directly from both of the cluster instances.

Instance I1: http://localhost:2110/echoService/echoPort/doc/lit/simple?wsdl

Instance I2: http://localhost:2111/echoService/echoPort/doc/lit/simple?wsdl

2.4 Verify Sun Webserver and Round-Robin Scheduling

Access the wsdl using the webserver port and you should see it alternatively (per round-robin load balancing) returning the wsdls from the two cluster instances. By inspecting the port of the soap address in the returned wsdl, you will know which instance's wsdl is being returned.

http://localhost:8889/echoService/echoPort/doc/lit/simple?wsdl

3. Demonstration

When the installation is complete, everything will be up and running including the sample composite application. We just need to send requests and verify they are being processed by the cluster instances.

The sample is a simple echo composite application made up of BPEL SE and HTTP BC. The GlassFish ESB netbeans projects for this composite application are located under <gfesbcluster-dir>\resources\netbeans.

3.1 Sending Requests

For sending the requests, the installation comes with soapui(http://www.soapui.org) projects so you could start testing right away. From soapui, open project <gfesbcluster-dir>\resources\soapui\cluster-loadbalancer-soapui-project.xml. This sends requests to port 8889 where the Sun Webserver Instance is configured to load balance the requests in a round-robin fashion between the GlassFish cluster instances I1 & I2.

The BPEL SE service units running in the cluster instances simply prepends “Echo ” to the received string and send back the response. So, when a request with string “Foo” is sent from soapui, it will receive “Echo Foo” in the response.

3.2 Monitoring

There are two ways we can verify the proper servicing of the requests by the cluster instances:

Inspecting server.log

During installation, the log levels of SOAP Normalizer and SOAP Denormalizer of HTTP BC were set to FINE. So, in the server.log of each cluster instance, you can view the complete messages being processed by the HTTP BC running in that cluster instance.

Inspecting the STATE table of the bpelseDB database.

Throughout the steps described below, we will verify the request processing using this method.

3.3 Steps

Send 5 requests and observe instances I1 and I2 process them in the round-robin fashion.

In Netbeans, using the connection we have created earlier, view the data in the STATE table of bpelseDB. You will notice the requests were processed alternatively between instances I1 and I2.

Stop cluster instance I1

Execute <gfesbcluster-dir>\resources\bin\stop-instance-I1.bat to stop instance I1.

Send 4 more requests and verify all requests were now processed by instance I2

The STATE table now should have 4 more entries that are all processed by I2.

Restart instance I1

Execute <gfesbcluster-dir>\resources\bin\start-instance-I1.bat to restart instance I1.

Send 4 more requests and verify round-robin processing.

Now you should see the new requests were alternatively processed again by I1 and I2.

Stop cluster instance I2

Execute <gfesbcluster-dir>\resources\bin\stop-instance-I2.bat to stop instance I2.

Send 4 more requests.

Now you will see all the requests were processed by instance I1.

Restart instance I2

Execute <gfesbcluster-dir>\resources\bin\start-instance-I2.bat to restart instance I2.

Send 4 more requests.

Now you will notice all the new requests were processed again in the round-robin fashion.

4. Known Limitations

Your windows box must not have a loop back IP address (127.0.0.1), because when a GlassFish cluster is created, an MQ cluster is created behind the scenes and an MQ clustered broker cannot be started if the IP is a loop back IP. The simple way to overcome this issue is to connect your windows box to the net [to get an IP assigned to it] before starting the install.

5. What's Next

Every directory under <gfesbcluster-dir> has a README.txt describing the contents of each dir. Look through these directories to get familiar with the artifacts. You could use the batch scripts located under the <gfesbcluster-dir>\resources\bin\ to perform various tasks. You could customize these scripts to fit your needs.

The netbeans projects of the composite application are available under <gfesbcluster-dir>\resources\netbeans. Once you get the handle on playing with this demo, you can open these projects in netbeans and add your own logic to experiment further. For further study and experimentation refer the Resources section below.

6. Configuration

If you want to install GlassFish and Sun Webserver in a directory other than C:\Sun, you can specify these dirs by updating GF_INSTALL_DIR, WS_INSTALL_DIR environment variables in setenv.bat.

If you change the WS_INSTALL_DIR, remember to update the pointers in the <gfesbcluster-dir>\resources\ws\install.inf before starting the install. You can also set the port values of Webserver in this file.

Bala [aka Anbu]