fintanr's weblog

fintanr's weblog

Twitter Updates

follow me on Twitter

Jeff Bonwick
Matt Ahrens

Dan Price
David Comay

Jonathan Schwartz

blogs.sun.com
Weblog
Login

Technorati Profile

« Previous day (Jan 30, 2005) | Main | Next day (Jan 31, 2005) »

Monday January 31, 2005

Zones, Message Queues and preliminary setup on a benchmark
My commute takes quite a bit of time at the moment, so to occupy myself during said commute, and in between my current vacation periods, I decided to setup Sonic Softwares Sonic MQ messaging benchmark. Its a pretty straight forward benchmark used for benchmarking java messaging, indeed there was quite a stunning result for Sonic MQ running on a SunFire v40z with Solaris 10 (see press release).

Anyway doing the initial setup on something like this on a commute presents a few problems, i.e. how do you do an inital setup of a multitier benchmark without multiple machines? So its zones to the rescue. And seeing as this is a pretty good example of how you could use zones for testing of your multitier application on a single machine I figured I would share it with you.

On a side note, if you want to actually follow what am I doing here on your own Solaris 10 machine, you can download the Sonic MQ benchmark from their website (registration required), and use the Sun Java System Message Queue which comes bundled with Solaris. The subsequent examples all use the Sun Java System Message Queue, and the initial config was actually done on my laptop (hence the iprb0 interface).

Overview

At a very high level the messaging config that we are going to test could be visualized as being similar to the image below.

For the purposes of our example we will just look at the Publish/Subscribe section of the benchmark.

Setting Up The Zones

First off zones, rather than going through the gory detail what zones are you should read David Comays (Zones Architect) blog, Dennis Clarkes excellent introduction to zones on blastwave.org, the zones documentation on docs.sun.com and the BigAdmin zones site.

The zone that we set up here is very simple, as we are only using the network interface, and (as will be shown) the message queue software has been configured to have no logging. I'm going to go through this in detail as you can use all of these details for automating zone creation from a jumpstart of your machine (this is of course left as an exercise for the reader). First up we add our entries to /etc/hosts and our netmask in /etc/netmask, so lets say our zones are sonicmq-z1, z2 and z3 respectively. For the sake of ease lets put these on a subnet 192.1.1.*, so we end up with the following in /etc/hosts

192.1.1.2	sonicmq-z1
192.1.1.3	sonicmq-z2
192.1.1.4	sonicmq-z3

and in /etc/netmasks

192.1.1.0	255.255.255.0

Next up we create our mountpoints for the zones, in this case /export/zones/z1, /export/zones/z2 and /export/zones/z3, and change the permissions on these to being root readable only (chmod 700 <dir>).

Okay, so now we create a command file to create the zone (we could of course type all of this in, but nah). A sample command file for our zone sonicmq-z1 looks like the following.

create -b
set zonepath=/export/zones/z1
set autoboot=true
add inherit-pkg-dir
set dir=/lib
end
add inherit-pkg-dir
set dir=/platform
end
add inherit-pkg-dir
set dir=/sbin
end
add inherit-pkg-dir
set dir=/usr
end
add net
set address=192.1.1.2
set physical=iprb0
end

So now we create the zone, this is done by using zonecfg(1M), and the the zone is installed using zoneadm(1M).

# zonecfg -z sonicmq-z1 -f /export/work/sonicmq/sonicmq-z1.zcf
# zoneadm -z sonicmq-z1 install
Preparing to install zone .
Creating list of files to copy from the global zone.
Copying <2436> files to the zone.
Initializing zone product registry.
Determining zone package initialization order.
Preparing to initialize <891> packages on the zone.
Initialized <891> packages on zone.
Zone  is initialized.
The file < /export/zones/z1/root/var/sadm/system/logs/install_log > contains a log of the zone installation.

Zone sonicmq-z1 installed, and then we repeat for sonicmq-z2 and z3. At this stage we now have three zones setup and configured.

# zoneadm list -c -v
  ID NAME             STATUS         PATH
   0 global           running        /
   - sonicmq-z1       installed      /export/zones/z1
   - sonicmq-z2       installed      /export/zones/z2
   - sonicmq-z3       installed      /export/zones/z3

Now the first time we use a zone we need to configure it, so we can do this manually or using sysidcfg (as you would do with jumpstart). In this case we create a simple sysidcfg file, and place it in the relevant zones /etc, so for the sonicmq-z1 zone this is /export/zones/z1/root/etc/sysidcfg. Our sample file looks like the following (again if your doing this on our own machine this root password for the zone ends up being zone123). As an aside name services are disabled for benchmarking purposes.

system_locale=C
terminal=dtterm
network_interface=primary {
        hostname=sonicmq-z1
}
security_policy=NONE
name_service=none
timezone=Australia/NSW
root_password=30h7toltBltcM

So we can watch whats going on, lets set the zones to being ready, and take a look at the consoles for the zones.

# zoneadm -z sonicmq-z1 ready
# zlogin -C sonicmq-z1

and then boot them. The zone will reboot once to allow the details in the sysidcfg to come into effect, and then its up and running.

# zoneadm -z sonicmq-z1 boot
.... console output...
[Connected to zone 'sonicmq-z1' console]

[NOTICE: Zone booting up]


SunOS Release 5.10 Version s10_72 32-bit
Copyright 1983-2004 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: sonicmq-z1

rebooting system due to change(s) in /etc/default/init

[NOTICE: Zone rebooting]

SunOS Release 5.10 Version s10_72 32-bit
Copyright 1983-2004 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: sonicmq-z1

sonicmq-z1 console login:

So thats our zones setup.

Basic Configuration of the Sun Java System Message Queue Software

The Sun Java System Message Queue Software is quite a complex piece of software (which I will freely admit I'm just learning about at the moment, part of the reason for looking into this benchmark), it is dealt with in detail in the relevant documentation. For the purposes of this example we will just do a very simple configuration.

First off we need to setup a broker, this is done via imqbrokerd(1M) and its related configuration files. At its very simplest, and using the default settings you can just run imqbrokerd

# imqbrokerd
[16/Dec/2004:19:06:42 EST]
================================================================================
Sun Java(tm) System Message Queue
Sun Microsystems, Inc.
Version:  3.5 SP1  (Build 48-G)
Compile:  Thu Jan 29 14:39:11 PST 2004

Copyright ? 2004 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.

This product includes code licensed from RSA Data Security.
================================================================================
Java Runtime: 1.4.2_06 Sun Microsystems Inc. /usr/j2se/jre
[16/Dec/2004:19:06:43 EST]    IMQ_HOME=/
[16/Dec/2004:19:06:43 EST] IMQ_VARHOME=/var/imq
[16/Dec/2004:19:06:43 EST] SunOS 5.10 x86 localhost (1 cpu) root
[16/Dec/2004:19:06:43 EST] Max file descriptors: 65536 (65536)
[16/Dec/2004:19:06:43 EST] Java Heap Size: max=195968k, current=32704k
[16/Dec/2004:19:06:43 EST] Arguments:
[16/Dec/2004:19:06:43 EST] [B1060]: Loading persistent data...
[16/Dec/2004:19:06:43 EST] Using built-in file-based persistent store: /var/imq/instances/imqbroker/
[16/Dec/2004:19:06:43 EST] [B1039]: Broker "imqbroker@localhost:7676" ready.

So this is all well and good, but at the very least (for the sake of the disk in my laptop), I'm going to lower the loglevel and use the 1.5 JVM that comes in Solaris 10. I also want the broker to start automatically when I boot my zone, so to do this I need to edit /etc/imq/imqbrokerd.conf, and so we end up with the following

AUTOSTART=YES
ARGS=-loglevel NONE -javahome /usr/jdk/instances/jdk1.5.0
RESTART=YES

The AUTOSTART=YES variable allows us to use /etc/init.d/imq to startup our imqbroker. It will appear under svcs(1) as a legacy service.

So thats a basic broker setup, nextup we want to setup some queues/topics with the broker, to this we use imqcmd(1M). First off lets just take a look at our broker, and make sure we can connect from another zone (fyi the default password here is admin)

# zonename
sonicmq-z1
# imqcmd query bkr -b sonicmq-z3:7676 -u admin
Password:
Querying the broker specified by:

--------------------------
Host          Primary Port
--------------------------
sonicmq-z3    7676

Version                                              3.5 SP1
Instance Name                                        zonedbroker
Primary Port                                         7676

Current Number of Messages in System                 0
Current Total Message Bytes in System                0

Max Number of Messages in System                     unlimited (-1)
Max Total Message Bytes in System                    unlimited (-1)
Max Message Size                                     70m

Auto Create Queues                                   true
Auto Create Topics                                   true
Auto Created Queue Max Number of Active Consumers    1
Auto Created Queue Max Number of Backup Consumers    0

Cluster Broker List (active)
Cluster Broker List (configured)
Cluster Master Broker


Cluster URL

Log Level                                            NONE
Log Rollover Interval (seconds)                      604800
Log Rollover Size (bytes)                            unlimited (-1)

Successfully queried the broker.

Okay, to this all works, next up, lets create those topic destinations. First off we take a look to see what exists already

# imqcmd list dst -b sonicmq-z3:7676 -u admin -p admin
Listing all the destinations on the broker specified by:

--------------------------
Host          Primary Port
--------------------------
sonicmq-z3    7676

---------------------------------------------------------------------
Name    Type    State    Max Active Consumers    Max Backup Consumers
---------------------------------------------------------------------

Successfully listed destinations.

As expected, seeing as this is a blank config theres nothing. So lets create five topics for this example. For each just run a command similar to

# imqcmd create dst -n TestTopic1 -t t -b sonicmq-z3:7676 -u admin -p admin
Creating a destination with the following attributes:

Destination Name    TestTopic1
Destination Type    Topic

On the broker specified by:

--------------------------
Host          Primary Port
--------------------------
sonicmq-z3    7676

Successfully created the destination.

And now lets take a look again

# imqcmd list dst -b sonicmq-z3:7676 -u admin -p admin
Listing all the destinations on the broker specified by:

--------------------------
Host          Primary Port
--------------------------
sonicmq-z3    7676

------------------------------------------------------------------------------
Name          Type     State      Max Active Consumers    Max Backup Consumers
------------------------------------------------------------------------------
TestTopic1    Topic    RUNNING    -                       -
TestTopic2    Topic    RUNNING    -                       -
TestTopic3    Topic    RUNNING    -                       -
TestTopic4    Topic    RUNNING    -                       -
TestTopic5    Topic    RUNNING    -                       -

Successfully listed destinations.

For the purposes of this example this is pretty much all that we need.

Updating the SonicMQ benchmark code

As JMS is an api, the whole idea is that your code should be vendor independent. For the purposes of the benchmark Sonic have created an inner class, InitialContext which stimulates an external JNDI InitialContext, it is in here that we reference the Sun Java System Message Queue TopicConnectionFactory and QueueConnectionFactory. This is a usefull trick for development purposes as well, and is documented in some more detail in the Message Queue 3.5 Java Client Developers Guide. The code differences are tiny and a sample diff is given below for PubSubPublisherApp.java.

--- sonicmq/PubSubPublisherApp.java     Sun Dec 26 10:54:24 2004
+++ imq/PubSubPublisherApp.java Sun Dec 26 10:57:24 2004
@@ -889,12 +889,17 @@
         public Object lookup(String str)
             throws javax.naming.NamingException, javax.jms.JMSException
         {
-            if( str.equalsIgnoreCase("TopicConnectionFactory") )
-                return (new progress.message.jclient.TopicConnectionFactory (m_BrokerURL));
-            else if( str.equalsIgnoreCase("QueueConnectionFactory") )
-                return (new progress.message.jclient.QueueConnectionFactory (m_BrokerURL));
-            else // you should use an independent 3rd-party service
-                return null;
+            if( str.equalsIgnoreCase("TopicConnectionFactory") ) {
+               com.sun.messaging.TopicConnectionFactory tcf = new com.sun.messaging.TopicConnectionFactory();
+               tcf.setProperty("imqAddressList", m_BrokerURL);
+                return (tcf);
+            } else if( str.equalsIgnoreCase("QueueConnectionFactory") ) {
+               com.sun.messaging.QueueConnectionFactory qcf = new com.sun.messaging.QueueConnectionFactory();
+               qcf.setProperty("imqAddressList", m_BrokerURL);
+                return (qcf);
+           } else {
+               return null;
+          }
         }
     }//end of inner class InitialContext

Compilation and initial configuration

To compile up our application we need to include the imq jar file in our classpath. On a default Solaris 10 installation this is located at /usr/share/lib/imq.jar, so we add this to our classpath, and compile up

export CLASSPATH=$CLASSPATH:/usr/share/lib/imq.jar
javac PubSubPublisherApp.java
javac PubSubSubscriberApp.java

which gives us

# ls PubSub*
PubSubPublisherApp$InitialContext.class     PubSubSubscriberApp$InitialContext.class
PubSubPublisherApp$PublisherObject.class    PubSubSubscriberApp$SubscriberObject.class
PubSubPublisherApp.class                    PubSubSubscriberApp.class
PubSubPublisherApp.java                     PubSubSubscriberApp.java

I will leave the dynamically generated classes for your own reading ;). As mentioned above I am using the 1.5 jvm that comes with Solaris 10.

As I compiled these up in the global zone, I have just copied them into the filesystems of each of the zones I am using for this demo, so the above classes end up in

/export/zones/z1/root/export/imq
and 
/export/zones/z2/root/export/imq

respectively.

And finally, kicking off a basic run

Okay, so we have our three zones setup, our message queue configured and our application set up, so all thats left is to actually kick off the run. To do this lets just ensure we are starting with a clean config lets reboot our zones with zoneadm(1), and then start.....

# zoneadm -z sonicmq-z1 boot
# zoneadm -z sonicmq-z2 boot
# zoneadm -z sonicmq-z3 boot
# zoneadm list -cv
  ID NAME             STATUS         PATH
   0 global           running        /
   1 sonicmq-z1       running        /export/zones/z1
   2 sonicmq-z2       running        /export/zones/z2
   3 sonicmq-z3       running        /export/zones/z3

Okay, zones all up, fyi I have a little script for creating my initial topics.
So lets get our consoles on our zones sonicmq-z1 and z2 respectively using zlogin(1).
Finally lets actually do something, we will keep this example very simple, so lets first off get the PubSubPublisherApp up and runing on sonicmq-z1.

# zonename
sonicmq-z1
# export CLASSPATH=$CLASSPATH:/usr/share/lib/imq.jar
# java PubSubPublisherApp -b sonicmq-z3:7676 -u admin -p admin -npublishers 1 -nconns 1 -tname TestTo>
[05/01/31 15:08:27]
===========================
CODE VERSION: 2.5
Broker url: sonicmq-z3:7676
Tx session: false
Msg/tx:     50
Username: admin
Password: admin
Number of Publishers: 1
Number of Connections: 1
Number of Sessions: 1
Number of Topics: 4
Topic name prefix: TestTopic
Message size: 1 Kb
Delivery Mode: PERSISTENT
Number of Reporting intervals: 10
Number of Rolling intervals: 8
Seconds per interval: 60
===========================
Please wait for PubSubPublisherApp to finish setting up for test...
**** Please press  to start test intervals.***

and then on sonicmq-z2 we will start the PubSubSubscriberApp

java PubSubSubscriberApp -b sonicmq-z3:7676 -u admin -p admin -nsubscribers 1 -nconns 1 -tname TestTopic
[05/01/31 15:12:29]
===========================
CODE VERSION: 2.5
Broker url: sonicmq-z3:7676
Tx session: false
Msg/tx:     50
Ack mode: DUPS_OK
Username: admin
Password: admin
Number of Subscribers: 1
Number of Connections: 1
Number of Sessions: 1
Number of Topics: 4
Topic name prefix: TestTopic
Number of Reporting intervals: 10
Number of Rolling intervals: 8
Seconds per interval: 60
Durable Subscriber (yes/no):no
Agent Name: MyDurSub
===========================
Please wait for PubSubSubscriberApp to finish setting up for test...
NOTE: all connections now explicitly setClientID
***Please press  to start test intervals.***

Okay, were ready to run, after hitting enter we get the following output.......
First up, sonicmq-z2

[05/01/31 15:12:33] Starting ...

Clearing id: 0 clearing: 0
----------------------------------------------------------------------------------------------------------------
Interval #
|
|                Interval Receive Rate (msg\sec)
|                |
|                |               Rolling Receive Rate (msg\sec) [Last 8] intervals
|                |               |
|                |               |               Interval Msg Total
|                |               |               |
|                |               |               |               Cumulative Msg Total
----------------------------------------------------------------------------------------------------------------
Interval 1       1376.2         1376.2          82572.0         82572.0
Interval 2       2127.4         1751.8          127644.0                210216.0
Interval 3       2098.4         1867.3          125903.0                336119.0
...snip...

and on sonicmq-z1

[05/01/31 15:12:35] Starting ...

----------------------------------------------------------------------------------------------------------------
Interval #
|
|                Interval Publish Rate (msg\sec)
|                |
|                |               Rolling Publish Rate (msg\sec) [Last 8] intervals
|                |               |
|                |               |               Interval Msg Total
|                |               |               |
|                |               |               |               Cumulative Msg Total
----------------------------------------------------------------------------------------------------------------
Interval 1       1444.8         1444.8          86685.0         86685.0
Interval 2       2123.8         1784.3          127426.0                214111.0
Interval 3       2128.0         1898.9          127682.0                341793.0

voila, the whole thing running on a single x86 laptop. Hopefully if you are developing multitier applications on Solaris 10 this will be of use to you during your testing and deployment phases. For more details on using zones in production environments please take a look through the related docs on the BigAdmin Zones Site, in particular the paper presented at Lisa is worth a read.

Technorati Tag Solaris

(2005-01-30 20:50:08.0) Permalink Comments [0]