Dileep Kumar's Weblog

Sometime back we have published a Sun Blueprint (Accelerating IBM HTTP Server Cryptographic Operations Using Sun Servers with CoolThreads Technology) detailing the steps needed to get your IBM HTTP Server use the on-chip crypto processor on Ultra SPARC T2 based system for SSL operation. This will give a free SSL operation boost without buying additional hardware for such operations.
The documentation lists all the steps needed to get your IBM HTTP Server and GSKit working  with this on-chip crypto module on Ultra SPARC T2 processor.  In addition to how to configure it, it also has results from some of the performance testing that has been done to measure the performance gain. Your milegae may vary depending on your type of workload but if you are making lot of new client connection and serving "HTTPS" traffic then this would something that is available to you free you want to consider. It wil help you take care of your SSL handshakes operations.
Another important aspect of this is that GSKit is a common library that has been used by IBM in lot of products. And as its evident from the name, Global Security Kit, it is security related implementation to be used across different products such as PKCS#11 or so. Some more details can be found at my prior blog about GSKit. This implies that if you want to hookup with PKCS#11 provider and take advantage of on-chip cryptography for other products that can be done too. You must note that this integration has happened at certain level of IBM HTTP Server so it requires certain version of GSKit embedded with the product for which you will try to take advantage.

It's been really busy since I came back from IMPACT 2009. It was a great event. Since then I have busy working on few stuffs for JavaOne and CommunityOne.
The opening ceremony was great and Billy Crystal was really hilarious.
As usual every company does some launches during these event and IBM followed the tradition by putting up "WebSphere Cloud Brust". It was quite confusing to lot of people including me but when I stopped by the Cloud Brust booth I got fair bit of idea about what it is and how it can provide the value add although I expected the booth demo to be live but is wasn't the case.
Lot of session has talked by processor advancement, memory placement, bus speed and what not..During these there were numerous references to Sun Ultra SPARC T2 line of processor which leads the multi-core revolution. You might have got different perspective of different speaker on this but if you have used it you know the power, processing and power efficiency included, of these systems. If you haven't tried your hand then do and know it for yourself.

I expected this to be on linux/xen combo but surprisingly it was based on VMWare hypervisor that was very interesting. Similar deployment toplogy can be achieved using no additional technology overhead with Solaris such  zones/container or if you are ready to try your hands on OpenSolaris you can use xen. Remember OpenSolaris has tool to get your VMWare images working on OpenSolaris.
Here are few important pictures I took.

WebSphere Cloud Brust announcement slide:

Wanted to let you all know that Sun/IBM Team will be hosting the following BOF during this year IMPACT 2009.
Title: Optimizing your IBM WebSphere® Infrastructure with Solaris
Session ID: BOF-3174A
Session Time: Tue, May 5 12:00 PM - 01:00 PM
Location: Venetian Galileo 904
This will give you an unique opportunity to talk to engineers from both Sun and IBM team. We can cover whole range of topics concerning your WebSphere deployment on Sun platform.

For rest of the discussion I suggest you stop by at our booth and we can help you get your question answered.

As always I got chance to try my hands on one of the new blade server that Sun has released based on Intel Xeon processor 5500 series namely, Sun Blade X6270 server module. The new Xeon processor has lot of features to deliver better performance based on the workload by which it is driven eg. automatically increasing processor frequency when needed and possible, as well as the capability to have extra hardware threads or taking advantage of the processor's power and thermal headroom. I would suggest you to visit http://blogs.sun.com for more details from domain expert including power consumption to virtualization and everything else supplemented by best OS on the planet "Solaris 10/OpenSolaris". I will leave rest of the discussion to my distinguished colleagues at Sun and will talk about what I observed when running WebSphere Application Server on this system.

As most current version of WebSphere Application Server available was V7.0 so I picked up this installed then made sure I am running with the latest fixpacks and ran it. Everything was breeze from installation, fix-pack upgrade to start/stop and then benchmarking. The Sun Blade X6270 I had was built on 2 sockets of the Intel Xeon L5520 processor and the Solaris 10 10/08 Operating System.

As always I used the same workload that I use for my WebSphere tests and there are other components which included the Database Server which was running DB2, and the test Drivers. For benchmark I used Faban which is very easy to use and run and it does the bookkeeping of benchmark so nicely that I don't have to keep records myself. I don't constrain my DB in terms of performance so I let it run in "/tmp" to just make everything simple so I don't spend too much time on tuning DB as I am working on WebSphere App Server.

The Intel Xeon processor 5500 series has two modes for running namely, Turbo mode and Hyper-Threading mode options in the BIOS setup. Which can be changed during boot by going to BIOS menu and doing the right selection. As Intel has put up "Nehalem, unleashes parallel processing performance enabled by an integrated memory controller and Intel® QuickPath Technology providing high-speed interconnects per independent processing core. The Intel® Turbo Boost Technology taking advantage of the processor's power and thermal headroom. This enables increased performance of both multi-threaded and single-threaded workloads and with Intel HT Technology, bringing high-performance applications into mainstream computing with 1-16+ threads optimized for new generation multi-core processor architecture." I did tried to run in both modes did not notice any significant difference for my tests while running in these modes.

Single instance of server was able to use much of the system capacity and stopped scaling. I was running all the tools and nicstat was one able to point me the moment when the n/w bandwidth was exhausted it stopped taking any more load. Then to work around the network capacity I put a direct connection with the DB Server which resulted in little better throughput and little more system utilization. On the network communication between DB and Server was happening over this interface. Still not enough and server had capacity neither the thread-dumps or system stats pointed out any problem other than network again being saturated. In which the public interface was full loaded and there was a very light load on the point to point connection interface. At this point I don't had any choice other than adding another instance of the server. To do this I created a Solaris Zone/Container and had one instance running into this zone. For this zone I used the point to point connection between the DB and the blade server which eliminated this and I was able to use the server to full capacity.

The network generated interrupts and there distribution can be checked by using the following command:
-bash-3.00# echo ::interrupts | mdb -k | grep igb
49 0x60 6 MSI-X 0 1 - igb_intr_tx_other
50 0x61 6 MSI-X 0 1 - igb_intr_rx
51 0x62 6 MSI-X 4 1 - igb_intr_tx_other
52 0x63 6 MSI-X 7 1 - igb_intr_rx
-bash-3.00#

I looked for igb only becuase my network drivers are as:
igb0 - public (ie. going over GB switch)
igb1 - private (point-to-point)
From the above it is clear that the tx and rx interrupts are going over virtual cpu or more technically threads 0, 4 and 7 which is very well distributed.

So what did I do for tuning the WebSphere ?
My tunings remains same as I was not changing the Operating System and Software which includes Java. I will list them here for reference:

Disabled the Performance Monitoring Infrastructure
Disabled the Application Profiling
I used the DynaCache for the App just as to save some DB overhead
Set the WebContainer thread to the same number of threads as load driver
For Connection Pool I made sure that the number of DB connections are more than my WebContainer thread pool so none of the thread end up waiting for DB connections.
JVM:
initialHeapSize="2048" maximumHeapSize="2048" -server -Xmn1024m -XX:+AggressiveOpts -XX:+UseParallelGC -XX:ParallelGCThreads=8 -XX:+UseParallelOldGC

If you have been being hit by the processing overhead of SSL stuff and have to buy special purpose hardware to speed up things, then may be its time to leave all this processing to Ultra SPARC T2 and T2 Plus based system. Which has builtin cryptogarphic support on the chip and websphere supports this:

Here are the details from IBM WebSite: http://www.ibm.com/developerworks/java/jdk/security/50/secguides/pkcs11implDocs/IBMPKCS11SupportList.html
http://www.ibm.com/developerworks/java/jdk/security/60/secguides/pkcs11implDocs/IBMPKCS11SupportList.html

For details of setup you can follow the WebSphere Infocenter.
Please note that this feature will be available only after certain fix pack release so it will not work unless you upgrade your JDK to required level as suggested in the doc.

If you want to learn more about the Cryptographic Acceleration offered by these processor I would encourage to read this Sun Blueprint titled "Using the Cryptographic Accelerators in the UltraSPARC® T1 and T2 Processors".

We will be hosting an Ask The Experts session on March 24th. 
We invite you all to join where we can discuss and share various ideas around how to optimally use WebSphere software on Solaris platform.
For  details please visit the WebSphere User Group web-site(http://www.websphere.org/websphere/jsp/ask_expert.jsp).

As lot of you may have tried already installing/using WebSphere on OpenSolaris and may have ran into some issues during install so I thought I will cover some of those issues into this blog and will provide possible workaround.
The instruction below applies to WebSphere Application Server v6.1 and v7.0. I havn not  tested 6.0.X but I assume it will work fine. If not then let me know. 

OpenSolaris brings in lot of new feature details can be found at opensolris.org. If you don't want to commit a hard disk for OpenSolaris experience you can use VirtualBox and then install OpenSolaris inside VirtualBox. Once you figure out which route you want to take and how you want to install OpenSolaris then you need to grab installer for WebSphere and try installing it over. You will have following few options available to you to install:
Silent installation:

You will have to edit the template response file. After you make all the necessary changes if you try to install it will fail and if you check the log it will complain that this is not one of supported OS. The reason for this can be checked by the output of the command "uname -a" and as you can see from output it is being reported as "5.11".  So this is what you will have to add in your response file:
-OPT disableOSPrereqChecking="true"
Now try again and installation should complete without any error and you will get message about the Operating System version. Just ignore this message and start using.

When you try to install using launchpad.sh it will come up but when it invokes the install command then it will fail complaining the installation cannot be started in GUI and suggest that you use "silent" installer. This error happens only with WAS v6.1 only WAS v7.0 is fine and you can install without any issues. If you want to get into details you can check the niflog  and you will understand that this is happening due to installshield issues. Googling for this will reveal that this is a known issue with lot of platform eg. AIX/Linux/Solaris and it has nothing to do with either the OS or WAS.
So we can use the java based installe by directly invoking the following command:
bash # java -jar setup.jar
This to succeed you must have java runtime in your path which is by default available in OpenSolaris. It will bring up the installer and as said earlier "Prerequisite Check" will fail and if you still continue everything will be installed right.

By this time you might have got your hand on new WebSphere Application Server namely WebSphere Application Server v7.0. There are lot of good thing about this version as compared to its predecessors. A list of which can be found easily from 7.0 InfoCenter.
Some of the feature as Solaris user you should be aware are:

• New JDK/JVM Support


• DTrace made easy


• Performance Enhancement


• New J2EE Spec Level Compliance

Today, we are announcing the next generation server based on UltraSPARC T2 Plus processor and this server happens to be a monster in terms of performance. It can consolidate a lot of servers in a single box providing the similar combined throughput. For the server details and other blogs related to this server I suggest you to visit T5440 Blog index(http://blogs.sun.com/allanp/entry/sun_s_4_chip_cmt).
In this blog, I summarize the performance of IBM WebSphere Application Server (WAS)  on T5440. To test and benchmark the performance, I use the most recent release WAS v7, that is based on Java EE 5 spec and JDK 6, with numerous features and performance enhancements. As I said earlier that this new Sun server is a monster in terms of performance, the WAS software set up and configuration require some planning and appropriate allocation of the system resources.   When you do the psrinfo(1M) command on this server, it will report to you that the system has 256 "processors".  This means that the processing power of this system far exceeds the software scalability of a single instance of the Application Server's. Thus, you will need multiple instances of WAS to drive the system to its full utilization.  Solaris Containers provides the most efficient way to accomplish such configuration as Solaris Containers provides process space isolation among different WAS instance as well as allocating proper system resources.
For maximizing the utilization of the system, I configured the environment as follows.

• I created 7 Solaris Containers, allocated 32 processor threads to each of the 6 Containers, and allocated the remaining processor threads for the other container and the global zone.  Then, I used the following WAS configuration:


• initialHeapSize="2500" maximumHeapSize="2500"


• -server -Xmn2048m -XX:+AggressiveOpts -XX:+UseParallelGC -XX:ParallelGCThreads=16 -XX:+UseParallelOldGC -Dcom.ibm.CORBA.TransportMode=Pluggable -Dcom.ibm.ws.pm.batch=true -Dcom.ibm.ws.pm.deferredcreate=true -Dcom.ibm.CORBA.FragmentSize=3000 -Dcom.ibm.ws.pm.useLegacyCache=false -Dcom.ibm.ws.pm.grouppartialupdate=true -Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl -Djavax.xml.parsers.SAXParserFactory=org.apache.xerces.jaxp.SAXParserFactoryImpl -Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl -Dorg.apache.xerces.xni.parser.XMLParserConfiguration=org.apache.xerces.parsers.XML11Configuration


• Disable the PMI feature.


• Disable the System.out logging from the admin console if you want you can do the same by editing the config file. by editing the trace service like - startupTraceSpecification="*=info:SystemOut=off"


• DynaCache - The DynaCache was used for this benchmark so the default size of the cahce was set to 2000 which wasn't enough for this test which can be adjusted accordingly based on the application need as I set it to 3000.


• For thread pool tuning the only thing that was relevant in this benchmark was tuning the WebContainer thread pool right so I did set it to 80 and which was more that enough and I think It can be scaled down a bit but havn't tried.


• Database connection pool was set it to same value as WebContainer pool and set the min and max to the same value.

Rajesh has put up the EC2 blog where you can check for breaking news about OpenSolaris on EC2. Here are some information which is available for you:
Meet the Sun OpenSolaris on EC2 Team
Information on GlassFish/LifeRay/MySQL AMI
Current beta program status.

Check and leave you comments there if you want to know anything about this.
There are other AMI's are in progress and should be made available as soon as it is ready.

If you run into issues when following my previous blog then you may running into problem with sysevent daemon.
If that happens here is what you need to do:
once done installing all the package and rebooting with the Sun xVM server option in the boot menu from the dom0 issues the following command.

Cut and paste everything in between ---- this line to your terminal and hit return and then you should be able to proceed with domU creation.

The error is like as:

bash-3.2# xm create -c install.py

Using config file "./install.py".
bash-3.2# Error: Device 0 (vif) could not be connected. Hotplug scripts not working.

-------------------------

BASEDIR=${BASEDIR:-/}

/usr/sbin/syseventadm list -R $BASEDIR -c EC_xendev > /dev/null 2>&1
if [ $? -ne 0 ]
then
/usr/sbin/syseventadm add -R $BASEDIR -c EC_xendev \
/usr/lib/xen/scripts/xpvd-event 'action=$subclass' \
'domain=$domain' 'vdev=$vdev' 'device=$device' \
'devclass=$devclass' 'febe=$fob'
fi

/usr/sbin/syseventadm list -R $BASEDIR -c EC_xpvsys > /dev/null 2>&1
if [ $? -ne 0 ]
then
/usr/sbin/syseventadm add -R $BASEDIR -c EC_xpvsys \
/usr/lib/xen/scripts/xpvsys-event 'subclass=$subclass' \
'shutdown=$shutdown'
fi

# restart daemon if the package is being added to the running system
if [ "$BASEDIR" = "/" -a $? -eq 0 ]
then
/usr/sbin/syseventadm restart
fi

---------------------

In this blog I will walk you through the steps of creating a domU within your OpenSolaris 2008.05 installation.
To accomplish this task you must have done your OpenSolaris 2008.05 install and I assume you are familiar with this if not please refer to OpenSolaris web-site.

Once you done installing this you need to use IPS(Image Packaging System) so that your system will have the packages related to xen. Make sure your pkg authority is set right and you are able to download the packages.
If this is set right i.e. if you haven't changed anything then it should look like this:

-bash-3.2# pkg authority
AUTHORITY URL
opensolaris.org (preferred) http://pkg.opensolaris.org:80/

Now you will need to add the following packages(you can cut and paste these commands to your terminal):

pkg install SUNWvirtinst
pkg install SUNWurlgrabber
pkg install SUNWlibvirt
pkg install SUNWxvmhvm
pkg install SUNWxvmdom
pkg install SUNWxvm
pkg install SUNWgccruntime
pkg install SUNWgnutls
pkg install SUNWlibsdl
pkg install FSWxwpft
pkg install FSWxwrtl

Once done doing this now you have all the necessary packages. Now you need to modify you GRUB BOOT Loader so that you have the option to load xVM when you system boots. This can be achieved by doing this:

Open the file "/rpool/boot/grub/menu.lst" in vi and add the following entry:
----
title Solaris xVM
bootfs rpool/ROOT/opensolaris
kernel$ /boot/$ISADIR/xen.gz
module$ /platform/i86xpv/kernel/$ISADIR/unix /platform/i86xpv/kernel/$ISADIR/unix -B $ZFS-BOOTFS
module$ /platform/i86pc/$ISADIR/boot_archive
----
Save your changes and exit out of the editor.
Now reboot your system and when you get to the grub loader select the "Solaris xVM" so that you will have the xVM related modules loaded.

Once the boot is complete you will need to start some of the services to enable you to use the xen management command. The services need to be enabled are as:

svcadm enable svc:/system/xvm/store:default
svcadm enable svc:/system/xvm/xend:default

svcadm enable svc:/system/xvm/console:default

Note: Enabling the console is really necessary otherwise when you run your xm commands you will get into error like following:
/usr/lib/xen/bin/xenconsole: Could not open tty `/dev/pts/7': No such file or directory

Once you have done this follow the instruction listed in MJR's blog.
http://blogs.sun.com/mrj/entry/installing_opensolaris_on

Which will have your domU up and running.

As many of you might have got the news by now about the planet's best Operating System has been made available on one of the most scalable Compute Cloud named Amazon EC2(Elastic Compute Cloud).
This brings you the best of the both worlds which includes the Amazon Web-Services and OpenSolaris code base. If you haven't checked how cool OpenSolaris is I would suggest to get a copy of OpenSolaris and check for your self. Once you get this just get your hand dirty on xVM and try to play with the virtualization it offers. There are may more things that has been included with this OpenSolaris and some of them are as:
. Developed under Open Source community.
. New subscription support.
. Various tools needed for development eg. Sun Studio, Netbeans and the SAMP stack.
. Different virtualization technology
. ZFS and DTRACE.
. Fast and easy installation with a single click and simply add software via the new Image Packaging System (IPS).

Combining Amazon EC2 on demand compute capacity with their integrated storage and running OpenSolaris on top of that gives you unparalleled env to run your application in secure, reliable and easily scalable.
OpenSolaris makes it possible to run your application on EC2 while taking or retaining the differentiating features of the Solaris Operating system and they include:

1. DTrace - You continue to do the same Dynamic Tracing over EC2.
2. Binary and Backward Compatibility - As with Solaris 10 you continue to run your application unmodified on OpenSolaris. So you can run your oldest code unmodified without breaking a thing.
3. Open Source - OpenSolaris is developed under OpenSolaris community with participation from thousands of developer worldwide. It has more predictable life cycle and support service so you can plan and upgrade your system when needed.
4. Easy to use - If you have given a try to install OpenSolaris you would know how easy it is.

So how does the whole thing work ?
Here is the what needed to be done to have your own OpenSolaris running on Amazon EC2:
You must have an account with Amazon WebSerives
Register for OpenSolaris AMI
Sun gets back to you with the details
Use the AMI/API tool to get your env.
Install/Update or simply customize your env by installing software or customizing your system.

For more details don't forget to check here.

We would like to thank all those who stopped by and talked to us during IMPACT. It was a great conference and it was great to talk to you all about your Solaris 10 deployments and how well your deployment is running on Solaris 10.
There were few really great session and I would suggest you get a copy of some of the interesting presentation eg.
1. IBM WebSphere and multicore platforms: Performance, scalability, and best practices
2. Exploring IBM Java 5 and Java 6: New features and enhancements
3. Understanding and tuning the Java Virtual Machine for IBM WebSphere
for you reference. They contain lots of good information which you may found useful later.

Albert has already posted the slides which we used during our presentation. They can be found here:
SIG2730: WebSphere Deployment on Solaris
TSM-2447: WebSphere Performance On Solaris

There was one other session which was on DumpAnalyzer which is not available on Solaris or HP-UX I will put another blog which will list out all the equivalent tools that already available in the Java you have on your system with WebSphere.

Using the jconsole tool
J2SE 5.0 has comprehensive monitoring and management support. Among the tools
included in the JDK download is a Java Management Extensions (JMX)-compliant monitoring
tool called jconsole. The tool uses the built-in JMX instrumentation in the Java Virtual
Machine to provide information on performance and resource consumption of running
applications. Although the tool is included in the JDK download it can also be used to monitor
and manage applications deployed with the Java 2 Platform Standard Edition Runtime
Environment 5.0 (JRE 5.0).
Jconsole can attach to any application that is started with the JMX agent. A system property
defined on the command line enables the JMX agent. Once attached jconsole can be used to
display useful information such as thread usage, memory consumption, and details about
class loading, runtime compilation, and the operating system.