David Dagastine's Weblog
Archives
« October 2006 »
SunMonTueWedThuFriSat
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
    
       
Today
XML
Search

Links


blogs.sun.com
Weblog
Archives
About
Login
Referrers

Today's Page Hits: 325

All | General | Java Performance | JVM Performance Tuning Basics | Music
« Previous month (Aug 2006) | Main | Next month (Oct 2006) »
20061016 Monday October 16, 2006
No Tuning Required: Java SE Out-of-Box Vs. Tuned Performance
In my last entry titled Java SE Out-of-Box Competitive Performance I stressed the importance of out-of-box performance to customers and developers and how it is a passionate focus for Sun HotSpot and JVM Performance engineering. The following is a comparison of out-of-box and hand-tuned performance. The charts below are run on the same system as my previous entry and the charts are normalized to the same baseline, therefore the two sets of charts are directly comparable.

I have to say the numbers are quite impressive (hence the "No Tuning Required" in the title). My colleagues are going to say I'm blogging us out of a job :-).

The system under test is a 2-way dual-core Opteron 280 Processors (2 CPUs, 4 cores, 2.4 Ghz) and 8GB of RAM. The Operating System is Red Hat EL 4.0 AS Update 4. The kernel version is unmodified from the base install, which is 2.6.9-42.ELsmp. The charts are statistical comparisons. No less than 10 samples were performed, and a T-test (single-tailed) was used to ensure confidence in the significance of the result. The data is normalized to the 32-bit Sun JDK 1.5.0_08 out-of-box result.

The following JVMs were tested:

The following command line arguments were used: The SPECjbb2005 numbers are impressive. JDK 5.0_08 is ~22% faster tuned compared to JDK 5.0_08 out-of-box. JDK 6 is only ~11% faster tuned vs. JDK 6 out-of-box, and JDK 6 out-of-box is only ~7% slower than highly tuned JDK 5.0_08. Nice.

Tuning only improved Scimark slightly when running 5.0_08. When running JDK 6 it's more or less a wash. The JDK 6 64-bit difference is statistically insignificant.

Tuning seems to hurt Volano, except when running 5.0_08 64-bit. Come to find out the negative differences are statistically insignificant so tuning is a wash with Volano as well.

In summary, meeting or exceeding tuned performance is the end game for out-of-box performance engineering. The above results make me quite proud of our accomplishments. Yes, every application is different and in some cases we'll find ourselves needing to tune. But chances are if you let us know the issues you're facing a release or two down the line you won't need to tune. Eventually it will just be us geeks who can't help it :-). Next step is a Solaris x86 vs Linux comparison. Stay tuned.

SPEC(R) and the benchmark name SPECjbb(TM) are trademarks of the Standard Performance Evaluation Corporation. Competitive benchmark results stated above reflect experiments performed by Sun Microsystems, Inc. For the latest SPECjbb2005 benchmark results, visit http://www.spec.org/osg/jbb2005.


Oct 16 2006, 04:41:40 PM EDT Permalink Comments [7]

20061006 Friday October 06, 2006
Java SE Out of Box Competitive Performance
Out-of-Box Performance, or no tuning options is in many ways our ultimate goal in HotSpot development. As a JVM performance engineer I too have spent countless hours tweaking command line arguments to squeeze out the last remaining bit of performance. In my last blog entry I asked if there was interest in an out of box competitive performance comparison and the second comment I received hit it on the nose. Command line tuning, albeit fruitful at times, can also be a royal waste of time. Especially when you're shooting in the dark trying any option you can find without any knowledge to what the flag is doing.

Your friends in HotSpot engineering don't want you spending time tuning either. That was the driving force behind Java SE 5.0 Ergonomics and why key performance features previously available via JVM options are now enabled by default in Java SE 6.

The intention of the data charts below is to highlight the importance of customer experience and out-of-box performance to Sun Java Engineering. These are not meant to be high performance benchmark results. Hand tuning can change the results significantly.

The following is an out-of-box performance comparison on a Sun Fire X4200. The system is configured with 2 dual-core Opteron 280 Processors (2 CPUs, 4 cores, 2.4 Ghz) and 8GB of RAM. The Operating System is Red Hat EL 4.0 AS Update 4. The kernel version is unmodified from the base install, which is 2.6.9-42.ELsmp. The only variable in this configuration is the JVM.

The JVM distributions and versions tested were the latest versions publicly available at the time of testing. I was sure to use the BEA JRockit JVM used in recent SPECjbb2005 submissions. The IBM JVM is the latest available on the IBM developer website.

As stated above and in the title no JVM tuning options were used for these results. The results below are statistical comparisons. No less than 10 samples were performed, and a T-test (single-tailed) was used to ensure confidence in the result. The data is normalized to the 32-bit Sun JDK 1.5.0_08 result.

The first chart is SPECjbb2005. SPECjbb2005 is SPEC's benchmark for evaluating the performance of server side Java. It evaluates server side Java by emulating a three-tier client/server system (with emphasis on the middle tier). It extensively stress Java collections, BigDecimal, and XML processing. The cool thing about SPECjbb2005 is that optimizations targeted for it also show performance gains in other competitive benchmarks, such as SPECjappserver2004, and a broad range of customer workloads. The benchmark results below are run in single instance mode. Notice the impressive gains with Java SE 6 with nearly a 15% improvement over JDK 5.0_08. Also notice there is very little difference between 32-bit and 64-bit BEA JRockit results.

SciMark 2.0 is a Java benchmark for scientific and numerical computing and is a benchmark where Sun's JVMs have continued to shine. Its a decent test of generated code, particularly for tight computational loops. However it is particularly sensitive to alignment issues and can show some level of variance from run to run, mostly in a bimodal fashion. All in all its a good set of microbenchmarks. Notice that 64-bit is faster than 32-bit for all of the JVMs under test. The additional registers available running 64-bit on AMD Opteron certainly do impact computational performance.

Volano is a popular Java chat server. The benchmark is quick and involves both a client and server instance. From a JVM perspective the workload is heavily dominated by classic Java socket I/O which is a bit long in the tooth, an NIO version would be quite interesting. That being said, some customers have found this benchmark quite useful so we continue to test it. Running Volano the performance gaps are not as large, most likely because this benchmark has very little garbage collection overhead. BEA JRockit is showing good performance here with a result thats 10% over the baseline. Sun Java SE 6 shines as well with a result thats nearly 20% over baseline.

In summary, we in Java SE Performance and HotSpot Engineering feel that out-of-box performance is extremely important to Java developers and customers, and I hope the results above differentiate our product and highlight our ongoing work and focus. Next step is a out-of-box vs. highly tuned comparison. Stay tuned.

SPEC(R) and the benchmark name SPECjbb(TM) are trademarks of the Standard Performance Evaluation Corporation. Competitive benchmark results stated above reflect experiments performed by Sun Microsystems, Inc. For the latest SPECjbb2005 benchmark results, visit http://www.spec.org/osg/jbb2005.


Oct 06 2006, 03:33:11 PM EDT Permalink Comments [8]