In my last entry titled Java SE Out-of-Box Competitive Performance I stressed the importance of out-of-box performance to customers and developers and how it is a passionate focus for Sun HotSpot and JVM Performance engineering. The following is a comparison of out-of-box and hand-tuned performance. The charts below are run on the same system as my previous entry and the charts are normalized to the same baseline, therefore the two sets of charts are directly comparable.

I have to say the numbers are quite impressive (hence the "No Tuning Required" in the title). My colleagues are going to say I'm blogging us out of a job .

• On SPECjbb2005 the numbers are impressive. JDK 5.0_08 is ~22% faster when tuned compared to JDK 5.0_08 right out of the box. JDK 6 is ~11% faster when tuned versus right out of the box, and JDK 6 out of the box is only ~7% slower than a highly tuned JDK 5.0_08. Very impressive indeed!
• On Scimark, tuning only improved slightly when running JDK 5.0_08. JDK 6 is more or less a wash.
• On Volano, except when running JDK 5.0_08 64-bit, the out-of-the-box configuration seems to work well, and doesn't require any explicit tuning.

The system under test is a 2-way dual-core Opteron 280 Processors (2 CPUs, 4 cores, 2.4 Ghz) and 8GB of RAM. The Operating System is Red Hat EL 4.0 AS Update 4. The kernel version is unmodified from the base install, which is 2.6.9-42.ELsmp. The charts are statistical comparisons. No less than 10 samples were performed, and a T-test (single-tailed) was used to ensure confidence in the significance of the result. The data is normalized to the 32-bit Sun JDK 1.5.0_08 out-of-box result.

The following JVMs were tested:

• Sun JDK 1.5.0_08
• 32-bit: java version "1.5.0_08" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_08-b03) Java HotSpot(TM) Server VM (build 1.5.0_08-b03, mixed mode)
• 64-bit: java version "1.5.0_08" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_08-b03) Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_08-b03, mixed mode)
• Sun Java SE 6 build 99
• 32-bit: java version "1.6.0-rc" Java(TM) SE Runtime Environment (build 1.6.0-rc-b99) Java HotSpot(TM) Server VM (build 1.6.0-rc-b99, mixed mode)
• 64-bit: java version "1.6.0-rc" Java(TM) SE Runtime Environment (build 1.6.0-rc-b99) Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b99, mixed mode)

The following command line arguments were used:

• SPECjbb2005
• J2SE 5.0_08 32-bit: -Xmn1g -Xms1500m -Xmx1500m -XX:+UseBiasedLocking -XX:+AggressiveOpts -XX:+UseLargePages -XX:+UseParallelOldGC -Xss128k
• J2SE 5.0_08 64-bit: -Xmn2g -Xms3g -Xmx3g -XX:+UseBiasedLocking -XX:+AggressiveOpts -XX:+UseLargePages -XX:+UseParallelOldGC -Xss128k
• Java SE 6 RC1 32-bit: -Xmn1g -Xms1500m -Xmx1500m -XX:+UseLargePages -XX:+UseParallelOldGC -Xss128k
• Java SE 6 RC1 64-bit: -Xmn2g -Xms3g -Xmx3g -XX:+UseLargePages -XX:+UseParallelOldGC -Xss128k
• SciMark2
• J2SE 5.0_08 32-bit: -XX:+UseBiasedLocking
• J2SE 5.0_08 64-bit: -XX:+UseBiasedLocking
• Java SE 6 RC1 32-bit: -XX:+DoEscapeAnalysis
• Java SE 6 RC1 64-bit: -XX:+DoEscapeAnalysis
• Volano 2.5.0.9
• J2SE 5.0_08 32-bit: -XX:CompileThreshold=1500
• J2SE 5.0_08 64-bit: -XX:CompileThreshold=1500
• Java SE 6 RC1 32-bit: -XX:CompileThreshold=1500 -XX:-UseBiasedLocking
• Java SE 6 RC1 64-bit: -XX:CompileThreshold=1500 -XX:-UseBiasedLocking

The SPECjbb2005 numbers are impressive. JDK 5.0_08 is ~22% faster tuned compared to JDK 5.0_08 out-of-box. JDK 6 is only ~11% faster tuned vs. JDK 6 out-of-box, and JDK 6 out-of-box is only ~7% slower than highly tuned JDK 5.0_08. Nice.

Tuning only improved Scimark slightly when running 5.0_08. When running JDK 6 it's more or less a wash. The JDK 6 64-bit difference is statistically insignificant.

Tuning seems to hurt Volano, except when running 5.0_08 64-bit. Come to find out the negative differences are statistically insignificant so tuning is a wash with Volano as well.

In summary, meeting or exceeding tuned performance is the end game for out-of-box performance engineering. The above results make me quite proud of our accomplishments. Yes, every application is different and in some cases we'll find ourselves needing to tune. But chances are if you let us know the issues you're facing a release or two down the line you won't need to tune. Eventually it will just be us geeks who can't help it . Next step is a Solaris x86 vs Linux comparison. Stay tuned.

SPEC(R) and the benchmark name SPECjbb(TM) are trademarks of the Standard Performance Evaluation Corporation. Competitive benchmark results stated above reflect experiments performed by Sun Microsystems, Inc. For the latest SPECjbb2005 benchmark results, visit http://www.spec.org/osg/jbb2005.