BEA has responded to my response to their latest press release, take a look at it here. First I agree with BEA on their first point. A valid JVM performance comparisons need to be on identical hardware, hopefully we're sharp enough to be sure we're comparing the same benchmarks :-) The response claims that the point of industry benchmarks is to get the highest possible score on like platforms. But this statement suggest a 32-bit JVM and a 64-bit JVM are one in the same. That unfortunately is not based in reality. The overhead of running a 64-bit application is substantial, and Sun has made targetted optimizations for the Sun Fire X4200 and AMD Opteron processors to achieve the level of performance shown in the 64-bit SPECjbb2000 benchmark results. The reality of SPECjbb2000 is that its been around for 5 years, and over those years several benchmark specific performance optimizations have been identified. Very few of these optimizations have any effect on real world customer applications. Does your application call System.gc() on a regular basis? Does your application call System.arraycopy() for 20% of the workload. Based on BEA's 32-bit SPECjbb2000 results they've done a good job targeting optimizations for this benchmark. So BEA, why don't you submit 64-bit SPECjbb2000 results on AMD Opteron? Or even better, submit more results on SPECjbb2005 the *New* industry standard JVM server benchmark. SPECjbb2000 retires on Jan. 4th, 2006. Which isn't too far away. Sun Hotspot is the world's fast JVM running SPECjbb2005. Hands down. SPECjbb2005 replaces SPECjbb2000 because it more closely models customer applications. Sun targets customer performance when prioritizing performance engineering work. Because of this, I am very happy to see BEA's challenge to test your applications with BEA and Sun JVMs. Please test your applications with Sun's JVMs (links listed below). Run your performance and *reliability* tests. Run Sun and BEA's JVM for days, not just a few minute comparison. I'm confident Sun's Hotspot will shine. If there is any problem whatsoever, contact us at the Performance Forum and Performance Project at java.net. Here's the latest J2SE 5.0 Update J2SE 5.0_05. Here's the latest Java SE 6 Mustang Developement Build.

Yes, its true. Sun Hotspot is the World's Fastest JVM and it scortches BEA JRockit. I feel I need to correct BEA latest press release on SOA and JVM performance. And yes! Not only is Sun Hotspot the fastest JVM on the Planet, I also have benchmarks to prove it. See BEA's press release here. BEA's press release and performance claims are based on SPECjbb2000 and SPECjappserver2004 results. Hotspot Performance SPECjbb2000 is an old, outdated benchmark. SPEC plans to replace SPECjbb2000 with SPECjbb2005 in the next few months. There are many problems with SPECjbb2000, so many it merits a blog entry on its own. Stay tuned, I plan on addressing this in the next few days. The biggest problem with SPECjbb2000 is the 32-bit space is its System.gc() before each measurement interval. This call effectively removes Full GCs from the measurement interval. This means that Full GC overhead is not part of the benchmark whatsoever. Because of this alone, optimizations specifically for SPECjbb2000 (also known as benchmark specials) have no real effect on customer applications. That's why Sun only publishes 64-bit SPECjbb2000 results. BEA's press release compares BEA JRockit 32-bit SPECjbb2000 results on a 4-way AMD Opteron 875 box to Sun Hotspot's 64-bit SPECjbb2000 results on similar hardware. They go on to say how Sun's result uses a 14GB heap and BEA's result uses a 1.8GB heap. They never specifically mention that the JRockit result is with a 32-bit JVM and Sun's result is with a 64-bit JVM. Sun is purposely running with a 14GB heap to highlight this. Now, yes, JRockit is quite fast running SPECjbb2000 on x86 hardware, and yes Sun has chosen not to compete in this space. We believe that benchmark specials don't benefit our customers and we are only interested in investing in real optimizations that will help every customer's application perform better. Now wouldn't you assume that if you claim you are the fastest JVM on the planet you'd be able to prove it with the latest benchmarks, not one which is 5 years old? (very, very old for Java software!). Here's the latest 2-way SPECjbb2005 results on 32-bit: 2 CPU 3.2 Ghz Xeon running BEA JRockit: 24208 bops 2 CPU Sun Fire X4200 running Sun J2SE 5.0_06: 27004 bops How is it that BEA's "World's fastest JVM" doesn't win running SPECjbb2005. Shouldn't BEA demonstrate winning performance on the most up to date benchmarks before making such a claim? Hotspot Scalability BEA also claimed that JRockit is the world's most scalable JVM. Again, doesn't make much sense when looking at the benchmarks. True JVM scalability is proven on today's largest systems. Sun has recently submitted record performance numbers on a 48-core Ultra SPARC IV+ E6900. The largest configuration JRockit has ever submitted SPECjbb2000 on is an 8-core Opteron box, and that was with a 32-bit JVM. True scalability challenges and bugs are hashed out on today's largest hardware, and 8-cores simply doesn't cut it. Sun has recently announced World record 16 and 24-core performance on SPECjbb2005. BEA has only been used in 2-core submissions to date. Sun Fire V890 Result Sun E4900 Result How can they possibly claim superior scalability? Where are their any benchmark results to support this claim? SPECjappserver2004 Performance Not much direct evidence to publically point out here, but I have as suspicion that BEA's own application server, running in Sun hardware, Solaris 10, and Sun J2SE 5.0_06 would perform better. Hold that thought for about two weeks, I'll come back to this later.

The following is the first edition of a series of articles on JVM performance tuning basics. The suggestions are generally applicable to J2SE releases starting with 1.3, I'll try to note when release defaults differ, and plan to give you many reasons why to upgrade to our latest releases. JVM Runtime Compilers – Client or Server? The Sun JVM has two runtime compilers. Here's a brief summary of each of the systems and when to use each. Client Compiler The client compiler is for use with client applications, generally short lived, where application startup time and memory footprint size are most important. There are many features in J2SE 5.0 address these requirements. A good example is class data sharing. The client compiler is the default option for J2SE 1.3 - 1.4.2, with a change in J2SE 5.0 with the addition of the server ergonomics (see below). When do I use the client compiler? Use the client runtime compiler when application startup and memory footprint are most important to you. How do I use the client compiler? J2SE releases 1.3 to 5.0: specify -client as the first JVM tuning option. Example: > java -client my.clientApp Note: The client compiler (-client) is the default, out of the box compiler choice for J2SE releases 1.3 to 1.4.2, and J2SE 5.0 on Windows (32-bit). Server Compiler The server compiler is targetted for long lived applications where runtime performance and throughput are essential, and startup speed and memory footprint are not. When do I use the sever compiler? Its important to use -server for all server applications. For older JVMs (1.3 – 1.4.1) be sure you're running the latest update release. Its also worth trying the server compiler for your long lived IDE's (Netbeans) or cpu intensive trading applications. Use cases where runtime performance is important are broad and include both client and server applications. How do I use the server compiler? J2SE releases 1.3 to 5.0: specify -server as the first JVM tuning option. Example: > java –server my.serverApp Note: The server compiler (-server) is generally not on by default and must be specified on the command line for J2SE releases 1.3 to 1.4.2. Server ergonomics changes this default in J2SE 5.0. J2SE 5.0 Server Ergonomics The server ergonomics feature in Tiger will modify default JVM tuning options and enable the server compiler if you're running on a server class machine machine. In the J2SE 5.0 a server-class machine has been defined as a machine with 2 or more physical processors and 2 or more Gbytes of physical memory. A single hyperthreaded cpu is counted as one physical processor. Upgrade to J2SE 5.0 There are many, many reasons to upgrade to J2SE 5.0 and to the latest update J2SE 5.0_05. See the J2SE 5.0 Performance Whitepaper for details on Tiger performance improvements. Below is a few of the features recently added in J2SE 5.0_05. Large Page Support for Solaris, Linux, and Windows. J2SE 5.0_05 adds large pages support to Linux and Windows. To enable on these platforms use: -XX:+UseLargePages. This feature is on by default on Solaris 10. For more details see: http://java.sun.com/docs/hotspot/VMOptions.html (bottom of the page) System.arraycopy optimizations on SPARC and x64 System.arraycopy optimizations first identified on x64 platforms have now been extended to include SPARC platforms as well. Synchronization Performance Improvements The J2SE 5.0_05 server compiler includes simple lock coarsening. This is the first phase of our uncontended synchronization performance improvements coming in upcoming update releases. Continued J2SE API Performance Improvements J2SE 5.0_05 includes many improvements to the J2SE API. This is part of Sun's continued Java performance worka nd you should expect to see more improvements in future updates as well.

Sun has chosen to only submit 64-bit SPECjbb2000 results because of limitations of the benchmark which have allowed our competition to make special case optimizations in the 32-bit case. JVM optimizations for this benchmark can only really apply to customer workloads using 64-bit JVMs with huge heaps. A good example of the special case optimizations is the SPECjbb2000 call for a full system GC during the initialization stage of the benchmark and before every measurment interval, effective removing Full GC performance from the benchmark. Notice the flags used for the JRockit and IBM JVMs for this benchmark: BEA: -XXfullsystemgc IBM: -Xcompactexplicitgc (see: http://www.spec.org for additional flag details.) These flags suggest special case optimizations for the unrealistic use case of SPECjbb2000. The BEA flags suggests enabling "a full system gc", and the IBM flags suggests "compact at a explicit gc call". Both special case for this benchmark. SPECjbb2000 is an old, unrealistic benchmark that is ripe for "benchmark special" optimizations. Sun is not interested in optimizing for benchmarks that have no real effect on our customers. We publish with 64-bit because it is a better match for environments where full gc's will be avoided (by using huge heaps), and therefore optimizations made in that space yield benefits for our customers as well. SPECjbb2005 on the other hand has addressed many of the benchmarks limitations. Explicit GC calls have been removed, allowing Full GC's to effect the performance during the measurement interval. XML and J2SE API's have replaced hand-rolled code. All in all the new benchmark has been successfully updated to model modern Java server applications. Please notice as well that BEA has only 1 public submission for SPECjbb2005 and IBM has none. Also notice that Sun Hotspot is the performance leader with SPECjbb2005. Why? Because the new benchmark actually attempts to model real customer applications through API use and GC behavior, and Sun's concentration has alway been on customer performance, not stunt JVM's for SPECjbb2000.