BM Seer Unofficial thoughts from an anonymous Sun employee

« New New News | Main | Solaris10 and the... »

Sun head-to-head wins again: SPECjbb2005

Wednesday Jan 25, 2006

The Sun Fire E6900 and Solaris10 (reminder that system has a maximum of 24 chips, get ready to see what it beats...) with 1.5GHz US-IV+ achieved World Record performance on the SPECjbb2005 server-side Java benchmark for up to 32 processor chips. Yep, Sun's 24-chip beats Fujitsu 32-chip system!

The Sun Fire E6900 beat the recently announced result from Fujitsu for the PRIMEQUEST 480 with Itanium 2 by 6%.
The 24-way Sun Fire E6900 with 1.5 GHz US-IV+ outperformed the 32-way Fujitsu PRIMEQUEST 480 with 1.6GHz Itanium 2 by 42% on a per processor basis.
The 24-way Sun Fire E6900 with 1.5 GHz US-IV+ outperformed the 16-way IBM System p5 570 with 1.9 GHz POWER5 by 40%.
But why no HP results? No 32-core IBM results? No 64-core IBM results?
See for Previous entry on Sun Fire T2000 SPECjbb2005 performance

See Dave Dagastine's Blog for other info

Example Disclosure Statement:

SPECjbb2005 Sun Fire E6900 (24-way, 24 chips, 48 cores) 342,578 bops, 28,548 bops/JVM submitted for review, Fujitsu PRIMEQUEST 480 (32 chips, 32 cores) 322,719 bops, 40,340 bops/JVM, IBM eServer p5 570 (8 chips, 16 cores, 16-way) 244,361 bops, 30,545 bops/JVM . SPEC, SPECjbb reg tm of Standard Performance Evaluation Corporation. Results as of 01/23/06 on www.spec.org.

SPECjbb2005 Benchmark Description, in case you were wondering...

SPECjbb2005 (Java Business Benchmark) measures the performance of a Java implemented application tier (server-side Java). The benchmark is based on the order processing in a wholesale supplier application. The performance of the user tier and the database tier are not measured in this test. The metrics given are number of bops (Business Operations per Second) and bops/JVM (bops per JVM instance).

...some vendors have made a big stink about number of JVMs (then started publishing on multiple-JVMs), if you want to really dig and see what the issues (or non-issues) are click here.

...next week more wins

[12] Comments

Like this post?

Posted at 12:19PM Jan 25, 2006 in Benchmarks
Tags: e6900

Comments:

Now that's real FUD! :) Why compare apples to oranges? Does it really make sense to compare $378,000 p5 570 with only 16 cores to $1,091,490 E6900 with 48 cores? 40% more performance with 3x more cores - I wouldn't call it 'beating' especially taking into account 2.8x price difference. This really makes difference if you run Oracle.

Posted by Mike on January 25, 2006 at 08:18 PM PST #

You need to check your prices, my friend --- and please add the IBM DDR2 memory that p5 570 systems costs a lot more than you think like double the IBM cost!!! IBM benchmarks DDR2 and prices DDR1 memory for many things. Let's look at your core argument. How about a 64 core IBM p595 with DDR2 memory. That IBM puppy costs $6.7M !!! Let's look at a 144 core Sun E25K (72 chips) that costs $4.1M. So don't let core count be any influence on how systems compare. My friends, any time you see IBM doing silly comparisons on core count, quickly ask if it is DDR2 pricing, and quickly look at system prices. Otherwise you'll be fooled like our friend, Mike. Also we really would love to see the IBM 32-core p5 590 JBB2005 or the IBM 64-core p5 595 JBB2005, if you have any influence Mike. If anyone needs to see how IBM p5 590 or IBM p5 595 compare on performance take a look at: http://blogs.sun.com/roller/page/bmseer#solaris10_and_the_big_and If you are too lazy to look the conclusion is the Sun E25K and Sun E20K are faster, cheaper, and .... better!

Posted by BM Seer on January 26, 2006 at 02:26 PM PST #

We'll keep covering this topic, remember the early posting: http://blogs.sun.com/roller/page/bmseer?entry=ibm_perf_on_ddr2_but

Posted by BM Seer on January 26, 2006 at 02:40 PM PST #

You are completely right about the price point. p 570 is priced with 32GBs, and each additional 32GBs cost 103,000$, so we have to add 206,000 to 378,000 which equals 584,000. Still $167490 cheaper than even 'MEDIUM' (with only 64GBs) config E6900. Where am I being fooled again? p5 595 should be quite affordable, if we can say so about such a hardware. If you don't 1024GB of RAM, which usually costs megabucks. And I don't get the point why per core performance comparison is silly if we license software per core. I'd love to see more p5 590/595 benchmarks, too as well as any Sun TPC-C results at all. I know the limitations of TPC-C but in my opinion it serves its point to roughly compare system throughput acceptably. If you don't like TPC-C, SAP SD is more 'real-life' scenario, I think. Regarding E25K vs p5 595 battle: I'd love to see E25k SAP SD 2-tier result with latest US IV+. Current standing is like this: p5 595 with only 64 cores does 100700 SAPS ( 20000 users) and E25K with 72 US IV (I assume that means 144 cores) only 51070 SAPS (10175 users). A whopping 97% difference in performance with E25K having 2.25x core count. Hopefully US IV+ will cure this.

Posted by Mike on January 26, 2006 at 08:05 PM PST #

I am glad you are able to congratulate yourself because I don’t think any customers will be congratulating you anytime soon. It looks like HP is the only company smart enough to stay out of this benchmark game, with no relevance in the real world. Maybe HP is spending time working on real customer problems instead of playing with fake benchmarks. I know that in this SPEC loving group it would be wrong to say this but running SPEC JBB on these huge servers (16,32,48 cpu) is totally useless. You can’t find a system in production that uses 48 processors to execute a business application without doing any network or disk IO. JBB pretends to measure the server-side performance of Java runtime environments but it is not at all representative of a real workload. Running unrealistic workloads to measure performance is a disservice to customers. How many customers do you have who run java applications on a 48-way machine that consume all the CPUs but performs no IO? SPEC JBB is like a very fast moving bus that never has to stop to pickup and drop passengers, not exactly useful. By the way since you always wonder why HP is not publishing JBB I guess it would be reasonable to ask you why you think they are not publishing. Are you suggesting that HP is not capable of running the benchmark? In my opinion HP does not want to give credit to a bad benchmark by publishing results. Why should they give you the satisfaction of jumping off the bridge after you? Clearly HP thinks the benchmark is not important.

Posted by Robin on January 26, 2006 at 09:46 PM PST #

Mike: you missed the point on the p5 595, I wasn't talking about 1TB of memory. The comparison is 64-core IBM p595 with 1/2TB DDR2 memory. That IBM puppy costs $6.7M !!! 144-core Sun E25K (72 chips) 1/2TB of memory that costs $4.1M. So don't let core count be any influence on how systems. Also a E6900 is a larger and more functional system than a IBM p5 570, Sun has other things further down the price point.

Posted by 192.18.42.10 on January 31, 2006 at 10:16 AM PST #

Also to your point of licensing software per core, yes IBM loves to do this, but even Oracle has changed the pricing structure, now an E25K 144 cores/72-chips full up with Oracle costs less than a full up (64-cores)P5 595 (that remember is the one E25K beat in this an other benchmarks). Sun E25K has less hardware costs (see previous comment), less Oracle costs (see you Sun or Oracle sales person), and has more cores --- cores don't matter my friend!

Posted by BM Seer on January 31, 2006 at 10:21 AM PST #

TPC-C, wasn't that created during the Johnson administration :) No seriously it was 1992 (Lawerence Welk was still alive, CD's for the first time outsold CDs, it was the start of text based browsers, and it wasn't until the end of that year that CERN said the few computers on the WWW were reasonably reliable). So how is TPC-C still credible with 7 or so simple queries on 9 tables? (ok the numbers may be off but neither of them are double digits!). See the posting: http://blogs.sun.com/roller/page/bmseer?entry=tpc_c_how_old_is "It's well-understood in the technical communities that TPC-C no longer represents current customer workloads since the transaction load that its models are made of are small, primitive and disconnected transactions. While this model was acceptable for the workloads of the late 1980s, it misses the mark..." Fun to think when this was stated, can you guess? Clearly it must have been stated by someone with sour grapes, but you'd be wrong. It was stated in a press release that announced Sun's World Record TPC-C result in August 2000! Over 5 years ago! Take a look at the press release. http://www.sun.com/smi/Press/sunflash/2000-08/sunflash.20000831.1.html You'll also notice it said, "Customer workloads nowadays require a more ad hoc workload than the TPC-C specifies." you are not likely to see a TPC-C result since it doesn't show much. More US-IV+ results to come, let's quite comparing to US-IV (Sun has shown 1.8x to 2x performance boost so expect at least that - probably more when we use the latest SW).

Posted by BM Seer on January 31, 2006 at 10:27 AM PST #

Actually the interesting thing is HP did participate fully in SPECjbb2000 in November of 2005. http://www.hp.com/hpinfo/newsroom/press/2005/051101b.html and of course October 2005. http://www.spec.org/jbb2000/results/res2005q4/jbb2000-20051025-00410.html ... as Dave Dagastine has pointed out SPECjbb2005 is a lot better workload than SPECjbb2000 http://blogs.sun.com/roller/page/dagastine?entry=specjbb2005_world_record_hotspot_and These benchmarks do play a role in improving product, and they do help real-world performance. I had a database company once tell me they didn't need more than 8 CPUs and 64GB of memory ever, wow have times changed. So we lead with the benchmarks and then all the good things trickle down to make everything more efficient. So if HP is not in the SPECjbb2005 game, then are they really stressing the OS, JVM, and hardware to make sure it works for real customers? I gotta wonder. jbb2005 only looks at the application tier and focuses on it, so no it doesn't do a lot of disk IO (that usually in most 3-tier datacenters is handled by the database servers). I don't know why HP isn't running the benchmark, it is simple to run, shows java application performance, it would help show Itanium or PA-risc performance, it would help show the performance of their systems... OK Sun will just have to compare to HP servers on TPC-H, SPECint_rate, Informatica, SAS, SPECjAppserver, etc. ...or as you say at customer sites were we are actually getting better comparisons than on standard benchmarks.

Posted by 192.18.42.11 on January 31, 2006 at 11:22 AM PST #

I was surprised to see HP mostly showing old benchmarks (3 of the 5 on this page) are EOL'ed with never versions to replace them. http://www.hp.com/products1/servers/integrity/performance.html ..and another is a 14 year old benchmark (TPC-C), and then the 5th is the old version 4.7 which isn't accepting any more results, because everyone is posting on SAP version 5.0? When will we see something new? I guess SPECjAppServer, but the Sun T2000/E6900 bested HP recently: http://blogs.sun.com/roller/page/bmseer?entry=new_new_news

Posted by BM Seer on January 31, 2006 at 11:32 AM PST #

You are saying that “jbb2005 only looks at the application tier and focuses on it, so no it doesn't do a lot of disk IO (that usually in most 3-tier datacenters is handled by the database servers)”. I assume that you will agree that in a 3-tier setup the application server has to communicate with the database server as well as the devices/users making request to the application server layer. This means that you have some network activity to keep the application server going. Jbb2005 is not a balanced “system” benchmark that exercises the relevant system components. Jbb2005 is simply a test of the JVM implementation – if you want to measure the performance of different JVM implementations or different version of the same JVM then jbb2005 is a reasonable benchmark to use. To push jbb2005 as a system benchmark is a bit of a stretch. TPC-C has both network and disk IO so it provides a system performance gage while exercising the OS, CPU, memory, IO subsystem, interrupt handling and so on. The balance between the various resource utilization may not be the same as a real customer workloads but at least it uses all the resource of the system. JBB on the other hand just uses the CPU to push memory around without the disruption from the pesky IOs. By the way, TPC-H is the same benchmark as TPC-D with some updated run rules. TPC-D/TPC-H workload (the queries and the schema) have been around since 1995 – that is 11 years old. If TPC-C is bad because the transaction and the schema are old then TPC-H is no better since the queries and the schema are old. In fact I think TPC-H has eight tables which is even fewer tables than TPC-C. How can you like TPC-H and not like TPC-C? Do you think TPC-H is representative of real customer workloads? You make up stories to justify your position. TPC-C has 9 tables so it must be a stupid benchmark and TPC-H has 8 tables so it must be a good benchmark (your stories are not very good). The truth is that you only like benchmarks that you are good at and dislike the benchmarks you are bad at – there is no shame in that – you are only a hw peddler.

Posted by robin on January 31, 2006 at 08:34 PM PST #

I'm glad you recognize that the application tier doesn't have lots of IO (I assumed you were confused and not just creating FUD). Yes, the application tier has network traffic, SPECjbb doesn't add that in. So it is just a approximation of a workload, welcome to the world of benchmarking. So we've done other workloads with network & application of course - we don't see a huge impact, Solaris just handles it. Sure jbb2000 alone isn't a good way to judge "a system" --that is why Sun' publishes a lot of benchmarks. Again on TPC-C, remember Sun established a world record on TPC-C in 2000 and then diminished it in the same press release talking about its shortcomings (which are a *lot more* than a small number of tables and queries, see previous posts). So Sun was best on this benchmark and fully acknowledged its problems when we were on top, over 5 years ago. Then Stopped publishing on this one benchmark, so your criticism of Sun only publishing on benchmarks we are good as totally *not* correct (again see the large variety we publish on). I'm still confused why HP published SPECjbb2000 at the end of 2005 and didn't continue submit on the improved SPECjbb2005 workload? We fully realize that these benchmarks are just one indication, we see the same advantages at lots of customer in their real-word problems, so Sun's benchmarks results and performance leadership are reasonable and reflecting things important to customers.

Posted by BM Seer on February 01, 2006 at 12:01 PM PST #