BM Seer Unofficial thoughts from an anonymous Sun employee

judging by the wrong things: IBM & TPC-C

Tuesday Feb 20, 2007

Is IBM 3.3x or 1.4x faster? - I guess it depends if you use a over-optimised benchmark like TPC-C. As mentioned yesterday, IBM doesn't publish on a variety of standard benchmarks like SPECint_rate2006 or SPECjbb2005 on their high-end systems so we have to look at the SPECint_rate2000 which is just about to be EOL'ed and completely replaced by SPECint_rate2006.

First let's compare an IBM p5 595 (Power5+ 2.3GHz 64p, 128thread) to a HP Integrity Superdome (Itanium2 1.6 GHz 64p, 64thread, single core/CPU) on SPECint_rate2000.

Constructing a SPECint_rate2000 ratio
1.4x = 1513/1108
we find that the IBM 595 is 1.4x faster, it makes sense because this isn't the latest HP dual-core Itanium2. Both IBM and HP systems have results on TPC-C U SPECint_rate2000.

OK now using TPC-C, let's compare a IBM p5 595 (Power5+ 2.3GHz 64p, 128thread) to a HP Integrity Superdome (Itanium2 1.6 GHz 64p, 64thread, single core/CPU).

Constructing a TPC-C ratio
3.3x = 4033378/1231433
what?
comparing the same systems the IBM is 3.3x faster ?!? Looks that TPC-C over-inflates what can be expected from IBM.

My guess is IBM over-optimised and played lots of tuning tricks on TPC-C, correct? So is TPC-C relavent to customers if this is the case?

...maybe that's why seven years ago Sun, upon publishing a world record TPC-C result said:

"It's well-understood in the technical communities that TPC-C no longer represents current customer workloads since the transaction load that its models are made of are small, primitive and disconnected transactions. While this model was acceptable for the workloads of the late 1980s, it misses the mark..."
http://www.sun.com/smi/Press/sunflash/2000-08/sunflash.20000831.1.html

You'll also notice the Aug 2000 press release said, "Customer workloads nowadays require a more ad hoc workload than the TPC-C specifies."

Disclosure Statements

IBM p5 595 (Power5+ 2.3GHz 64p, 128thread) 4,033,378 tpmC, 2.97 US $/tpmC, Avail 01/22/07, IBM DB2 9, IBM AIX 5L V5.3, Microsoft COM+. HP Integrity Superdome (Itanium2 1.6 GHz 64p, 64thread), 1,231,433 tpmC, 4.82 US $/tpmC, Avail 06/05/06, Microsoft SQL Server 2005 Enterprise Edt SP1, Microsoft Windows Server 2003 Datacenter Ed.(64-bit)SP1. Results as of 2/15/07, see http://www.tpc.org.

IBM System p5 595 (Power5+ 2.3GHz 64p, 128thread), 64 cores, 32 chips, 2 cores/chip (SMT on), 1513 SPECint_rate2000. HP Integrity Superdome (Itanium2 1.6 GHz 64p, 64thread, 16 cells), 64 cores, 64 chips, 1 core/chip, 1108 SPECint_rate2000. SPEC, SPECint, SPECfp reg tm of Standard Performance Evaluation Corporation. Results from http://www.spec.org. as of 2/15/07.

World record TPC-C results referenced above was an overall performance world record at August 31, 2000. Sun Enterprise 10000 server (Starfire) running Sybase Adaptive Server Enterprise (ASE), 156,873.03 tpmC, $48.81 price/tpmC, available February 28, 2001. A full disclosure report and executive summary are available through the TPC Web site located at http://www.tpc.org.

[7] Comments