BM Seer Unofficial thoughts from an anonymous Sun employee

« IBM saying two diffe... | Main | a while before more... »

Critical reading absolutely required for IBM

Friday Oct 05, 2007

First:

"First of all, the math is really simple. 4.7 is greater than 1.4. IBM's POWER6 4.7 GHz chip is faster than Sun's 1.4 GHz UltraSPARC T1 chip. And second of all, the IBM System p 570 remains the #1 SPECjbb2005 2-core result (1)."

Issues:

Did not compare system or chip performance but only quoted GHz of a chip.

Made a true statement about core count but ignored that that IBM core costs an order of magnitude more than Sun T1 on a per core basis

"Even more impressive, the processor bandwidth of the POWER6 chip – 300 gigabytes per second -- could download the entire iTunes catalog in about 60 seconds"

Issues:

Added every bandwidth (L3 cache, address bandwidth?!?,...) in a chip, even though peak memory bandwidth is limited to at least a 10th of that, delivered is a lot less.

stated "processor bandwidth", even though "delivered" system bandwidth would actually be required to move the data (not address

"IBM calculates that 30 SunFire v890s can be consolidated into a single rack of the new IBM machine, saving more than $100,000 per year on energy costs (3)."

Issues:

used 2 year old sun result compared to power6 yet to be shipped as of may press release

implied V890, so that people think it is a current comparison, had to read in the footnotes that it was 1.5 GHz slower CPU. Sun has introduced 1.8GHz, and 2.1GHz since.

made a "conservative" comparisons by giving IBM another 15% in performance

claimed Sun at 20% util and IBM at 60% util to get bogus 3x factor

never showed exactly what power was drawn by a 4.7GHz, 64GB memory system, at ??MHz DDR2 used in the comparison, etc.

...and many more.

"...And the good news is that about 40-70% of the stuff we do in performance tuning actually ends up helping end users,"

"This is feasible in the TPC-C benchmark because there are only five tables and only ten to fifteen columns in each table. In a more realistic application, where there are many more queries to be considered, the tables are typically much, much wider, in the 80 to 100 column range; and there are dozens if not thousands of tables. Then this kind of analysis(ed note: tuning) is no longer practical." -- Bruce Lindsay(IBM Fellow since '96), May 06, http://www.sigmod.org/sigmod/record/issues/0506/p71-column-winslet.pdf

"The idea is to get the numbers by hook and by crook." -- Bruce Lindsay(IBM Fellow since '96), May 06, http://www.sigmod.org/sigmod/record/issues/0506/p71-column-winslet.pdf

The TPC-C benchmark is an industry standard for measuring the ability of a system to process complex online transactions and large volumes of business data. The TPC-C benchmark is unique in the way it exercises all components of a system, including processors, memory, networking, storage, operating system and database software, demonstrating total system performance in a way that many of the other benchmarks touted by some competitors do not. -- Bruce Lindsay(IBM Fellow since '96), July 25, 2006, http://www-03.ibm.com/solutions/sap/doc/content/news/pressrelease/1623288130.html

Issues:

This means that 30% to 60% of IBM's TPC-C tuning is useless for customers.

IBM clearly over-hyped TPC-C, just 2-3 months after they publicly showed all of its problems and "optimizations" they used.

"Significantly, the high utilization rate of the System z9 mainframes -- systems can and do operate at 80 to 100 percent utilization -- combined with its ability to "virtualize" workloads, can enable a single mainframe processor to perform far more work than a single x86 processor running Microsoft Windows. The latter may run as low as 5 percent utilization."

Issues:

very convenient to use different work for mainframe and for its competitor.

"mainframes do operate at 80-100%", "x86 processor running Microsoft Windows. The latter may run as low as 5%". So it is a valid but totally useless statement.

"do" and "may" mean very different things

An equally invalid statement: x86 do operate at 80-100% and mainframes may run as low as 5%.

Moral: Be VERY VERY CAREFUL when you read big blue.

Disclosure statement:

TPC-C is a trademark of Transaction Processing Performance Council (TPC). More info www.tpc.org.

Like this post?