BM Seer Unofficial thoughts from an anonymous Sun employee

« Bottled water vs.... | Main | SPEC APC SolidWorks... »

careful reading shows a lot

Wednesday Oct 24, 2007

You have to read some things carefully

"...And the good news is that about 40-70% of the stuff we do in performance tuning actually ends up helping end users,"

"This is feasible in the TPC-C benchmark because there are only five tables and only ten to fifteen columns in each table. In a more realistic application, where there are many more queries to be considered, the tables are typically much, much wider, in the 80 to 100 column range; and there are dozens if not thousands of tables. Then this kind of analysis(ed note: tuning) is no longer practical." -- Bruce Lindsay(IBM Fellow since '96), May 06, http://www.sigmod.org/sigmod/record/issues/0506/p71-column-winslet.pdf

"The idea is to get the numbers by hook and by crook." -- Bruce Lindsay(IBM Fellow since '96), May 06, http://www.sigmod.org/sigmod/record/issues/0506/p71-column-winslet.pdf

The TPC-C benchmark is an industry standard for measuring the ability of a system to process complex online transactions and large volumes of business data. The TPC-C benchmark is unique in the way it exercises all components of a system, including processors, memory, networking, storage, operating system and database software, demonstrating total system performance in a way that many of the other benchmarks touted by some competitors do not. -- Bruce Lindsay(IBM Fellow since '96), July 25, 2006, http://www-03.ibm.com/solutions/sap/doc/content/news/pressrelease/1623288130.html

Issues:

This means that 30% to 60% of IBM's TPC-C tuning is useless for customers.

IBM clearly over-hyped TPC-C, just 2-3 months after they publicly showed all of its problems and "optimizations" they used.

"Significantly, the high utilization rate of the System z9 mainframes -- systems can and do operate at 80 to 100 percent utilization -- combined with its ability to "virtualize" workloads, can enable a single mainframe processor to perform far more work than a single x86 processor running Microsoft Windows. The latter may run as low as 5 percent utilization."

Issues:

used different work for mainframe and for its competitor.

"do" and "may" mean very different things

"mainframes do operate at 80-100%", "x86 processor running Microsoft Windows. The latter may run as low as 5%". So it is a valid but totally useless statement.

An equally invalid statement: x86 do operate at 80-100% and mainframes may run as low as 5%.

"First of all, the math is really simple. 4.7 is greater than 1.4. IBM's POWER6 4.7 GHz chip is faster than Sun's 1.4 GHz UltraSPARC T1 chip. And second of all, the IBM System p 570 remains the #1 SPECjbb2005 2-core result (1)."

Issues:

Did not compare system or chip performance but only quoted the GHz of a chip?

Made a true statement about core count but ignored that that IBM cores cost much more than Sun UltraSPARC T1 and/or UltraSPARC T2 on a per core basis, I know this is hard to verify since IBM isn't public about pricing, so you'll have to ask your IBM people to price specific configurations for you, be specific so you understand exactly what is priced.

"Even more impressive, the processor bandwidth of the POWER6 chip – 300 gigabytes per second -- could download the entire iTunes catalog in about 60 seconds"

Issues:

Added every bandwidth (L3 cache, address bandwidth?!?,...) in a chip, even though peak memory bandwidth is limited to at least a 10th of that, delivered is a lot less.

stated "processor bandwidth", even though "delivered" system bandwidth would actually be required to move the data (not address

"IBM calculates that 30 SunFire v890s can be consolidated into a single rack of the new IBM machine, saving more than $100,000 per year on energy costs (3)."

Issues:

used 2 year old sun result compared to power6 yet to be shipped as of may press release

said V890, so that people think it is a current comparison, had to read in the footnotes that it was 1.5 GHz slower CPU. Sun has introduced 1.8GHz, and 2.1GHz since.

made a "conservative" comparisons by giving IBM another 15% in performance

claimed Sun at 20% utilisation and IBM at 60% utilisation, that is one way to get 3x over your competition

never showed exactly what power was drawn by a 4.7GHz, 64GB memory system, at ??MHz DDR2 used in the comparison, etc.

This was a bit of a repeat, but some things should not be forgotten.

I've never been about popularity or names. You don't need my expertise to see funny things in IBM's statements. Don't attack me, attack the facts. Anonymously yours, Sun's BM Seer.

Disclosure statement:

TPC-C is a trademark of Transaction Processing Performance Council (TPC). More info www.tpc.org.

[1] Comments

Like this post?

Posted at 03:14PM Oct 24, 2007 in Benchmark Issues
Tags: cmt ibm power6 t1 t2 tpc-c ultrasparc

Comments:

Oh boy you make me laugh with all the crap that you find from Sun's competitors! Geez, comparing a v890 with not-yet-released hardware *and* giving the not-yet-released hardware an assumed boost? Are IBM's customers really that stupid they'd actually believe that? Geeeeez!

Keep up the good work.

Posted by James McPherson on October 25, 2007 at 07:17 AM PDT #