IBM too tricky for good of others?
Thursday Feb 22, 2007
IBM's TPC-C results not worthy of belief? Lots of unrealistic
optimisations? Sometimes you never know what you find when you start searching the web. After yesterday's posting I started looking. Here is info from June 2005 IBM interview: (who knows what they've done since that doesn't benefit users?)
http://www.sigmod.org/sigmod/record/issues/0506/p71-column-winslet.pdf
-
"And the good news is that about 40-70% of the stuff we do in performance tuning actually ends up helping end users. " Bruce Lindsay, IBM fellow
Ouch! Sun aims for benchmark tuning that end users actually use! Does this explain IBM's over-inflated TPC-C results?
-
Q: "Is there any particularly sneaky but still totally legal aspect of
TPC-C tuning that you would like to mention?"
A: "Well, we do things that are very, what should I say? Intense. We get down to the level of worrying about the physical column order in the table so the reference columns are near each other, minimizing cache misses during fetching. This is feasible in the TPC-C benchmark because there are only five tables and only ten to fifteen columns in each table. In a more realistic application, where there are many more queries to be considered, the tables are typically much, much wider, in the 80 to 100 column range; and there are dozens if not thousands of tables. Then this kind of analysis is no longer practical." Bruce Linsay, IBM fellow
Good reason to make benchmarks messy and change them often. Is this why IBM hasn't published SPECint_rate2006 because they can't do the above?
We were right with these past postings:
http://blogs.sun.com/bmseer/entry/ibm_continues_to_abuse_and
http://blogs.sun.com/bmseer/entry/selective_vision
...interesting...
|
|
|
|
Interesting document. More quotes:
"Th...