BM Seer Unofficial thoughts from an anonymous Sun employee

Benchmark Dreaming or simply wishing too hard?

Tuesday Mar 31, 2009

I had a dream two weeks ago...

  • IBM, HP, Dell, etc. would publish measured watts on all current SPEC benchmarks.
  • Oracle and IBM DB2 would publish results on TPC-E
  • HP, IBM, Dell would publish SPECpower on the same configurations they use for other SPEC benchmarks.
  • Vendors who use Intel-based processors, would only publish SPEC results with default BIOS, particularly on SPECjbb and SPECpower
  • That all vendors would make power-management software on by default.
  • Dell, IBM, HP, etc. would stop overhyping 15 year old benchmarks (like TPC-C)
... alas this hasn't come true yet. One can only hope that these other vendors would disclose as much information as Sun does on performance & watts.

disclosure statement:

SPEC, SPECjbb, SPECpower, reg tm of Standard Performance Evaluation Corporation. TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

I've been saying for years these things about the TPC-C benchmark!!! http://blogs.sun.com/bmseer/tags/tpc-c

We even know that IBM has tuned it useless: IBM's TPC-C "tuning"(?) that won't apply to anything in the real world

June 2005 Interview with Bruce Lindsay (IBM Fellow) at http://www.sigmod.org/sigmod/record/issues/0506/p71-column-winslet.pdf

"And the good news is that about 40-70% of the stuff we do in performance tuning actually ends up helping end users."

Who will have the maturity to kill TPC-C. Sun has quite publishing a while back, remember when Sun had the world record TPC-C and in that announcement Sun said: "It's well-understood in the technical communities that TPC-C no longer represents current customer workloads since the transaction load that its models are made of are small, primitive and disconnected transactions. While this model was acceptable for the workloads of the late 1980s, it misses the mark..." Sun's World Record TPC-C Press release, August2000

Disclosure Statement

TPC-C results referenced above was the fastest overall performance world record at August 31, 2000. Sun Enterprise 10000 server (Starfire) running Sybase Adaptive Server Enterprise (ASE), 156,873.03 transactions per minute (tpmC), $48.81 price/tpmC, available February 28, 2001. A full disclosure report and executive summary are available through the TPC Web site located at www.tpc.org.

[8] Comments
Like this post? del.icio.us | furl | slashdot | technorati | digg
Posted at 08:36AM Mar 31, 2009 in Benchmark Issues
Tags:
Comments:

While it seems your dreams are not coming true, it seems someone else's might be:

http://www.spec.org/power_ssj2008/results/res2009q1/power_ssj2008-20090310-00128.html

Posted by rick jones on April 01, 2009 at 08:51 AM PDT #

I would love to see both bmseer and IBM advocates to comment on Intel's Pat Gelsinger's widely publicised comment, on Nehalem EP's launch, that comparing against a Sun T2 server, a Nehalem box was half the cost and 1.7x faster; and compared to a IBM Power6 570, Nehalem was 1/10 the cost and 2.45x faster... the famous quote, "Comparing to the IBM Power environment, it is almost humorous.".

I know that these claims are worth nothing without full disclosure, perhaps the Sun and IBM systems are much superior in other benchmarks or in other qualities like reliability. It's also not surprising that a brand-new CPU is better than the competitor's CPUs from 1-2 years ago (comparing to Sun Rock should be more interesting if it's available soon). But I'm more interested in the "associative" comparison that Intel offers of Sun versus IBM: normalizing their numbers, IBM is 1.44x slower than Sun and 5x more expensive.

Posted by Osvaldo Pinali Doederlein on April 01, 2009 at 12:27 PM PDT #

Perhaps you should start dreaming about career opportunities once the IBM acquisition goes through...

Posted by Kevin Q on April 02, 2009 at 01:59 PM PDT #

Are you telling me that IBM isn't interested in getting important information to customers to make informed decisions?

I spend little time dreaming and lots of time working.

Posted by BM Seer on April 02, 2009 at 02:13 PM PDT #

Figures lie & Liars figure. But I guess from your perspective, Server market share is just an indication of the ignorance of the entire industry.

Posted by Kevin Q on April 02, 2009 at 03:16 PM PDT #

*NOW* I understand why IBM is rumored to be interested in buying Sun - it is all to provide "cover" for BM Seer and Ms Stahl to get hitched :)

Posted by rick jones on April 02, 2009 at 07:47 PM PDT #

Well now it seems that IBM is not anymore swallowing Sun (thanks god - I say that not as a Sun rooter but as a developer burned by WebSphere for years).

So the only outcome, for now anyway, is that IBM just made a public admission that Sun has several technologies/products that are superior to IBM's competing stuff. This is the only logical conclusion from facts like: IBM and Sun portfolios are a near-100% overlap, and Sun's client base and IP assets, or even some further control of Java, are valuable to IBM but certainly not worth an awesome $8bn price.

Posted by Osvaldo Pinali Doederlein on April 06, 2009 at 12:49 PM PDT #

Fascinating that many of your comments are about rumors on corporate things... and a couple of slams on me.

I was blogging about getting more valuable information to customers.

...guess that changing of subjects sometimes has a relation to the IP addresses where some comments come from...

Posted by BM Seer on April 07, 2009 at 07:03 AM PDT #

Post a Comment:
Comments are closed for this entry.