BM Seer Unofficial thoughts from an anonymous Sun employee

IBM real & real

Wednesday Aug 15, 2007

disingenuous = "giving a false appearance of simple frankness".

IBM bloggers try to make valid SPEC estimates (preliminary results) look like "fake", they keep using words like "virtual" "not real" and "strange", then for IBM they use word like "real" 7 times.

All UltraSPARC T2 SPEC CPU and SPEC OMP metrics quoted are from full “reportable” runs, but are nevertheless designated as “estimates” because they use pre-production systems. Sun customer systems, to be announced later, are expected to perform similarly. SPEC rules do allow comparing these preliminary scores and published result.

For details see:
"Estimated" what does that mean for Sun's UltraSPARC T2, which is at:
http://blogs.sun.com/bmseer/entry/estimated_what_does_that_mean

IBM bloggers also imply that IBM doesn't estimate results, I was seeing lots of IBM POWER6 estimates from IBM 2 years ago and they were estimates not based on runs.

SPEC, SPECint, SPECfp, and SPEComp registered trademarks of Standard Performance Evaluation Corporation. Results from www.spec.org as of August 6, 2007.

[8] Comments
Like this post? del.icio.us | furl | slashdot | technorati | digg
Posted at 02:28PM Aug 15, 2007 in Benchmark Issues
Tags:
Comments:

Let me add another definition:

Fear (noun):
1: an unpleasant often strong emotion caused by anticipation or awareness of danger
2: anxious concern
3: reason for alarm

I think this explains much of IBM's comments. And it is not just Niagara 2, it is the promise of CMT delivered by Niagara 2, Victoria Falls, and Rock in the SPARC space, as well as AMD Barcelona, Shanghai, and Montreal, and Intel's future Xeons in the x86 space.

POWER6 is looking less and less relevant in the new multicore world. POWER6 will not be a volume chip.

Posted by Mark on August 15, 2007 at 08:21 PM PDT #

Oh, I think the POWER6 is a fine bit of engineering, but I did
optimize it inthe wrong ways and really made it pretty expensive.
Then, I think that IBM decided to try to use silly activation fees
and focusing on per core performance -- all in an effort to market
around their per core hardware costs.

Posted by BM Seer on August 15, 2007 at 11:19 PM PDT #

There is no reason that numbers taken on pre-production systems _have_ to be labeled estimates so long as:

*) the configuration will be generally available within 3 months from the date of first public disclosure
*) the entity publishing the numbers is willing/able to provide copies of the full disclosure report(s)

Posted by rick jones on August 16, 2007 at 09:04 AM PDT #

OK, then why the many Intel and AMD "estimated" results?

Actually, and I don't sit on any SPEC committee, I don't know why
they allow preliminary, but they do and others have lead the way for a long time.

Posted by BM Seer on August 16, 2007 at 04:05 PM PDT #

That would be a question for either Intel or AMD. Or now even Sun :) Why would anyone releasing estimated results do so? Perhaps the actual general availability dates for the whole kit and kaboodle were more than three months out; in some cases perhaps they didn't want to provide full disclosure reports which might reveal details they didn't want others to see;perhaps they were hedging their bets in case production units weren't as fast and they didn't want any "NCs" left in www.spec.org; in other cases, perhaps the estimates were actually estimates based on analytic models and expectations for future CPUs/compilers/chipsets. Perhaps entirely different reasons.

While estimates are allowed (for some SPEC benchmarks), results actually reviewed by SPEC and published on www.spec.org are better.

Posted by rick jones on August 16, 2007 at 04:38 PM PDT #

Here are some IBM (estimates from July 13, 2004):

2 - Based on estimated SPEC OMPM2001 results
as of July 13, 2004. IBM 4-way 1.65 GHz
POWER5 p5-550 result of 9,884 (est.) submitted
July 12, 2004, vs. hp server rx7620 1.5 GHz
Itanium® 2 result of 6,886. See
http://www.specbench.org/omp/results/ompm2001.html

3 - Based on estimated SPEC OMPM2001 results
as of July 13, 2004. IBM 2-way 1.65 GHz
POWER5 p5-520 result of 5,228 (est.) submitted
July 12, 2004, vs. hp server rx2600 1.0 GHz
Itanium 2 result of 2,637. See
http://www.specbench.org/omp/results/ompm2001.html

Here are some HP:
from HP (estimated SPECs: 13 SPECint95 and 18 SPECfp95).

yeah... Sun does uses the same clause that others have been doing
for years, and you give me heat????

I agree that submitted are best. I'm sure you'll see more.

SPEC, SPECint, SPECfp, and SPEComp registered trademarks of Standard Performance Evaluation Corporation. Results from www.spec.org as of August 6, 2007.

Posted by BM Seer on August 16, 2007 at 05:22 PM PDT #

The "heat" was as much for implying that SPEC requires runs made on pre-production systems to be estimates as anything else. As for the IBM/HP estimates, looks like the IBM ones were saying they had also submitted to SPEC.

Posted by rick jones on August 16, 2007 at 07:12 PM PDT #

that should read "have to be estimates"

Posted by rick jones on August 17, 2007 at 11:41 AM PDT #

Post a Comment:
Comments are closed for this entry.