BM Seer Unofficial thoughts from an anonymous Sun employee

Lack of proper information hurts consumers, Computer vendors wake up!

Tuesday Mar 25, 2008

Not having power measurements on every benchmark is a very bad idea. SPEC should simply add power measurements to every SPEC benchmark as the benchmarks are currently designed. But if they have a separate SPECpower benchmark then they should require critical configuration details (GB & GHz) be specified in vendor claims (where the claim is written not on a link somewhere). See yesterday's posting on HP DL580 G5 where HP didn't mentioned CPU GHz and memory size in their brochure).

SPEC states:

    Currently, many vendors report some energy efficiency figures, but these are often not directly comparable due to differences in workload, configuration, test environment, etc.

SPEC members should read their own mission carefully and make sure that vendors provide proper information with the most important that use watts (like memory size & GHz) up front!

Look what you can do with not mentioning important information, let's look at the following claims about a car...

  • a) Dodge Charger gets 18/26 mpg due to official testing.
  • b) Dodge Charger top speed over 244 mph.
Sounds great I'll get one because both statements are true... or is this being misleading? One might assume they are basically the same car, but unfortunately only the nameplate is the same. It is very misleading unless you know that one is the street car and one is the NASCAR "Car of Tomorrow" (COT). It would have been very clear to the consumer if it was written as follows:
  • a) Dodge Charger (2.7L V6 190hp) gets 18/26 mpg due to official testing.
  • b) Dodge Charger (5.86L Pushrod V8 ~865hp) top speed over 244 mph.

SPEC there are only two more things to add to a disclosure statement (GHz & amount of memory)! By the way...

  • a) Dodge Charger (2.7L V6 190HP) gets 18/26 mpg due to official testing, MSRP: $21,320 - $36,355.
  • b) Dodge Charger (5.86L Pushrod V8 ~865hp) top speed over 244 mph. cost?
  • c) Dodge Charger (diecast 1/24 scale) $67.99
The consumer should be provided information in a form that makes a difference to them. SPEC you owe us more. HP, oh I can't even say...

I should have just stayed at lunch today. argggh.

disclosure statement:

SPEC, SPECpower reg tm of Standard Performance Evaluation Corporation. Results on SPEC benchmarks can be found at www.spec.org.

[2] Comments
Like this post? del.icio.us | furl | slashdot | technorati | digg
Posted at 10:58AM Mar 25, 2008 in Benchmark Issues
Tags:
Comments:

I don't see how this is any different than any other SPEC benchmarks, where systems are tested with many different processors. The disclosure rules for SPECpower_ssj2008 are already cumbersome; adding more rules would make it so only lawyers could follow them (and bloggers could ignore them and claim ignorance). I happen to agree with Mr. Jones that this isn't SPEC's problem, the cut line is blurred, and the "important" things will vary over time.

If you really think it's such a huge issue, why not take it up with SPEC's president, a Sun employee? www.blogs.sun/wbays

Posted by C on March 26, 2008 at 08:22 PM PDT #

SPEC's job is to make SPEC benchmarks useful to customers (not vendors).
Adding CPU GHz & memory size is easy (5 words) and would prevent the confusion above. CPU GHz and memory are important to watts. If they NOT then "C" why don't you post some data and prove me wrong?

This is such a huge matter that I'm taking it public so a thousand people a day can see it and I hope they will demand vendors publish power-performance on every benchmark. Sun publishes measured watt-performance on a volunteer basis on all UltraSPARC T1/T2 SPEC benchmarks. No other vendor rises to this challenge. Why?

I've been screaming about getting power measurements on every SPEC benchmark in public for a very long time. I've complained to all kinds of people inside Sun.

Clearly you know a lot about SPEC, if you are a SPEC member you know reporting rules for each benchmark are decided in the committee by members (vendors with their agendas), and *not* the President of SPEC (whose role is to deal with other matters).

Posted by BM Seer on March 27, 2008 at 01:44 PM PDT #

Post a Comment:
Comments are closed for this entry.