BM Seer Unofficial thoughts from an anonymous Sun employee

SPECpower_ssj2008 power-benchmark needs work

Tuesday Dec 18, 2007

A new SPEC benchmark called SPECpower_ssj2008 was announced last week.. I've taken a little time to look at the new results and benchmark rules.

Quite simply, I don't know why SPEC didn't add power measurement to every benchmark. It is quite easy to add rules that puts one SUT (system under test) on a power meter and a few rules to accurately measure power. That should have been done months ago, it is so much easier than creating a new benchmark.

So why didn't SPEC do this? I have no idea. I can only guess some vendors to show off their power-management software at different utilisation levels. As we've talked about here for a very long time, running servers at low-utilisation (below 50% utilisation) is the worst way in the world to waste power... Even with the most extreme power-saving HW/SW.

With a different benchmark only for power-benchmarking it is really possible to game results. Did anyone?

  • HP submitted a SPECpower_ssj2008 on the DL 580 G5 QC Xeon. Cool now we can see what the DL580 G5 does. But HP PICKED 1.86GHz Xeon L7345 for SPECpower_ssj and a very DIFFERENT processor (2.9GHz Xeon) for performance on SPECweb2005, well at least they are not confusing things...
  • HP's DL 580 G5 SPECpower_ssj2008 documentation does NOT EVEN mention processor GHz or type! Why?
  • The only SPECpower_ssj2008 results were on 4GB, 8GB, 16GB configs. Memory is a huge power draw. So why configure such small systems for power? Why do these vendors use 32GB and 64GB configs on performance benchmarks?
  • The HP DL580 G5 used one 60GB 5400 RPM SATA drive. My laptop has better :)
  • HP's DL580 G5 Power supplies used are 2x1200 watts. The 1.86GHz Xeon DL580 G5 draws 387 watts at 100% on SPECpower_ssj2008. Since all vendors know that you use power supplies near their rating for best energy efficiency, why does HP not sell properly-size power supplies? Maybe because HP typically expects to sell configurations that are different than SPECpower_ssj2008 DL580 configs?
  • Is running at low utilisation efficient? SIMPLE answer NO! Let's look at SPECpower-ssj2008 HP DL580 G5. Comparing 70% utilisation with 20% utilisation we see that one wastes 3 TIMES more watts for the same amount of work at 20% utilization than at 70% utilization!!! 3 = round(2.93) = (294w/71409ssj_ops) / (359w/255512ssj_ops)
  • SPECpower_ssj allows default BIOS settings to be changed to make better results by turning off prefetch (also see SPECjbb & prefetch) - yeah, right, every customer does that :( get real.

So what serves the industry? Using the same configurations that you benchmark for performance and power-performance.

Sun does this and has been doing this for years:
2007: UltraSPARC T2
2005: UltraSPARC T1 & T2000 blogs with power-performance

The only thing I really can take away from SPECpower_ssj2008 is that running at low utilization is silly, read carefully, and avoid silly configurations.

Disclosure statement

SPECpower_ssj2008: HP Proliant DL160 G5 (2-chip QC Xeon E5450 3GHz), 698 overall ssj_ops/watt. SPECpower_ssj2008:HP Proliant DL580 G5 (4-chip QC Xeon L7345 1.86GHz), 546 overall ssj_ops/watt, 359523 ssj_ops and 387 watt at 100% target load, 255512 ssj_ops and 359 watt at 70% target load, and 71409 ssj_ops and 294 watt at 20% target load. SPECpower_ssj2008: Colfax CX2266-N2 (2-chip AMD Opteron DC 2216HE 2.4GHz), 203 overall ssj_ops/watt. SPEC, SPECpower reg tm of Standard Performance Evaluation Corporation. Results from www.spec.org as of 12/11/07. HP ProLiant DL580 G5 (16 cores, 4 chips) 30261 SPECweb2005. SPEC, SPECweb reg tm of Standard Performance Evaluation Corporation. Results from www.spec.org as of Oct 8, 2007. HP DL580 power consumption from HP Power Calculator system configured with 4 x2.93GHz processors, redundant PSU, 16 x 4GB DIMMs, 8 x 36GB SAS drives,1 x PCI card, 80% utilisation on 9/10/07: http://h30099.www3.hp.com/configurator/powercalcs.asp HP DL385G2 power consumption from HP Power Calculator for system configured with 2 x AMD 2220 2.8GHz processors, redundant PSU, 8 x 4GB DIMMs, 2 x HBAs and 2 x 146GB SAS drives, 80% utilisation on 6/4/07: http://h30099.www3.hp.com/configurator/powercalcs.asp

[1] Comments
Like this post? del.icio.us | furl | slashdot | technorati | digg
Posted at 02:16PM Dec 18, 2007 in Wattage-Power
Tags:
Comments:

I agree with you 100%. All SPEC benchmarks must be amended to include power draw by the SUT and should be reported as a graph for the duration of the test. I wonder why SPEC took the approach they took.

Posted by Anantha on December 19, 2007 at 09:02 AM PST #

Post a Comment:
Comments are closed for this entry.