BM Seer Unofficial thoughts from an anonymous Sun employee

Power-performance benchmarks show Linear estimates very accurate at different utilisation levels

Friday May 15, 2009

Servers have near-linear power response between Idle and 100% utilisation, that is just knowing idle and 100% utilisation you can come up with very good estimates of watts used at any utilisation level for a given workload.

I might even venture to guess that run to run variation may be greater than the error of this linear estimation?

For more details on Sun's three new SPECpower_ssj2008 results see yesterday's posting where I blogged about SPECpower and used Sun's new results to show directly some of the things vendors do to really boost scores on SPECpower_ssj2008. That blog entry can be found here:
http://blogs.sun.com/bmseer/entry/specpower_ssj2008_sun_netra_x4250

Let's look at the results on typical configuration Sun Netra X4250 (32GB default BIOS). Many companies use 2GB/core on Xeon processors and a good heuristic on memory sizings and that is why I call this a typical configuration for this blog posting.
Relative ops/sec (%util) Actual watts watts (linear estimate) Error
100%296w296w est0.0%
90%291w289w est0.7%
80%286w282w est1.5%
70%280w275w est1.9%
60%273w268w est2.0%
50%267w261w est2.5%
40%259w253w est2.2%
30%251w246w est1.9%
20%243w239w est1.6%
10%235w232w est1.3%
0% (Idle)225w225w est0.0%

OK so the error is less than 2.5%, that is very very close. No need to set up complicated tests if you want to measure you own watts and guess your utilisation-watts curve.

Benchmark Description

SPECpower_ssj2008 is the first SPEC benchmark intended to measure the power efficiency of a server. It is based on SPECjbb2005 but the workload has been modified so that the performance portion of the results are not comparable to SPECjbb2005 results. In addition, the workload is varied from unconstrained (ie. maximum) throughput performance to idle (but active) state in 10% decrements, during which the average power consumption is measured. The power and performance is measured, and the ratio of performance to power is computed, for each load point. The overall metric, denoted overall ssj_ops/watt, is the ratio of the sum of performance at each point to the sum of average power at each point, to include the idle point.

Some of the competitive results use non-redundant fans and non-redundant power supplies and minimize other aspects of the configuration.

The Sun Netra X4250 includes a realistic:

  • 2x redundant power supplies
  • redundant fan modules
  • standard I/O expansion mezzanine
  • standard Telco dry contact alarm
    • which are only present in the Netra X4250 among SPECpower_ssj2008 configurations.

To see the effect of changing configurations and using non-default {HACKED :) } BIOS see:
http://blogs.sun.com/bmseer/entry/specpower_ssj2008_sun_netra_x4250

Disclosure Statement:

Sun Netra X4250 server 600 overall ssj_ops/watt and (226 watts, 244832 ssj_ops) @ 100% target load, (210 watts, 121828 ssj_ops) @ 50% target load, (181 watts, 24150 ssj_ops) @ 10% target load, (174 watts) @ active idle target load. Sun Netra X4250 server 478 overall ssj_ops/watt and (294 watts, 251555 ssj_ops) @ 100% target load, (226 watts) @ active idle target load. Sun Netra X4250 server 437 overall ssj_ops/watt and (0 ssj_ops, 225 watts) @ 0% target active idle target load, (22866 ssj_ops, 235 watts) @ 10% target load, (45752 ssj_ops, 243 watts) @ 20% target load, (68959 ssj_ops, 251 watts) @ 30% target load, (92768 ssj_ops, 259 watts) @ 40% target load, (115284 ssj_ops, 267 watts) @ 50% target load, (138548 ssj_ops, 273 watts) @ 60% target load, (162384 ssj_ops, 280 watts) @ 70% target load, (184875 ssj_ops, 286 watts) @ 80% target load, (208601 ssj_ops, 291 watts) @ 90% target load, (229828 ssj_ops, 296 watts) @ 100% target load, SPEC and the benchmark names SPECpower_ssj, SPECpower are trademarks of the Standard Performance Evaluation Corporation. Benchmark results stated above reflect results published on http://www.spec.org as of March 30, 2009. For the latest SPECpower_ssj2008 benchmark results, visit http://www.spec.org/power_ssj2008. See Also: SPECpower_ssj2008 Benchmark Reports

System Configuration

Sun's three results all used the same software components and processors.

    Processor: 2 x Intel L5408 QC 2.13 GHz
    Operating System: Windows Server 2003 R2 Enterprise x64 Edition SP2
    JVM: Java HotSpot(TM) 32-Bit Server VM on Windows, version 1.6.0_14
The following result was produced using a more typical larger configuration including fully configured disk drives and an option NIC card. Standard BIOS tuning was used to demonstrate the advantage obtained by special BIOS tuning which benefits this benchmark.
    Reference Date: May 6, 2009
    Results 437 overall ssj_ops/watt
    System: Sun Netra X4250 (32GB, 8 x 4096MB as PC2-5300F)
    BIOS: default (normal prefetch)
  • 4 x Sun 146GB 10K RPM SAS drive
  • 1 x Sun x8 PCIe Quad Gigabit Ethernet option card (X4447A-Z)
  • 2 x 658watt redundant AC power supplies
  • redundant fans
  • standard I/O expansion mezzanine
  • standard Telco dry contact alarm

[4] Comments
Like this post? del.icio.us | furl | slashdot | technorati | digg
Posted at 08:16AM May 15, 2009 in Benchmark Issues
Tags:
Comments:

You should change the "Relative ops/sec (%util)" heading to "Relative ops/sec (%load)" because it would be too easy to assume that "50% util" was a "50% CPU util" which may or may not be the case. The SPECpower_ssj2008 disclosures seem to go to some pain to use the term "load" rather than "utilization/util."

English is such a wonderful langage because it has so many subjective terms. While you may consider a 2.5% error to be "very accurate" and "very very close" others, such as myself, might only be willing to call that "close." If that.

You may think them overly picky, but the SPECpower_ssj2008 Run and Reporting rules seem to insist on an uncertainty of 1% or less from the power analyzer:

http://www.spec.org/power_ssj2008/docs/SPECpower_ssj2008-Run_Reporting_Rules.html#2.13.2

I think it is particularly interesting that the maximum error in your linear interpolation is at 50% load, which is the load level you used for the power per unit work calculation in your other blog entry referenced in this one.

Further, your linear estimate is *underestimating* the power consumption when if one is going to be estimating something like power consumption, it would be better, IMO to have an overestimate than an
underestimate.

If we apply the linear estimate to the 8GB, 1 disc, no PCIe card x4250 configuration peak error is still at 50% load, but is now ~4.75% which I don't think anyone would consider very very close. If we apply a
linear estimate to the Powerleader result you mention in the blog url quoted above, the error at 50% load is still 4% and instead of tapering-off to <1% at 10% and 90% load is above ~2.3% which suggests
that perhaps some of those results aren't as linear as your eyes were leading you to believe.

"I might even venture to guess that run to run variation may be greater than the error of this linear estimation?" Sounds like something you should back-up with some actual data. Being in for a penny by making the three runs you are using as the basis for your recent blog entries, perhaps you can convince Sun to be in for a pound and provide three more runs to show the run-to-run variation?

As for those "hacked" BIOS tunes, I suppose you consider them equally evil in SPECjbb2005 submittals for Sun systems which have used them:

http://www.spec.org/osg/jbb2005/results/res2008q1/jbb2005-20080208-00452.html
http://www.spec.org/osg/jbb2005/results/res2008q1/jbb2005-20080208-00451.html
http://www.spec.org/osg/jbb2005/results/res2009q2/jbb2005-20090330-00702.html
http://www.spec.org/osg/jbb2005/results/res2009q2/jbb2005-20090330-00701.html
http://www.spec.org/osg/jbb2005/results/res2009q2/jbb2005-20090330-00703.html

and even on some SPARC results:

http://www.spec.org/osg/jbb2005/results/res2008q4/jbb2005-20081027-00552.html
http://www.spec.org/osg/jbb2005/results/res2008q4/jbb2005-20081027-00553.html
http://www.spec.org/osg/jbb2005/results/res2009q1/jbb2005-20081202-00561.html

Posted by rick jones on May 16, 2009 at 10:05 AM PDT #

I can't defend Sun's use of HACKED BIOS, I imagine the powers that be say if HP, IBM, Dell, Fujitsu, and others are doing it on SPECjbb then Sun should as well show that Sun systems perform the same or better. Personally I don't think anyone should do low-level BIOS hacking. Remember Sun is the only vendor on SPECpower to show the effect directly.

These were all done with the same configuration, 2.5% is in the noise. Measure two points and be done. If HP or IBM lost in performance by 3% you'd claim results were identical.

Posted by BM Seer on May 18, 2009 at 09:53 AM PDT #

If your linear estimate of power consumption based on the 100% load and Active Idle power consumption is applied to a pair of recent HP and IBM results using Intel 55XX processors and DDR3 RAM rather than 54XX processors and FD-DIMMs:

http://www.spec.org/power_ssj2008/results/res2009q2/power_ssj2008-20090330-00142.html
http://www.spec.org/power_ssj2008/results/res2009q2/power_ssj2008-20090325-00136.html

The maximum error isn't the 2.5% you claim based on the 32GB +PCIe card and extra power supply x4250 config, nor even the ~4.75% I got with the data from what you call the "tiny" x4250 config, but between 11% and 12%.

Posted by rick jones on May 20, 2009 at 08:43 AM PDT #

Rick, Ok even if we look at the HP result with 11-12% error in the middle
of the graph I think that is a pretty good approximation.

Remember that only doing some of the configuration hacking Sun showed a 30% difference.

So server users are much better measuring idle and 100% on the configuration they actually buy than looking at SPECpower results on a system of the same name but drastically minimized config.

Posted by BM Seer on May 27, 2009 at 10:15 AM PDT #

Post a Comment:
Comments are closed for this entry.