BM Seer Unofficial thoughts from an anonymous Sun employee

SPECpower_ssj2008 Sun Netra X4250 - SPECpower issues highlighted

Thursday May 14, 2009

Sun has finally published some results that show the configuration issues with SPECpower_ssj2008, these three results substantiate a just a few of the issues that I've been talking about with SPECpower_ssj. Clearly now is not the time for special power benchmarks but publication of watts on all benchmarks - which Sun has been doing for YEARS! Now, on to information that I saw in an email that went out yesterday...

The Sun Netra X4250 8GB server (two 2.13 GHz Intel L5408 QC) obtained a peak overall ssj_ops/watt metric of 600 (with special BIOS tuning). A more typical 32GB configuration of the same system achieved results of 478 (with special BIOS tuning) and a lower 437 (with standard BIOS settings). Note 8GB is only 0.5GB/core which is much smaller than 32GB (2GB/core) which is used for many 2-socket QC benchmarks that just measure performance).

These results were obtained on the Sun Netra X4250 server using Microsoft Windows Server 2003 R2 Enterprise x64 Edition SP2 and Sun Java HotSpot 32-Bit Server VM on Windows, version 1.6.0_14.

SPECpower_ssj results shows that servers (even those with the industry's best power-management) running at low-utilization levels use many times more watts per unit-of-work than systems running at higher utilization levels. Datacenters can realize the biggest energy savings by running fewer servers at higher utilization levels (50% utilization or above).

Sun's results on the 8GB (or 0.5GB/core) configuration show that running at 10% utilization requires 4.4 times more power per unit of work than running at 50% utilization.

    4.4 times = (581 performance-to-power @ 50% utilization /133 performance-to-power @ 10% utilization)

Most SPECpower_ssj2008 are published on small-memory configurations that are much smaller than typical customer deployments. Sun is the only vendor to publish multiple results to clearly show effect of memory configuration.

A more normal-sized memory configuration of 32GB (or 2GB/core) uses 30% more watts than a tiny 8GB (or 0.5GB/core) configuration at 100% load. At active-idle the wattage difference is also 30%. Some competitors use additional configuration differences such as non-redundant fans, non-redundant power supplies, and single slow disk to further reduce the wattages and significantly improve SPECpower_ssj scores.

Most published SPECpower_ssj2008 results make low-level BIOS changes to turn off hardware prefetch. Sun shows that non-default BIOS changes improve Peak Performance ssj_ops by 9%. This non-default BIOS change hurts the performance of other workloads.

Dramatic minimization of memory configuration (0.5GB/core) and the use of a non-standard BIOS provided a 39% improvement to the peak Performance-to-Power Ratio (ssj_ops/watt) on small 8GB configuration Sun Netra X4250 compared to the same server with more-typically configured 32GB (2GB/core) of memory with the default BIOS.

    Also when I look at the data, I find a _very_ linear relationship on wattage from active-idle to 100%. In other words if you measure 100% and Idle you can easily and very accurately estimate the watts at any utilization level. If anyone has data please to the contrary please post a comment with data & analysis - Thanks from your friendly BM Seer.

SPECpower_ssj2008 Low-Power Harpertown (QC L5400-series) Performance Chart (ordered by benchmark primary metric, overall ssj_ops/watt) Selected low-power Harpertown leading and major-manufacturer results. Metric: overall ssj_ops/watt (bigger is better)

Some of the competitive results use NON-redundant fans and NON-redundant power supplies and minimize other aspects of the configuration.

The Sun Netra X4250 includes:

  • 2x redundant power supplies
  • redundant fan modules
  • standard I/O expansion mezzanine
  • standard Telco dry contact alarm
    • which are only present in the Netra X4250 among SPECpower_ssj2008 configurations.

System Processors Performance
Model GHz Metric
overall ssj_ops/watt 		Peak Performance
ssj_ops 		Peak Power
watts 		Idle Power
watts
Powerleader PR2510D2 (8GB non-std BIOS) L5430 2.66 1135 285970 161 84.6
NEC ECO CENTER (8GB non-std BIOS) L5420 2.5 1010 288502 175 102
HP ProLiant DL180 G5 (8GB non-std BIOS) L5430 2.66 930 282281 189 106
Fujitsu PRIMERGY TX300 S4 (8GB non-std BIOS) L5430 2.66 917 326128 221 136
Dell PowerEdge R300 (8GB non-std BIOS) L5410 2.33 800 155342 117 75.1
Sun Netra X4250 (8GB non-std BIOS) L5408 2.13 600 244832 226 174
Sun Netra X4250 (32GB non-std BIOS) L5408 2.13 478 251555 294 226
Sun Netra X4250 (32GB default BIOS) L5408 2.13 437 229828 296 225

Complete benchmark results may be found at the SPEC benchmark website http://www.spec.org. Results as of May 8, 2009.

Benchmark Description

SPECpower_ssj2008 is the first SPEC benchmark intended to measure the power efficiency of a server. It is based on SPECjbb2005 but the workload has been modified so that the performance portion of the results are not comparable to SPECjbb2005 results. In addition, the workload is varied from unconstrained (ie. maximum) throughput performance to idle (but active) state in 10% decrements, during which the average power consumption is measured. The power and performance is measured, and the ratio of performance to power is computed, for each load point. The overall metric, denoted overall ssj_ops/watt, is the ratio of the sum of performance at each point to the sum of average power at each point, to include the idle point.

Disclosure Statement:

Sun Netra X4250 server 600 overall ssj_ops/watt and (226 watts, 244832 ssj_ops) @ 100% target load, (210 watts, 121828 ssj_ops) @ 50% target load, (181 watts, 24150 ssj_ops) @ 10% target load, (174 watts) @ active idle target load. Sun Netra X4250 server 478 overall ssj_ops/watt and (294 watts, 251555 ssj_ops) @ 100% target load, (226 watts) @ active idle target load. Sun Netra X4250 server 437 overall ssj_ops/watt and (296 watts, 244832 ssj_ops) @ 100% target load, (225 watts) @ active idle target load. SPEC and the benchmark names SPECpower_ssj, SPECweb, SPECjbb, SPECjAppServer, SPEComp are trademarks of the Standard Performance Evaluation Corporation. Benchmark results stated above reflect results published on http://www.spec.org as of March 30, 2009. For the latest SPECpower_ssj2008 benchmark results, visit http://www.spec.org/power_ssj2008. See Also: SPECpower_ssj2008 Benchmark Reports

System Configuration

Sun's three results all used the same software components and processors.

    Processor: 2 x Intel L5408 QC 2.13 GHz
    Operating System: Windows Server 2003 R2 Enterprise x64 Edition SP2
    JVM: Java HotSpot(TM) 32-Bit Server VM on Windows, version 1.6.0_14

The following result was produced using a non-typical, small configuration and special BIOS tuning.

    Reference Date: March 30, 2009
    Results 600 overall ssj_ops/watt
    System: Sun Netra X4250 (8GB, 4 x 2048MB as PC2-5300F 2Rx8)
    BIOS: non-standard (hardware prefetch off)
  • 1 x Sun 146GB 10K RPM SAS drive
  • 2 x 658watt redundant AC power supplies
  • redundant fans
  • standard I/O expansion mezzanine
  • standard Telco dry contact alarm

The following result was produced using a more typical larger configuration including fully configured disk drives and an option NIC card. Special BIOS tuning was retained to boost performance and allow a direct comparison between BIOS tunings and standard BIOS.

    Reference Date: May 6, 2009
    Results 478 overall ssj_ops/watt
    System: Sun Netra X4250 (32GB, 8 x 4096MB as PC2-5300F)
    BIOS: non-standard (hardware prefetch off)
  • 4 x Sun 146GB 10K RPM SAS drive
  • 1 x Sun x8 PCIe Quad Gigabit Ethernet option card (X4447A-Z)
  • 2 x 658watt redundant AC power supplies
  • redundant fans
  • standard I/O expansion mezzanine
  • standard Telco dry contact alarm
The following result was produced using a more typical larger configuration including fully configured disk drives and an option NIC card. Standard BIOS tuning was used to demonstrate the advantage obtained by special BIOS tuning which benefits this benchmark.
    Reference Date: May 6, 2009
    Results 437 overall ssj_ops/watt
    System: Sun Netra X4250 (32GB, 8 x 4096MB as PC2-5300F)
    BIOS: default (normal prefetch)
  • 4 x Sun 146GB 10K RPM SAS drive
  • 1 x Sun x8 PCIe Quad Gigabit Ethernet option card (X4447A-Z)
  • 2 x 658watt redundant AC power supplies
  • redundant fans
  • standard I/O expansion mezzanine
  • standard Telco dry contact alarm

[4] Comments
Like this post? del.icio.us | furl | slashdot | technorati | digg
Posted at 08:06AM May 14, 2009 in Benchmarks
Tags:
Comments:

I think this may be a trifle overstated - "but publication of watts on all benchmarks - which Sun has been doing for YEARS!" Sun has been publishing Watts on some perhaps even many benchmarks but certainly not all. Some of the most recent examples to back that up:

http://blogs.sun.com/bmseer/entry/sun_new_world_record_on which seems to flow back to http://www.sun.com/servers/blades/x6270/benchmarks.jsp which has several "watts-free" benchmark results.
http://blogs.sun.com/bmseer/entry/sun_fire_x4270_server_two which seems to reference http://www.sun.com/servers/x64/x4270/benchmarks.jsp#1

"A more normal-sized memory configuration of 32GB (or 2GB/core) uses 30% more watts than a tiny 8GB (or 0.5GB/core) configuration at 100% load." At active-idle the wattage difference is also 30%." If we go back to the (in)famous power calculators the difference between the 4x2048 MB DIMMs and 8x4094 MB DIMMs is 40-44 Watts (Idle vs 100%). However, it wasn't simply the memory that was changed - there were also 4X the Discs which the power calculator says is a delta of 24 Watts, and the PCIe card which the power calculator says is 15 Watts. That wattage difference for the memory is only ~15% at active idle, not 30%. If you want to make the 30% claim your sentence needs to include that discs and a PCIe card were added instead of leaving it to someone to parse from the configuration data at the end of the post.

"running at 10% utilization" - IIRC SPECpower_ssj2008 goes to some pains to use the term "load" rather than "utilization" because it is the ops load level that is being altered and that may or may not correlate with a CPU utilization.

Why is the power consumption so high even for the as you put it "tiny" 8GB Netra x4250 configuration compared to the other results in your table save perhaps the Fujitsu result? It is using the same quantity of RAM, but rather lower wattage CPUs and one would expect it to consume less Watts not more. Is that the "price" of the redundant power supply and fan modules? If so, the data in your table seems to suggest that such redundancy in small form-factor systems isn't terribly "green" because it seems the redundant configuration uses so much more power without increasing performance. Is that redundant power supply really wasting that many Watts?!?

Posted by rick jones on May 16, 2009 at 09:04 AM PDT #

IIRC SPECpower_ssj2008? Do you work on this benchmark? What is an IIRC? I am not familiar with that term.

Clearly there are even more "slash and burn" configuration changes to make even higher SPECpower results. I don't run SPECpower nor do any of the submissions. I would have submitted a bunch more to show the effect of everything. But that is still why we need to have every vendor show watts on normal benchmarks. Sun has run lots of benchmarks with power especially on CMT and more and more on X64. Sure not perfect but no other vendor HAS SHOWN MEASURED WATTS ON ANY OTHER SPEC, ISV, VMware, or other publicly scrutinized benchmark.

Your power calculator argument is weak, here is real data.

Hmmm.... load verus utilisation, what that is not much of a point, who's parsing words now?

Yes Sun's power consumption is higher than competitive systems simply because it's a realistic configuration that provides features (redundant power supplies, IO mezzanine, telco dry contact alarm and... ) that real customers want and actually purchase.

Rick if you remember my paper analysis in this blog years ago talked about all of these problems and yes the data bears me out.

As my neighbor says, ..."when you don't have much to add ...yer gonna nit pick."

Posted by BM Seer on May 18, 2009 at 09:42 AM PDT #

IIRC -Some may claim it means Irascible Internet Reactionary Curmudgeon :), but the actual translation is If I Recall Correctly.

You *really* do mean to say that simply one extra power supply, an I/O mezzanine and a telco dry contact alarm together consume between 40 and 100W more than those other systems??? (Taking the delta of the active idle's of the 8GB x4250 config and the other non-Sun systems in your table). http://www.sun.com/servers/netra/docs/netra_x4250_x4450_server_architecture.pdf makes the I/O mezzanine sound simply as the means by which I/O slots are expressed from the motherboard.

"redundant fan modules ... which are only present in the Netra X4250 among SPECpower_ssj2008 configurations." -While I would be willing to agree that the SPECpower_ssj2008 disclosures could make it easier by calling-out levels of redundancy for power and cooling, if you meant to imply that only the Netra X4250 results had redundant fans among all SPECpower_ssj2008 results you or your data source need to do a bit more homework. I spent about 20 minutes poking around on websites and I think if you look-up the specs for the IBM x3650 M2, the Dell PowerEdge R710 and HP ProLiant DL785 (none included in your table) you will find that they have redundant fans and it appears they are standard. So you may need to caveat the claim a bit more.

Posted by rick jones on May 19, 2009 at 09:39 AM PDT #

Wow...look at all this BIOS changes for the IBM x3650 M2!!!!!!
# Turbo Mode disabled in BIOS.
# Processor Data Prefetch disabled in bios.
# Processor C-States enabled in bios.
# QPI Frequency set to 4.8GT/s in bios.
# Memory frequency downrated from 1333MHz to 1066MHz in bios.
# Commands on USB Interface disabled in bios.

HP ProLiant DL785 G5 (2.30GHz, AMD Opteron 8376 HE) is 4-socket which is not the 2-sockets shown above!

FAR TOO MANY CONFIGURATION ISSUES IN SPECpower! All vendors should show power on other SPEC benchmarks.

Posted by BM Seer on May 27, 2009 at 10:32 AM PDT #

Post a Comment:
Comments are closed for this entry.