Last Friday, Chris posted SPECpower results on our new group blog BestPerf called "Interpreting Sun's SPECpower_ssj2008 Publications", it is well worth a read.
In the coming days you will see a wide variety of other results posted, these may include: speccpu, specfp, specint, specjappserver, specweb, specjbb, specomp, specpower, specjvm, specmail, speccpu, igen, Ansys, Nastran, sap-sd, Siebel, peoplesoft, TPC-C, TPC-E, TPC-H, etc.
Sun recently entered the SPECpower fray with the publication of three results on the SPECpower_ssj2008 benchmark. Strangely, the three publications documented results on the same hardware platform (Sun Netra X4250) running identical software stacks, but the results were markedly different. What exactly were we trying to get at?
Sun produces robust industrial-grade servers with a range of redundancy features we believe benefit our customers. These features increase reliability, at the cost of additional power consumption. For example, redundant power supplies and redundant fans allow servers to tolerate faults, and hot-swap capabilities further minimize downtime.
The benchmark run and reporting rules require the incorporation within the tested configuration of all components implied by the model name. Within these limitations, the first publication was intended to be the best result (that is, the lowest power consumption per unit of performance) achievable on the Sun Netra X4250 platform, by minimizing the configured hardware to the greatest extent possible.
All tested configurations had the following components in common:
And the same software stack:
In addition to the common hardware components, the tiny configuration was limited to:
This is called the tiny configuration because it seems unlikely that most customers would configure an 8-core server with only one disk and only 1 GB available per core. Nevertheless, from a benchmark point of view, this configuration gave the best result.
The other two results were both produced on a configuration we considered much more typical of configurations that are actually ordered by customers. In addition to the common hardware, these typical configuration included:
Nothing special was done with the additional components. The added memory increased the performance component of the benchmark. The other components were installed and configured but allowed to sit idle, so consumed less power than they would have under load.
So one thing we're getting at is the difference in power consumption between a small configuration optimized for a power-performance benchmark and a typical configuration optimized for customer workloads. Hardware (power consumption) is only half of the benchmark--the other half being the performance achieved by the System Under Test (SUT).
In all three publications the identical tunings were applied at the software level: identical java command-line arguments and JVM-to-processor affinity. We also applied, in the case of the better results, the common (but usually non-default) BIOS-level optimization of disabling hardware prefetcher and adjacent cache line prefetch. These optimizations are commonly applied to produce optimized SPECpower_ssj2008 results but it is unlikely that many production applications would benefit from these settings. To demonstrate the effect of this tuning, the final result was generated with standard BIOS settings.
And just so we couldn't be accused of sand-bagging the results, the number of JVMs was increased in the typical configurations to take advantage of the additional memory populated over and above the tiny configuration. Additional performance was achieved but sadly it doesn't compensate for the higher power consumption of all that memory.
So in summary we tuned:
At the OS level, all tunings were identical.
The results are summarized in this table:
|
System |
Processors |
Performance |
||||
|---|---|---|---|---|---|---|
|
Model |
GHz |
Metric |
Peak |
Peak |
Idle |
|
|
L5408 |
2.13 |
600 |
244832 |
226 |
174 |
|
|
L5408 |
2.13 |
478 |
251555 |
294 |
226 |
|
|
L5408 |
2.13 |
437 |
229828 |
296 |
225 |
|
The measurement and reporting methods of the benchmark encourage small memory configurations. Comparing the first and second result, adding additional memory yielded minimal performance improvement (from 244832 to 251555) but a large increase in power consumption, 68 watts at peak.
In our opinion, unrealistically small configurations yield the best results on this benchmark. On the more typical system, the benchmark overall metric decreased from 600 overall ssj_ops per watt to 478 overall ssj_ops per watt, despite our best effort to utilize the additional configured memory.
On typical configurations, reverting to default BIOS settings resulted in a significant decrease in performance (from 25155 to 229828) with no corresponding decrease in power consumption (essentially identical for both results).
Configurations typical of customer systems (with adequate memory, internal disks, and option cards) consume more power than configurations which are commonly benchmarked, while providing no corresponding improvement in SPECpower_ssj2008 benchmark performance. The result is a lower overall power-performance metric on typical configurations and a lack of published benchmark results on robust systems with the capacities and redundancies that enterprise customers desire.
SPEC, SPECpower, and SPECpower_ssj are trademarks of the Standard Performance Evaluation Corporation. All results from the SPEC website (www.spec.com) as of June 5, 2009. For a complete set of accepted results refer to that site.
This blog copyright 2009 by John Henning
