SPECpower_ssj obscuring important factors?

Wednesday Feb 25, 2009

The disclosure reports to SPECpower_ssj need to be redesigned. It seems to violate many of Dr. Edward Tufte (author of "The Visual Display of Quantitative Information"). For review, they are basically:

• "consists of complex ideas communicated with clarity, precision, and efficiency.
• is that which gives to the viewer the greatest number of ideas in the shortest time with the least ink in the smallest space.
• requires telling the truth about the data."

  He continues:

• induce the viewer to think about the substance rather than about methodology,..., or something else
• avoid distorting what the data have to say
• present many numbers in a small space
• encourage the eye to compare different pieces of data
• reveal the data at several levels of detail, from a broad overview to the fine structure

In my opinion these are not even close to being followed...

What are some of the key factors in server power? Memory size, Redundant Fans, Redundant Power supplies, Reasonable disk/network configuration, LOW-LEVEL WORKLOAD SPECIFIC TUNINGS.

What do I mean by that? Memory size is critical to power consumption and for example is listed as "4" Not even 4GB. Nicely vague. It also is listed in the 3rd section in the 13th row of the table -- VERY IMPORTANT YET SO VERY BURIED. What is listed in the 1st section 2nd row, something of no value to customers. Do you have a guess? ... "Test location". Very silly.

Also, by the by, I see several server submission on SPECpower_ssj with ONLY 4GB of total server memory -- that is 1GB/core. TINY!!! I don't know any real customer that has anywhere near this ratio for most servers in their datacentre.

ok the form shows this as 1x or 2x, but all of them seem to be 1x, meaning non-redundant.

I do not see this specified, does anyone else?

This is specified, for example most results show, "BIOS Settings: Adjacent Sector Prefetch Disabled, and Hardware Prefetch Disabled." This seems very low-level tuning to get around the fact that the benchmark Java workload does lots of pointer chasing and overloads the prefetcher. OK how does a customer know when to do that. SPEC NEEDS TO ONLY ALLOW DEFAULT BIOS!

Disclosure Statement

SPEC and SPECpower benchmark name are registered trademarks of the Standard Performance Evaluation Corporation. www.spec.org for details.

Like this post? | | | |

Posted at 11:08AM Feb 25, 2009 in Benchmark Issues
Tags: performance specpower specpower_ssj

Dont' use mpg & perf/watt, please use gpm & watt/perf!

Friday Jun 20, 2008

miles/gallon is as misleading to consumers! Remember when I said perf/watt is misleading. How do we all avoid these 'math illusions'? Duke University researchers tell us this is simple, just "flip 'em"

Posting a vehicle’s fuel efficiency in “gallons per mile” (GPM) rather than “miles per gallon” (MPG) would help consumers make better decisions about car purchases and environmental impact, researchers from Duke University’s Fuqua School of Business report in the June 20 issue of Science magazine.

Check out the above video, you can see that people try to judge by linear improvement in miles/gallon, but this is very misleading. The recommend that we switch to gallons/mile!

Remember back in March 2007, where I said the metric is watt/performance and not perf/watt. http://blogs.sun.com/bmseer/entry/power_efficiency_metrics_clearing_up. Time for SPEC to reconsider their metrics, and only allow default settings to be measured in benchmarks (if power-management is not on by factory default it should NOT be measured in a test - that way customers are best served.

Improving inefficient cars saves a lot of gas, the same valid reasoning shows improving %utilisation IS the big win especially when coupled with efficient servers.

Nothing like a little vindication to start the weekend, OK it's getting late, cya next week

for a table on savings at differnet miles/gallon see: http://www.fuqua.duke.edu/news/mpg/table.pdf

Like this post? | | | |

Posted at 03:02PM Jun 20, 2008 in Benchmark Issues
Tags: performance specpower specpower_ssj watt/performance

SPECpower_ssj too many measurements?

Monday Jun 09, 2008

A posting last week, clearly demonstrated that even small increases in utilisations provide HUGE savings".

Then I started looking at the data, it seems that one only really needs two points(!) {active-idle & 100%) to determine the watts used at any utilisation. Let's take a look at the HP DL580 SPECpower_ssj result.

%util	Measured Watts	Linear Predict	watts diff	%Diff
100%	387w	-	-	0%
90%	376w	375.4w	0.6w	0%
80%	368w	363.8w	4.2w	1%
70%	359w	352.2w	6.8w	2%
60%	347w	340.6w	6.4w	2%
50%	335w	329w	6w	2%
40%	322w	317.4w	4.6w	1%
30%	309w	305.8w	3.2w	1%
20%	294w	294.2w	-0.2w	0%
10%	280w	282.6w	-2.6w	-1%
idle	271w	-	-	0%

I'll look at more at at other SPECpower_ssj results. But it seems that SPEC should just simply add idle watts and wattage measurements at 100% utilisation to ALL SPEC benchmarks and not redesign benchmarks to measure watts at 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, and 90%. In the worst case, above the linear prediction was ONLY 2% different than actual watts!

I have long said SPEC should just at watt/perf to all of their benchmarks as currently designed.

Disclosure statement

SPECpower_ssj2008:HP Proliant DL580 G5 (4-chip QC Xeon L7345 1.86GHz), 546 overall ssj_ops/watt, 359,523 ssj_ops and 387 watt at 100% target load, 325,931 ssj_ops and 376 watt at 90% target load, 291,991 ssj_ops and 368 watt at 80% target load, 255,512 ssj_ops and 359 watt at 70% target load, 217,222 ssj_ops and 347 watt at 60% target load, 180,262 ssj_ops and 335 watt at 50% target load, 145,079 ssj_ops and 322 watt at 40% target load, 110,173 ssj_ops and 309 watt at 30% target load, 71,409 ssj_ops and 294 watt at 20% target load, 36,070 ssj_ops and 280 watt at 10% target load, and Active Idle 271 watts. SPEC, SPECpower reg tm of Standard Performance Evaluation Corporation. Results from www.spec.org as of 12/11/07.

In a more realistic configuration the HP DL580 G5, from HP's own power calculators, a HP DL580 G5 with four QC Xeon 2.93GHz Tigerton and 64 GB memory should draw 1,072watts. HP DL580 power consumption from HP Power Calculator system configured with 4 x2.93GHz processors, redundant PSU, 16 x 4GB DIMMs, 8 x 36GB SAS drives,1 x PCI card, 80% utilisation on 9/10/07: http://h30099.www3.hp.com/configurator/powercalcs.asp

Like this post? | | | |

Posted at 05:07AM Jun 09, 2008 in Benchmark Issues
Tags: performance specpower_ssj wattage watts