InfoWorld weighs in on Energy Star for Servers

Wednesday May 27, 2009

New article on Energy Star for Servers http://www.infoworld.com/print/76258

Idle servers are the devil's tools So what's wrong with the new Energy Star criteria? Perhaps the most significant drawback is that a qualifying server need only demonstrate energy efficiency when it's in idle -- that is, powered on but doing no work.

To better illustrate the problem, imagine you're at a used car lot where a shifty salesperson is trying to push you to buy an SUV. His selling point: "This baby uses the same amount of fuel as a hybrid sedan -- when you're not moving."

But what about SPECpower, see these:
http://blogs.sun.com/bmseer/entry/specpower_ssj2008_sun_netra_x4250
http://blogs.sun.com/bmseer/entry/chopped_configs_and_specpowerssj2008

Like this post? | | | |

Posted at 10:44AM May 27, 2009 in Benchmark Issues
Tags: performance power-performance watts

Miles per gallon or GPM, same as perf/watt or watt/perf

Tuesday May 19, 2009

As a reminder: "Miles per gallon is misleading and can play tricks on our intuitions," Jack Soll said.

Take this little test, if you think he is wrong:
http://www.fuqua.duke.edu/news/mpg/mpg.html

In computing the same is true perf/watt is misleading and plays tricks on our intuitions. Sun tries to promote watt/performance. You'll even hear Sun's VPs talking about power-performance!

Today in email, I saw a pointer to this gem of a website: http://www.mpgillusion.com

Like this post? | | | |

Posted at 11:42AM May 19, 2009 in Wattage-Power
Tags: performance power-performance watts

Power-performance benchmarks show Linear estimates very accurate at different utilisation levels

Friday May 15, 2009

Servers have near-linear power response between Idle and 100% utilisation, that is just knowing idle and 100% utilisation you can come up with very good estimates of watts used at any utilisation level for a given workload.

I might even venture to guess that run to run variation may be greater than the error of this linear estimation?

For more details on Sun's three new SPECpower_ssj2008 results see yesterday's posting where I blogged about SPECpower and used Sun's new results to show directly some of the things vendors do to really boost scores on SPECpower_ssj2008. That blog entry can be found here:
http://blogs.sun.com/bmseer/entry/specpower_ssj2008_sun_netra_x4250

Let's look at the results on typical configuration Sun Netra X4250 (32GB default BIOS). Many companies use 2GB/core on Xeon processors and a good heuristic on memory sizings and that is why I call this a typical configuration for this blog posting.

Relative ops/sec (%util)	Actual watts	watts (linear estimate)	Error
100%	296w	296w est	0.0%
90%	291w	289w est	0.7%
80%	286w	282w est	1.5%
70%	280w	275w est	1.9%
60%	273w	268w est	2.0%
50%	267w	261w est	2.5%
40%	259w	253w est	2.2%
30%	251w	246w est	1.9%
20%	243w	239w est	1.6%
10%	235w	232w est	1.3%
0% (Idle)	225w	225w est	0.0%

OK so the error is less than 2.5%, that is very very close. No need to set up complicated tests if you want to measure you own watts and guess your utilisation-watts curve.

Benchmark Description

SPECpower_ssj2008 is the first SPEC benchmark intended to measure the power efficiency of a server. It is based on SPECjbb2005 but the workload has been modified so that the performance portion of the results are not comparable to SPECjbb2005 results. In addition, the workload is varied from unconstrained (ie. maximum) throughput performance to idle (but active) state in 10% decrements, during which the average power consumption is measured. The power and performance is measured, and the ratio of performance to power is computed, for each load point. The overall metric, denoted overall ssj_ops/watt, is the ratio of the sum of performance at each point to the sum of average power at each point, to include the idle point.

Some of the competitive results use non-redundant fans and non-redundant power supplies and minimize other aspects of the configuration.

The Sun Netra X4250 includes a realistic:

• 2x redundant power supplies
• redundant fan modules
• standard I/O expansion mezzanine
• standard Telco dry contact alarm
which are only present in the Netra X4250 among SPECpower_ssj2008 configurations.

To see the effect of changing configurations and using non-default {HACKED } BIOS see:
http://blogs.sun.com/bmseer/entry/specpower_ssj2008_sun_netra_x4250

Disclosure Statement:

Sun Netra X4250 server 600 overall ssj_ops/watt and (226 watts, 244832 ssj_ops) @ 100% target load, (210 watts, 121828 ssj_ops) @ 50% target load, (181 watts, 24150 ssj_ops) @ 10% target load, (174 watts) @ active idle target load. Sun Netra X4250 server 478 overall ssj_ops/watt and (294 watts, 251555 ssj_ops) @ 100% target load, (226 watts) @ active idle target load. Sun Netra X4250 server 437 overall ssj_ops/watt and (0 ssj_ops, 225 watts) @ 0% target active idle target load, (22866 ssj_ops, 235 watts) @ 10% target load, (45752 ssj_ops, 243 watts) @ 20% target load, (68959 ssj_ops, 251 watts) @ 30% target load, (92768 ssj_ops, 259 watts) @ 40% target load, (115284 ssj_ops, 267 watts) @ 50% target load, (138548 ssj_ops, 273 watts) @ 60% target load, (162384 ssj_ops, 280 watts) @ 70% target load, (184875 ssj_ops, 286 watts) @ 80% target load, (208601 ssj_ops, 291 watts) @ 90% target load, (229828 ssj_ops, 296 watts) @ 100% target load, SPEC and the benchmark names SPECpower_ssj, SPECpower are trademarks of the Standard Performance Evaluation Corporation. Benchmark results stated above reflect results published on http://www.spec.org as of March 30, 2009. For the latest SPECpower_ssj2008 benchmark results, visit http://www.spec.org/power_ssj2008. See Also: SPECpower_ssj2008 Benchmark Reports

System Configuration

Sun's three results all used the same software components and processors.

Processor:			2 x Intel L5408 QC 2.13 GHz
Operating System:			Windows Server 2003 R2 Enterprise x64 Edition SP2
JVM:			Java HotSpot(TM) 32-Bit Server VM on Windows, version 1.6.0_14

Reference Date:			May 6, 2009
Results			437 overall ssj_ops/watt
System:			Sun Netra X4250 (32GB, 8 x 4096MB as PC2-5300F)
BIOS:			default (normal prefetch)

• 4 x Sun 146GB 10K RPM SAS drive
• 1 x Sun x8 PCIe Quad Gigabit Ethernet option card (X4447A-Z)
• 2 x 658watt redundant AC power supplies
• redundant fans
• standard I/O expansion mezzanine
• standard Telco dry contact alarm

Like this post? | | | |

Posted at 08:16AM May 15, 2009 in Benchmark Issues
Tags: performance power specpower utilization

SPECpower_ssj2008 Sun Netra X4250 - SPECpower issues highlighted

Thursday May 14, 2009

Sun has finally published some results that show the configuration issues with SPECpower_ssj2008, these three results substantiate a just a few of the issues that I've been talking about with SPECpower_ssj. Clearly now is not the time for special power benchmarks but publication of watts on all benchmarks - which Sun has been doing for YEARS! Now, on to information that I saw in an email that went out yesterday...

The Sun Netra X4250 8GB server (two 2.13 GHz Intel L5408 QC) obtained a peak overall ssj_ops/watt metric of 600 (with special BIOS tuning). A more typical 32GB configuration of the same system achieved results of 478 (with special BIOS tuning) and a lower 437 (with standard BIOS settings). Note 8GB is only 0.5GB/core which is much smaller than 32GB (2GB/core) which is used for many 2-socket QC benchmarks that just measure performance).

These results were obtained on the Sun Netra X4250 server using Microsoft Windows Server 2003 R2 Enterprise x64 Edition SP2 and Sun Java HotSpot 32-Bit Server VM on Windows, version 1.6.0_14.

SPECpower_ssj results shows that servers (even those with the industry's best power-management) running at low-utilization levels use many times more watts per unit-of-work than systems running at higher utilization levels. Datacenters can realize the biggest energy savings by running fewer servers at higher utilization levels (50% utilization or above).

Sun's results on the 8GB (or 0.5GB/core) configuration show that running at 10% utilization requires 4.4 times more power per unit of work than running at 50% utilization.

4.4 times = (581 performance-to-power @ 50% utilization /133 performance-to-power @ 10% utilization)

Most SPECpower_ssj2008 are published on small-memory configurations that are much smaller than typical customer deployments. Sun is the only vendor to publish multiple results to clearly show effect of memory configuration.

A more normal-sized memory configuration of 32GB (or 2GB/core) uses 30% more watts than a tiny 8GB (or 0.5GB/core) configuration at 100% load. At active-idle the wattage difference is also 30%. Some competitors use additional configuration differences such as non-redundant fans, non-redundant power supplies, and single slow disk to further reduce the wattages and significantly improve SPECpower_ssj scores.

Most published SPECpower_ssj2008 results make low-level BIOS changes to turn off hardware prefetch. Sun shows that non-default BIOS changes improve Peak Performance ssj_ops by 9%. This non-default BIOS change hurts the performance of other workloads.

Dramatic minimization of memory configuration (0.5GB/core) and the use of a non-standard BIOS provided a 39% improvement to the peak Performance-to-Power Ratio (ssj_ops/watt) on small 8GB configuration Sun Netra X4250 compared to the same server with more-typically configured 32GB (2GB/core) of memory with the default BIOS.

Also when I look at the data, I find a _very_ linear relationship on wattage from active-idle to 100%. In other words if you measure 100% and Idle you can easily and very accurately estimate the watts at any utilization level. If anyone has data please to the contrary please post a comment with data & analysis - Thanks from your friendly BM Seer.

SPECpower_ssj2008 Low-Power Harpertown (QC L5400-series) Performance Chart (ordered by benchmark primary metric, overall ssj_ops/watt) Selected low-power Harpertown leading and major-manufacturer results. Metric: overall ssj_ops/watt (bigger is better)

Some of the competitive results use NON-redundant fans and NON-redundant power supplies and minimize other aspects of the configuration.

The Sun Netra X4250 includes:

• 2x redundant power supplies
• redundant fan modules
• standard I/O expansion mezzanine
• standard Telco dry contact alarm
which are only present in the Netra X4250 among SPECpower_ssj2008 configurations.

System	Processors		Performance
	Model	GHz	Metric overall ssj_ops/watt	Peak Performance ssj_ops	Peak Power watts	Idle Power watts
Powerleader PR2510D2 (8GB non-std BIOS)	L5430	2.66	1135	285970	161	84.6
NEC ECO CENTER (8GB non-std BIOS)	L5420	2.5	1010	288502	175	102
HP ProLiant DL180 G5 (8GB non-std BIOS)	L5430	2.66	930	282281	189	106
Fujitsu PRIMERGY TX300 S4 (8GB non-std BIOS)	L5430	2.66	917	326128	221	136
Dell PowerEdge R300 (8GB non-std BIOS)	L5410	2.33	800	155342	117	75.1
Sun Netra X4250 (8GB non-std BIOS)	L5408	2.13	600	244832	226	174
Sun Netra X4250 (32GB non-std BIOS)	L5408	2.13	478	251555	294	226
Sun Netra X4250 (32GB default BIOS)	L5408	2.13	437	229828	296	225

Complete benchmark results may be found at the SPEC benchmark website http://www.spec.org. Results as of May 8, 2009.

Benchmark Description

SPECpower_ssj2008 is the first SPEC benchmark intended to measure the power efficiency of a server. It is based on SPECjbb2005 but the workload has been modified so that the performance portion of the results are not comparable to SPECjbb2005 results. In addition, the workload is varied from unconstrained (ie. maximum) throughput performance to idle (but active) state in 10% decrements, during which the average power consumption is measured. The power and performance is measured, and the ratio of performance to power is computed, for each load point. The overall metric, denoted overall ssj_ops/watt, is the ratio of the sum of performance at each point to the sum of average power at each point, to include the idle point.

Disclosure Statement:

Sun Netra X4250 server 600 overall ssj_ops/watt and (226 watts, 244832 ssj_ops) @ 100% target load, (210 watts, 121828 ssj_ops) @ 50% target load, (181 watts, 24150 ssj_ops) @ 10% target load, (174 watts) @ active idle target load. Sun Netra X4250 server 478 overall ssj_ops/watt and (294 watts, 251555 ssj_ops) @ 100% target load, (226 watts) @ active idle target load. Sun Netra X4250 server 437 overall ssj_ops/watt and (296 watts, 244832 ssj_ops) @ 100% target load, (225 watts) @ active idle target load. SPEC and the benchmark names SPECpower_ssj, SPECweb, SPECjbb, SPECjAppServer, SPEComp are trademarks of the Standard Performance Evaluation Corporation. Benchmark results stated above reflect results published on http://www.spec.org as of March 30, 2009. For the latest SPECpower_ssj2008 benchmark results, visit http://www.spec.org/power_ssj2008. See Also: SPECpower_ssj2008 Benchmark Reports

System Configuration

Sun's three results all used the same software components and processors.

Processor:			2 x Intel L5408 QC 2.13 GHz
Operating System:			Windows Server 2003 R2 Enterprise x64 Edition SP2
JVM:			Java HotSpot(TM) 32-Bit Server VM on Windows, version 1.6.0_14

The following result was produced using a non-typical, small configuration and special BIOS tuning.

Reference Date:			March 30, 2009
Results			600 overall ssj_ops/watt
System:			Sun Netra X4250 (8GB, 4 x 2048MB as PC2-5300F 2Rx8)
BIOS:			non-standard (hardware prefetch off)

• 1 x Sun 146GB 10K RPM SAS drive
• 2 x 658watt redundant AC power supplies
• redundant fans
• standard I/O expansion mezzanine
• standard Telco dry contact alarm

The following result was produced using a more typical larger configuration including fully configured disk drives and an option NIC card. Special BIOS tuning was retained to boost performance and allow a direct comparison between BIOS tunings and standard BIOS.

Reference Date:			May 6, 2009
Results			478 overall ssj_ops/watt
System:			Sun Netra X4250 (32GB, 8 x 4096MB as PC2-5300F)
BIOS:			non-standard (hardware prefetch off)

• 4 x Sun 146GB 10K RPM SAS drive
• 1 x Sun x8 PCIe Quad Gigabit Ethernet option card (X4447A-Z)
• 2 x 658watt redundant AC power supplies
• redundant fans
• standard I/O expansion mezzanine
• standard Telco dry contact alarm

Reference Date:			May 6, 2009
Results			437 overall ssj_ops/watt
System:			Sun Netra X4250 (32GB, 8 x 4096MB as PC2-5300F)
BIOS:			default (normal prefetch)

• 4 x Sun 146GB 10K RPM SAS drive
• 1 x Sun x8 PCIe Quad Gigabit Ethernet option card (X4447A-Z)
• 2 x 658watt redundant AC power supplies
• redundant fans
• standard I/O expansion mezzanine
• standard Telco dry contact alarm

Like this post? | | | |

Posted at 08:06AM May 14, 2009 in Benchmarks
Tags: performance power specpower watts x4250

Sun Fire X4270 Server Two-tier SAP-SD ERP 6.0 Enhancement Pack 4 (Unicode) Standard

Wednesday Apr 15, 2009

World Record 2-processor performance result on Two-tier SAP ERP 6.0 Enhancement Pack 4 (Unicode) Standard Sales and Distribution (SD) Benchmark

World Record 2-processor performance result on the two-tier SAP ERP 6.0 enhancement pack 4 (unicode) standard sales and distribution (SD) benchmark as of March 30th, 2009.

The Sun Fire X4270 server with two Intel Xeon X5570 processors (8 cores, 16 threads) achieved 3,700 SAP SD Benchmark users running SAP ERP application release 6.0 enhancement pack 4 benchmark with unicode software, using Oracle 10g database and Solaris 10 OS.

This benchmark result highlights the optimal performance of SAP ERP on Sun Fire servers running the Solaris OS and the seamless multilingual support available for systems running SAP applications.

The Sun Fire X4270 server beats both 2-processor HP ProLiant DL380 G6 and HP ProLiant BL460c G6 by 12% even though all three systems use the same processors and memory.

In January 2009, a new version, the Two-tier SAP ERP 6.0 Enhancement Pack 4 (Unicode) Standard Sales and Distribution (SD) Benchmark, was released. This new release has higher cpu requirements and so yields from 25-50% fewer users compared to the previous Two-tier SAP ERP 6.0 (non-unicode) Standard Sales and Distribution (SD) Benchmark.

The IBM System x3650 result was done with the older SAP ERP 6.0 (non-unicode) version. (why didn't they move to the new version like Sun did?) IBM has not published a benchmark using the new SAP ERP 6.0 Enhancement Pack 4 (Unicode) Standard Sales and Distribution (SD) Benchmark. Trying comparing IBM's non-unicode result to HP ProLiant DL380 G6 non-unicode result on similar hardware to their new result...

SAP-SD 2-Tier Performance Table (in decreasing performance order).

SAP ERP 6.0 Enhancement Pack 4 (Unicode) Results
(New version of the benchmark as of January 2009)

System	OS Database	Users	SAP ERP/ECC Release	SAPS	SAPS/ Proc	Date
NEC Express 5800 8xIntel Xeon X7460 @2.66GHz 256 GB	Windows Server 2008 Datacenter Edition SQL Server 2008	4,485	2009 6.0 EP4 (Unicode)	25,280	12,640	09-Feb-09
Sun Fire X4270 2xIntel Xeon X5570 @2.93GHz 48 GB	Solaris 10 Oracle 10g	3,700	2009 6.0 EP4 (Unicode)	20,300	10,150	30-Mar-09
HP ProLiant BL380 G6 2xIntel Xeon X5570 @2.93GHz 48 GB	Windows Server 2008 Enterprise Edition SQL Server 2008	3,310	2009 6.0 EP4 (Unicode)	18,070	9,035	27-Mar-09
HP ProLiant DL380 G6 2xIntel Xeon X5570 @2.93GHz 48 GB	Windows Server 2008 Enterprise Edition SQL Server 2008	3,300	2009 6.0 EP4 (Unicode)	18,030	9,015	27-Mar-09

SAP ERP 6.0 (non-unicode) Results
(Old version of the benchmark retired at the end of 2008)

System	OS Database	Users	SAP ERP/ECC Release	SAPS	SAPS/ Proc	Date
IBM System x3650 2xIntel Xeon X5570 @2.93GHz 48 GB	Windows Server 2003 EE DB2 9.5	5,100	2005 6.0	25,530	12,765	19-Dec-08
HP ProLiant DL380 G6 2xIntel Xeon X5570 @2.93GHz 48 GB	Windows Server 2003 EE SQL Server 2005	4,995	2005 6.0	25,000	12,500	15-Dec-08

Complete benchmark results may be found at the SAP benchmark website http://www.sap.com/benchmark.

Benchmark Description

The SAP Standard Application SD (Sales and Distribution) Benchmark is a two-tier ERP business test that is indicative of full business workloads of complete order processing and invoice processing, and demonstrates the ability to run both the application and database software on a single system. The SAP Standard Application SD Benchmark represents the critical tasks performed in real-world ERP business environments.

SAP is one of the premier world-wide ERP application providers, and maintains a suite of benchmark tests to demonstrate the performance of competitive systems on the various SAP products.

Disclosure Statement:

Two-tier SAP Sales and Distribution (SD) standard SAP ERP 6.0 2005/EP4 (Unicode) application benchmark as of 03/30/09: Sun Fire X4270 (2 processors, 8 cores, 16 threads) 3,700 SAP SD Users, 2x 2.93 GHz Intel Xeon x5570, 48 GB memory, Oracle 10g, Solaris 10, Cert# 2009005. HP ProLiant BL380 G6 (2 processors, 8 cores, 16 threads) 3,310 SAP SD Users, 2x 2.93 GHz Intel Xeon x5570, 48 GB memory, SQL Server 2008, Windows Server 2008 Enterprise Edition, Cert# 2009003. HP ProLiant DL380 G6 (2 processors, 8 cores, 16 threads) 3,300 SAP SD Users, 2x 2.93 GHz Intel Xeon x5570, 48 GB memory, SQL Server 2008, Windows Server 2008 Enterprise Edition, Cert# 2009004. IBM System x3650 M2 (2 Processors, 8 Cores, 16 Threads) 5,100 SAP SD users,2x 2.93 Ghz Intel Xeon X5570, DB2 9.5, Windows Server 2003 Enterprise Edition, Cert# 2008079. HP ProLiant DL380 G6 (2 processors, 8 cores, 16 threads) 4,995 SAP SD Users, 2x 2.93 GHz Intel Xeon x5570, 48 GB memory, SQL Server 2005, Windows Server 2003 Enterprise Edition, Cert# 2008071.

SAP, R/3, reg TM of SAP AG in Germany and other countries. More info www.sap.com/benchmark

Results approved and published by SAP.

Certified Results
	Performance:		3700 benchmark users
	Server:		Sun Fire X4270
	Processors:		2.93 GHz Intel Xeon Processor X5570, 2 processors / 8 cores / 16 threads
	Memory:		48 GB
	Operating system:		Solaris 10
	Database S/W:		Oracle 10g
	SAP software:		SAP ERP 6.0 Enhancement Pack 4 (Unicode)
	SAP Certification:		2009005
	Storage:		3 x STK2540, 3 x STK2501 each with 12 x 146GB/15KRPM disks

Like this post? | | | |

Posted at 10:31AM Apr 15, 2009 in Benchmark Issues
Tags: performance sap-sd x4270 x5570

Sun's New World Record on SPECcpu

Tuesday Apr 14, 2009

Today Sun announced world records for SPECfp2006: 50.4 on a 2-chip Nehalem (Intel Xeon X5570) Sun Blade X6270 as well as SPECint2006: 36.9 on a 2-chip Nehalem (Intel Xeon X5570) Sun Blade X6270.

Read more at: http://blogs.sun.com/jhenning/entry/sun_studio_trounces_intel_compiler.

Yes, even on servers based on the same CPUs as others, Sun can make a difference. Congrats to those on the Sun Studio Compiler team. They beats Intel's own compiler on this Intel chip by 20%, due to the optimization technologies found in the Sun Studio 12 Update 1 compiler.

See John's posting above for more info. On a different note, notice how much information Sun puts out our benchmarks - lots! Fun to look at IBM bloggers, some of whom spend 90% of their blog on "cute" and only 10% talking about benchmark results. Information is not ones enemy.

Disclosure Statement:

SPEC, SPECint, SPECfp reg tm of Standard Performance Evaluation Corporation. Results from www.spec.org as of 4/14/2009. Sun Blade X6270 (Intel Xeon X5570 / 2 chips / 8 cores) 50.4 SPECfp2006, IBM System p 570 (POWER6, 1 chip / 1 core) 24.9 SPECfp2006.

Like this post? | | | |

Posted at 08:49AM Apr 14, 2009 in Benchmark Issues
Tags: blade intel nehalem opensolaris performance sun x6270 x64 xeon

Sun Fire X4275 Sybase IQ TPC-H 1000GB World Record Price Performance, Non-Clustered

Tuesday Apr 14, 2009

This TPC-H result demonstrates that the Sun Fire X4275 server, powered by 2 Quad-core 2.93 GHz Intel Nehalem X5570 processors, using only 12 internal disks (SAS 300GB 15K RPM), achieved a QphH@1000GB of 23,365 with a price performance of $2.41. This is the best price performance among all non-clustered server results at 1000GB.

It is the Best 2-chip or 2-socket server result, even better than many 4-sockets servers.

To put this result in perspective, the best non Sun single server submission at 1000GB was the HP Superdome. The Superdome achieved  a QphH of 69,999 (about 3 times the Sun Fire X4275 performance) BUT:  it required almost 100 times the number of disks, more than 35 times the price and 8 times the number of cores when compared  to the Sun Fire X4275 configuration!

Once again, the Sun/SybaseIQ combination has produced a self-contained (i.e. a server without any external storage or external processing engines) data warehousing solution. Only Sun has the hardware and expertise to produce such TPC-H results. To date, Sun has published self-contained TPC-H results for each of the 100GB, 300GB, 1000GB and 3000GB scale-factors.

This is a extremely compact solution requiring only 2 rack units in total. Again contrast the Sun result with the HP Superdome, using 97 storage arrays at 3 RU each plus a 48 inch cabinet for the server.

Extremely efficient power consumption; peak power consumption throughout the entire benchmark run was 825 Watts with an average consumption of 750 Watts.

{humor: Any comments from HP or Dell or IBM why they never publish watts on any standard benchmarks with real size memory (i.e. anything above 16GB) ? } I'll take comments from incognito HP, IBM, or Dell employees below, as always.

Performance Results

In order to put the Sun Fire X4275 TPC-H result in perspective, the table below shows the top non-clustered TPC-H@1000 results from Sun, Bull and HP in ascending order of  $/QphH as of April 14, 2009.

Legend:

so/co/th = sockets, cores, threads
QphH  = Overall TPC-H Composite Metric (bigger is better).
$/QphH  = TPC-H Price/Performance metric (smaller is better)
Data Ratio = Total disk to actual data ratio

Complete benchmark results may be found at http://www.tpc.org.

Benchmark Description

The results reported here were performed on a Sun Fire X4275 system and used Sybase IQ as the database manager. Sybase IQ is a special product designed specifically for data warehousing applications. Sybase IQ was developed as a totally separate product from the more widely known Sybase database management system (Sybase Adaptive Server).

The TPC-H benchmark is a performance benchmark established by the Transaction Processing Council (TPC) to demonstrate Data Warehousing/Decision Support Systems (DSS). TPC-H measurements are produced for customers to evaluate the performance of various DSS systems. These queries and updates are executed against a standard database under controlled conditions. Performance projections and comparisons between different TPC-H Database sizes (300GB, 300GB, 1000GB, 3000GB and 10000GB) are not allowed by the TPC.

TPC-H is a data warehousing-oriented, non-industry-specific benchmark that consists of a large number of complex queries typical of decision support applications. It also includes some insert and delete activity that is intended to simulate loading and purging data from a warehouse. TPC-H measures the combined performance of a particular database manager on a specific computer system.

The main performance metric reported by TPC-H is called the TPC-H Composite Query-per-Hour Performance Metric (QphH@SF, where SF is the number of GB of raw data, referred to as the scale factor). QphH@SF is intended to summarize the ability of the system to process queries in both single and multi user modes. The benchmark requires reporting of price/performance, which is the ratio of QphH to total HW/SW cost plus 3 years maintenance. A secondary metric is the storage efficiency, which is the ratio of total configured disk space in GB to the scale factor.

The QphH composite metric is the Geometric Mean of 2 components: (1) a single user component, called Power, and a (2) multi-user component, called Throughput.  Power is a performance measurement of a single user stream of 22 queries, one batch insert and one batch delete, all run serially. The Throughput metric, instead, consists of essentially N concurrent Power streams (or “users” submitting queries), where N is a minimum number of required streams dependent upon the database size. For example, at 300GB, N must be at least 5 and at 300GB N must be at least 6. Both Power and Throughput are calculated metrics and each is inversely proportional to the queries elapsed time: thus the faster the queries finish, the larger the metric becomes and the better the result.

Disclosure Statement:

TPC-H, QphH, $/QphH are registered trademarks of the Transaction Processing Performance Council (TPC). More info at http://www.tpc.org/. Sun Fire X4275 23,365@1000GB, $2.41/QphH@1000GB, available 4/14/09.

Results Summary

Audited Results
	Database Size:		1000 GB (Scale Factor 1000)
	TPC-H Composite:		23,365.3
	Price/performance:		$2.41
	Available		4/14/09
Number of Systems:			1
Total Number Processors:			2
Total Number of Cores			8
Total Number of Threads			16
Processor/MHz of Server:			Intel Nehalem 2.93 GHz X5570 Quad Core
Storage:			12 x 15K SAS drives (all internal)
Database:			Sybase IQ 15
Operating System:			Solaris 10
Total 3 year Cost:			$56,263.91
Other Performance Metrics
	TPC-H Power:		29,824.6
	TPC-H Throughput:		18,304.9
	Database Load Time:		5 Hr 39 Min
	Storage Ratio:		3.35

Like this post? | | | |

Posted at 08:03AM Apr 14, 2009 in Benchmarks
Tags: intel nehalem opensolaris performance sun x4275 x64 xeon

SPECjvm2008 on Sun Blade X6270 World record result

Tuesday Apr 14, 2009

The Sun Blade X6270 server demonstrates Sun's position of leadership in Java based computing by publishing world record results for the SPECjvm2008 benchmark. The Sun Blade X6270 server delivered a result of 317.13 SPECjvm2008 Base ops/m using the Sun Java JDK 1.6.0_14 Performance Release with the OpenSolaris 2008.11 Operating System.

SPECjvm2008 Performance Chart (ordered by performance)

base: SPECjvm2008 Base ops/m (bigger is better)
peak: SPECjvm2008 Peak ops/m (bigger is better)
Ch/Co/Lc: Chips, Cores, Logical CPUs

SPECjvm2008 (Java Virtual Machine Benchmark) is a benchmark suite for measuring the performance of a Java Runtime Environment (JRE), containing several real life applications and benchmarks focusing on core java functionality. The suite focuses on the performance of the JRE executing a single application; it reflects the performance of the hardware processor and memory subsystem, but has low dependence on file I/O and includes no network I/O across machines. The SPECjvm2008 workload mimics a variety of common general purpose application computations. These characteristics reflect the intent that this benchmark will be applicable to measuring basic Java performance on a wide variety of both client and server systems.

SPEC also finds user experience of Java important, and the suite therefore includes startup benchmarks and has a required run category called base, which must be run without any tuning of the JVM to improve the out of the box performance.

SPECjvm2008 benchmark highlights:

• Leverages real life applications (like derby, sunflow, and javac) and area-focused benchmarks (like xml, serialization, crypto, and scimark).
• Also measures the performance of the operating system and hardware in the context of executing the JRE.

Disclosure Statement:

SPEC, SPECjvm reg tm of Standard Performance Evaluation Corporation. Results as of 4/14/08 on http://www.spec.org. Sun Blade X6270(2 chips, 8 cores) 317.13 SPECjvm2008 Base ops/m submitted to SPEC for review. Sun Fire X4450(4 chips, 24 cores) 283.79 SPECjvm2008 Base ops/m Sun Fire X4450(4 chips, 16 cores) 260.08 SPECjvm2008 Base ops/m

System Configuration

Results
	Performance:		317.13 SPECjvm2008 Base ops/m
	Reference Date:		Apr 14, 2009
Systems:			Sun Blade X6270
Total Number Processors:			2
Processor/ GHz of Server:			Intel Xeon X5570 QC 2.93 GHz
Operating System:			OpenSolaris 2008.11
JVM:			Java HotSpot(TM) 64-Bit Server VM on Solaris, version 1.6.0_14 Performance Release

Like this post? | | | |

Posted at 06:41AM Apr 14, 2009 in Benchmarks
Tags: blade fire intel nehalem opensolaris performance sun x6270 x64 xeon

Green Grid tool assesses free cooling options

Friday Apr 10, 2009

effective ones is to simply run servers at high-utilization which can save energy both to power the servers (2x to 5x more work done per unit of work watt!) and therefore you need less energy to cool your datacenters.

Another way to save on the cooling costs side of the equation is use free cooling.

Free cooling comes in the form of air-side and water-side economizers. With air-side economizers, often called fresh-air cooling, outside air is taken into a data center, usually filtered, and then used to cool the IT equipment. With water-side economizers, the outside air cools water in the outside chiller or water tower, which in turn cools a data center.

The Green Grid's online calculator includes a series of inputs -- location, temperature and humidity thresholds, IT load, and the cost of electricity are a few -- and at the other end spits out how many hours that data center can use air-side and water-side economizers, and how much money it could save.

For more info also see: www.thegreengrid.org

Like this post? | | | |

Posted at 12:41PM Apr 10, 2009 in Benchmark Issues
Tags: utilization watts

Sun's Try-and-Buy Program keeps running and making people happy

Wednesday Apr 08, 2009

On this blog I spend my time talking about realistic comparisons. Sure I get digs by Sun's competitors who often try to change the subject and rarely answer my questions.

Test our products yourself at your location. Sun has a Try and Buy program that is available in 28 countries. Sun is getting a phenomenal return -- most importantly happy customers. People are often buying that hardware (servers/storage) and then buying even more. The program has been running 3 years and keeps getting stronger and bigger.

HP & IBM are NOT doing this. Too scared to have their potential customers see their product issues? Being this open scares many.

Need more information on Sun's very open program see: http://www.sun.com/tryandbuy/ ...on the page it starts out with...

How do you know how well a Sun product will work in your environment? You have to try it. Test it. Stress it. It's easier than ever to plug in Sun's newest products free for 60 days, with full technical support. You pay nothing—not even shipping. And if you decide to buy your trial system it is 20 - 40% off.

Apply by May 31, 2009 to be eligible for discount pricing when you purchase your trial system. Discount pricing rules.

There is a lot more information at the URL listed above.

Like this post? | | | |

Posted at 10:48AM Apr 08, 2009 in Benchmark Issues
Tags: performance

Benchmark Dreaming or simply wishing too hard?

Tuesday Mar 31, 2009

I had a dream two weeks ago...

• IBM, HP, Dell, etc. would publish measured watts on all current SPEC benchmarks.
• Oracle and IBM DB2 would publish results on TPC-E
• HP, IBM, Dell would publish SPECpower on the same configurations they use for other SPEC benchmarks.
• Vendors who use Intel-based processors, would only publish SPEC results with default BIOS, particularly on SPECjbb and SPECpower
• That all vendors would make power-management software on by default.
• Dell, IBM, HP, etc. would stop overhyping 15 year old benchmarks (like TPC-C)

disclosure statement:

SPEC, SPECjbb, SPECpower, reg tm of Standard Performance Evaluation Corporation. TPC-C ,TPC-H, and TPC-E are trademarks of the Transaction Performance Processing Council (TPPC).

I've been saying for years these things about the TPC-C benchmark!!! http://blogs.sun.com/bmseer/tags/tpc-c

We even know that IBM has tuned it useless: IBM's TPC-C "tuning"(?) that won't apply to anything in the real world

June 2005 Interview with Bruce Lindsay (IBM Fellow) at http://www.sigmod.org/sigmod/record/issues/0506/p71-column-winslet.pdf

"And the good news is that about 40-70% of the stuff we do in performance tuning actually ends up helping end users."

Who will have the maturity to kill TPC-C. Sun has quite publishing a while back, remember when Sun had the world record TPC-C and in that announcement Sun said: "It's well-understood in the technical communities that TPC-C no longer represents current customer workloads since the transaction load that its models are made of are small, primitive and disconnected transactions. While this model was acceptable for the workloads of the late 1980s, it misses the mark..." Sun's World Record TPC-C Press release, August2000

Disclosure Statement

TPC-C results referenced above was the fastest overall performance world record at August 31, 2000. Sun Enterprise 10000 server (Starfire) running Sybase Adaptive Server Enterprise (ASE), 156,873.03 transactions per minute (tpmC), $48.81 price/tpmC, available February 28, 2001. A full disclosure report and executive summary are available through the TPC Web site located at www.tpc.org.

Like this post? | | | |

Posted at 08:36AM Mar 31, 2009 in Benchmark Issues
Tags: performance

TPC Benchmarks Don't Matter Anymore (Forrester, March 6, 2009)

Thursday Mar 12, 2009

TPC Benchmarks Don't Matter Anymore by Noel Yuhanna with Mike Gilpin, David D'Silva (Forrester)

Forrester authors write:

Transaction Processing Performance Council (TPC) benchmarks, once widely accepted as the standard DBMS benchmark, are becoming obsolete. Why? First, all top-tier DBMS vendors such as IBM, Microsoft, Oracle, Sybase, and Teradata are delivering high performance and scalability to support most large workloads.

Second, TPC benchmarks no longer reflect the complex workloads of today's real-world deployments. Third, customers that need high-end performance often prefer internal benchmarks to TPC benchmarks.

Finally, virtualization, cloud computing, and database-as-a-service are changing the way customers deploy databases, and TPC does not address these architectures. Enterprise architects performing assessments should not waste their time on TPC benchmarks of top-tier DBMS products, and tech industry marketers and product managers should redirect the millions they spend on benchmark engineering toward automated tuning and performance optimization

I've been saying for years these things about the TPC-C benchmark!!!
http://blogs.sun.com/bmseer/tags/tpc-c

We even know that IBM has tuned it useless: IBM's TPC-C "tuning"(?) that won't apply to anything in the real world

June 2005 Interview with Bruce Lindsay (IBM Fellow) at http://www.sigmod.org/sigmod/record/issues/0506/p71-column-winslet.pdf

"And the good news is that about 40-70% of the stuff we do in performance tuning actually ends up helping end users."

Who will have the maturity to kill TPC-C. Sun has quite publishing a while back, remember when Sun had the world record TPC-C and in that announcement Sun said: "It's well-understood in the technical communities that TPC-C no longer represents current customer workloads since the transaction load that its models are made of are small, primitive and disconnected transactions. While this model was acceptable for the workloads of the late 1980s, it misses the mark..." Sun's World Record TPC-C Press release, August2000

Disclosure Statement

TPC-C results referenced above was the fastest overall performance world record at August 31, 2000. Sun Enterprise 10000 server (Starfire) running Sybase Adaptive Server Enterprise (ASE), 156,873.03 transactions per minute (tpmC), $48.81 price/tpmC, available February 28, 2001. A full disclosure report and executive summary are available through the TPC Web site located at www.tpc.org.

Like this post? | | | |

Posted at 10:20AM Mar 12, 2009 in Benchmark Issues
Tags: benchmarks tpc-c

SPECpower Issues Watts up? Two views

Wednesday Mar 04, 2009

A couple of postings ago in http://blogs.sun.com/bmseer/entry/specpower_ssj_obscuring_important_factors I mentioned how I thought they had too much detail in the wrong places. I do want critical things listed, but can we please focus on the important issues to power of REALISTIC CONFIGURATIONS. What does spec place in the summary table? Information on the JVM, GREAT, but what about size memory? Look at the results page and see that some vendors used 110+ characters to describe the JVM used in the main summary table. Misplaced details.

Standard bodies need to concentrate focus on important system issues FIRST have extraneous like testing location ruthlessly located in the back of the document.

To me it looks like the SPECpower was designed by people who focus on CPUs rather than the system that's likely to be deployed in a REAL customer environment. I also see this when looking at one of our more frequent commenters, rick jones who wrote:

Still my take on the power draws in a SPECpower_ssj2008 benchmark, at least as they have been published thusfar it would be processors, dimms, what I've seen called alternatively the CEC (Core Electronics Complex) or glue or perhaps "other motherboard components," and either the power supply or the boot disc.

This is a very CPU-centric view, in my opinion Rick's list boils down to:

• processors,
• dimms,
• CEC (Core Electronics Complex) or motherboard components,
• the power supply,
• boot disc.

Rather funny that he considers one of the things you can't configure in a system to be 3rd on the list.

I'd rather take a more system's view on a wide variety of tests! Shouldn't we just show power WHENEVER WE RUN ANY BENCHMARK! Sun does. Are others avoiding this because they can't beat Sun? Likely.

Having looked at REAL customer configurations and lots of SPECpower results, my view of the most important things to report on a system configuration when measuring power are:

• Memory Size (GBs), some use boutique LV-DIMMS
• Fans (need to mention when non-redundant) - most SPECpower results use NON-redundant ?!?
• Power supply (need to mention when non-redundant) - many SPECpower use NON-redundant
• Power management SW - Please tell me why is this NOT on by default for all servers?
• Processor type & GHz
• Disks - most deployed servers have standard RPM good size disk or disks! - most SPEC power results have a single 60GB-160GB?!? 7200RPM antique

SPEC and SPECpower benchmark name are registered trademarks of the Standard Performance Evaluation Corporation. www.spec.org for details.

Like this post? | | | |

Posted at 07:47AM Mar 04, 2009 in Wattage-Power
Tags: performance power specpower watts

IBM avoiding system comparison

Thursday Feb 26, 2009

I'm seeing more articles from IBM trying to get customers too look at wrong things! Shame, Shame.

What customers would like to know about CMT vs IBM POWER6:

• How does system performance compare?
• How much do IBM's 16-thread/core systems cost? (answer: $1,000,000)
• What is the system $/Perf?
• What is the measured watts on benchmarks?

• Comparing per-thread or per-core performance is more important than system performance.
• Hardware threads all cost the same (server price divided by number of threads).
• IBM power6 16-core system costs the same as T5440?!?
• Single-thread performance is the most important aspect of a medium or large system with 128threads (example: IBM p595 has 128 threads).

Like this post? | | | |

Posted at 07:00AM Feb 26, 2009 in IBM Follies
Tags: cmt ibm performance power6

SPECpower_ssj obscuring important factors?

Wednesday Feb 25, 2009

The disclosure reports to SPECpower_ssj need to be redesigned. It seems to violate many of Dr. Edward Tufte (author of "The Visual Display of Quantitative Information"). For review, they are basically:

• "consists of complex ideas communicated with clarity, precision, and efficiency.
• is that which gives to the viewer the greatest number of ideas in the shortest time with the least ink in the smallest space.
• requires telling the truth about the data."

  He continues:

• induce the viewer to think about the substance rather than about methodology,..., or something else
• avoid distorting what the data have to say
• present many numbers in a small space
• encourage the eye to compare different pieces of data
• reveal the data at several levels of detail, from a broad overview to the fine structure

In my opinion these are not even close to being followed...

What are some of the key factors in server power? Memory size, Redundant Fans, Redundant Power supplies, Reasonable disk/network configuration, LOW-LEVEL WORKLOAD SPECIFIC TUNINGS.

What do I mean by that? Memory size is critical to power consumption and for example is listed as "4" Not even 4GB. Nicely vague. It also is listed in the 3rd section in the 13th row of the table -- VERY IMPORTANT YET SO VERY BURIED. What is listed in the 1st section 2nd row, something of no value to customers. Do you have a guess? ... "Test location". Very silly.

Also, by the by, I see several server submission on SPECpower_ssj with ONLY 4GB of total server memory -- that is 1GB/core. TINY!!! I don't know any real customer that has anywhere near this ratio for most servers in their datacentre.

ok the form shows this as 1x or 2x, but all of them seem to be 1x, meaning non-redundant.

I do not see this specified, does anyone else?

This is specified, for example most results show, "BIOS Settings: Adjacent Sector Prefetch Disabled, and Hardware Prefetch Disabled." This seems very low-level tuning to get around the fact that the benchmark Java workload does lots of pointer chasing and overloads the prefetcher. OK how does a customer know when to do that. SPEC NEEDS TO ONLY ALLOW DEFAULT BIOS!

Disclosure Statement

SPEC and SPECpower benchmark name are registered trademarks of the Standard Performance Evaluation Corporation. www.spec.org for details.

Like this post? | | | |

Posted at 11:08AM Feb 25, 2009 in Benchmark Issues
Tags: performance specpower specpower_ssj