Open letter to TPC & SPEC committees

Tuesday Jan 27, 2009

This blog approaches 250,000 visits, first of all thanks I hope you have all learnt a few things of value from this anonymous Sun employee. It is not about me. But the stats are quite an honour.

I decided to have an open letter to the benchmark committees, here are my blue sky suggestions:

• Can you please have the exact words you need for a disclosure statement clearly listed with any submission.
  ex: SPEC Disclosure Statement:
  Sun SPARC Enterprise X5220 (8 cores, 1 chip) 41847 SPECweb2005. SPEC, SPECweb reg tm of Standard Performance Evaluation Corporation. Results from www.spec.org as of Dec 4, 2008.
  ex: TPC Disclosure Statement:
  Sun SPARC Enterprise M9000 118,573.3 QphH@1000GB, $23.38/QphH@1000GB, avail 09/10/08. TPC-H, QphH, $/QphH tm of Transaction Processing Performance Council (TPC). More info www.tpc.org.
  That way one could just copy&paste from your form and not try to dig up your rules. When one needed several different results for for comparisons it would only take very simple editing.
• Make all vendors publish exact power as measured on the performance bragged about for all benchmarks. No to make new benchmarks to show watts on current benchmarks. We all must save energy now. Let the public see the data. This table should be filled in: "New Table of published power-performance data"
• Only allow power-performance to be reported on systems with default power-management software. We need power-management on by default for all results. Also only allow standard BIOS (of course tell Sun "no" as well, as I know Sun has published results with the same sorts of modifications). The reason why is we need to move the industry in a way that saves energy without having to turn "on" or "off" features on a per server basis. That is way to much work in a complex datacentre. Standard & default should be the direction.

Like this post? | | | |

Posted at 12:10PM Jan 27, 2009 in Benchmark Issues
Tags: benchmarks performance spec tpc