Open Letter to SPEC part Deux: The Cantrill Chronicles

Tuesday Feb 03, 2009

Just saw this this morning, a good writeup on a SPEC benchmark. Bryan Cantrill writes:

"The result is such a deformed monstrosity that -- like the index case of some horrific new pathogen -- its only remaining utility lies on the autopsy table: by dissecting SPEC SFS and understanding how it has failed, we can seek to understand deeper truths about benchmarks and their failure modes.

For more read: "Eulogy for a benchmark: I come to bury SPEC SFS, not to praise it."
http://blogs.sun.com/bmc/entry/eulogy_for_a_benchmark

Basically I think there are some very good SPEC & TPC benchmarks, but I think there is also some partisanship that needs to be removed to just do the right thing for the industry. See my previous posting for some ideas on good things to do.

I've covered the problems with ageing TPC-C, but maybe I should have used the same language that Bryan used.

Short-stroking 224 15K RPM drives is the equivalent of fueling a dragster with nitromethane -- it is top performance at a price so high as to be useless off the dragstrip.

Does this apply to the system configurations used in SPECpower. hmmmm, looks like the same colour.

Open letter to TPC & SPEC committees

Tuesday Jan 27, 2009

This blog approaches 250,000 visits, first of all thanks I hope you have all learnt a few things of value from this anonymous Sun employee. It is not about me. But the stats are quite an honour.

I decided to have an open letter to the benchmark committees, here are my blue sky suggestions:

• Can you please have the exact words you need for a disclosure statement clearly listed with any submission.
  ex: SPEC Disclosure Statement:
  Sun SPARC Enterprise X5220 (8 cores, 1 chip) 41847 SPECweb2005. SPEC, SPECweb reg tm of Standard Performance Evaluation Corporation. Results from www.spec.org as of Dec 4, 2008.
  ex: TPC Disclosure Statement:
  Sun SPARC Enterprise M9000 118,573.3 QphH@1000GB, $23.38/QphH@1000GB, avail 09/10/08. TPC-H, QphH, $/QphH tm of Transaction Processing Performance Council (TPC). More info www.tpc.org.
  That way one could just copy&paste from your form and not try to dig up your rules. When one needed several different results for for comparisons it would only take very simple editing.
• Make all vendors publish exact power as measured on the performance bragged about for all benchmarks. No to make new benchmarks to show watts on current benchmarks. We all must save energy now. Let the public see the data. This table should be filled in: "New Table of published power-performance data"
• Only allow power-performance to be reported on systems with default power-management software. We need power-management on by default for all results. Also only allow standard BIOS (of course tell Sun "no" as well, as I know Sun has published results with the same sorts of modifications). The reason why is we need to move the industry in a way that saves energy without having to turn "on" or "off" features on a per server basis. That is way to much work in a complex datacentre. Standard & default should be the direction.

Note: If you are a TPC or SPEC committee member I would appreciate any comments, these are blue sky ideas. But if you comment you owe it to all of the readers to state your affiliations and vendor name if you get are employed by a computer vendor.