Monday Apr 30, 2007
When Sun was had the world record we said it was too simplistic and old, and
that was yeast ago. TPC-C has problems, IBM has heavily tuned it like this.
Why does IBM still point to this 14+ year old benchmark? Why do they
avoid new benchmarks with the lastest GHz full-system IBM p595 on:
- SPECjbb2005?
- SPECint_rate2006?
- SPECfp_rate2006?
- Linpack?
- SPECint_2006?
- SPECfp_2006?
- ....the list goes on...
Doesn't IBM want fair comparisons? I guess IBM would just be beaten by Sun
in performance and $/perf so they want to avoid comparisons.
It is funny that last year I egged HP on about SPECjbb2005, "why no results?"
Someone commented that HP thinks it is a bad benchmark, so they won't publish
on it. Now HP has the top result. Changed their tune?
Notice how this is different than when established a World Record TPC-C, Sun told the world the benchmark was too simplistic back then
and is sticking to it? The world became a lot more complicated in the past 7 years
and computing has evolved a lot so we won't go back to something that was created
13 years ago. Sun never quotes 23-year old Dhrystones benchmark anymore either. 
The press and analysts are overwhelmingly
seeing TPC-E the successor to the simplistic 14 year-old TPC-C.
IBM's TPC-C "tuning"(?) that won't apply to anything in the real
world
June 2005 Interview with Bruce Lindsay (IBM Fellow) at
http://www.sigmod.org/sigmod/record/issues/0506/p71-column-winslet.pdf
"And the good news is that about 40-70% of the stuff we do in
performance tuning actually ends up helping end users."
This means that 30% to 60% of IBM's TPC-C tunings don't help users.
Really beyond the huge disk size of the large TPC-C results (which has a lot to do with the TPC-C being 14 years old),
the quote below points to tuning that is legal but seems a bit too "tricky" for my taste...
"We get down to the level of worrying about the physical column order in the table so the reference columns are near each other, minimizing cache misses during fetching. This is feasible in the TPC-C benchmark because there are only five tables and only ten to fifteen columns in each table. In a more realistic application, where there are many more queries to be considered, the tables are typically much, much wider, in the 80 to 100 column range; and there are dozens if not thousands of tables. Then this kind of analysis is no longer practical." Bruce Linsay, IBM fellow"
For those who may not remember, IBM didn't even end the EOL'ed SPECint_rate2000 on a high note. See:
http://www.spec.org/cpu2000/results/rint2000.html and search for "1644" and "1513"
various footnotes:
"It's well-understood in the technical communities that TPC-C no longer represents current customer workloads since the transaction load that its models are made of are small, primitive and disconnected transactions. While this model was acceptable for the workloads of the late 1980s, it misses the mark..." Sun's World Record TPC-C Press release, August2000
Disclosure Statement
TPC-C results referenced above was the fastest overall performance world record at August 31, 2000. Sun Enterprise 10000 server (Starfire) running Sybase Adaptive Server Enterprise (ASE), 156,873.03 transactions per minute (tpmC), $48.81 price/tpmC, available February 28, 2001. A full disclosure report and executive summary are available through the TPC Web site located at
www.tpc.org.
Monday Mar 19, 2007
The new OLTP database benchmark TPC-E is finally getting more press, it is designed to be much
better than the overly simplistic TPC-C. Here is a sample of the stories today:
http://www.enterprisenetworksandservers.com/newsflash/art.php?707 writes:
"The TPC-E benchmark is the much-anticipated successor to TPC-C, the popular yardstick for comparing On-Line Transaction Processing (OLTP) performance on various hardware and software configurations. TPC-E defines a sophisticated workload representative of real-world OLTP applications."
http://www.itjungle.com/tfh/tfh031907-story07.html writes:
"TPC-C test, which is a lot more transaction than such a box in the field can usually do. This just goes to show how relatively skinny and simple TPC-C code is."
http://www.computerweekly.com/Articles/2007/03/19/222516/brokers-offered-new-transactions-testing-standard.htm writes:
"The new standard replaces the outdated TPC-C standard, which does not fully account for the way servers and databases now interact with each other, since processors have become ever more powerful and storage technology has changed."
http://news.zdnet.co.uk/software/0,1000000121,39286370,00.htm writes:
"...TPC-C dealt with simpler data and operations stored in nine tables that simulated inventory management at a warehouse."
"...TPC-C was widely used for this task — but servers and databases have changed a lot since its debut in July 1992."
"'TPC-C is at the end of its life. It's time for a new benchmark,' said Andreas Hotea, who leads the effort to publicise TPC-E. "
http://www.channelregister.co.uk/2007/03/19/tcp_new_tests/ writes:
"The introduction of a new benchmark is bound to reinvigorate enterprise software and server vendors. For years the innovation reflected in their product strategies has been matched only by the ingenuity they have shown in gaming the TPC's benchmarks, and selectively quoting results in their marketing materials."
Oh, that last one really hit it, we've talked about that a lot in this blog, and show specific
"""ingenuity""" ...what a word.
http://blogs.sun.com/bmseer/entry/ibm_too_tricky_for_good
Monday Mar 12, 2007
Currently TPC-C is not to be replaced by TPC-E, but it should.
TPC-C is really a simplistic database benchmark that is far too old
and simple.
It was reported by Timothy Prickett Morgan last week in
http://www.itjungle.com/breaking/bn030507-story02.html
"While the TPC-C test will soon be replaced by the TPC-E OLTP test, which is designed to address some of the shortcomings of the TPC-C workload and the benchmark methodology"
In my view, TPC-C should be shunned by the industry, joining
Sun's well thought out avoidance of a simplistic benchmark years ago.
We already uncovered IBM's numerous benchmark techniques that took
advantages of the TPC-C benchmarks simplicity.
http://blogs.sun.com/bmseer/entry/ibm_too_tricky_for_good
I'm kinda surprised that people would still cover TPC-C, long ago we gave
up on Dhrystones because of people following the benchmark rules but taking advantages of shortcomings to a high degree...
The article goes on to say:
"nonetheless one of the few independent metrics for assessing the performance of a server"
Actually there are actually a number of
independent workloads to assess server performance by both
TPC and
SPEC committees as well as several other ISV benchmarks such as
SAP.
Monday Feb 26, 2007
The new TPC-E benchmark simulates the OLTP workload of a brokerage firm,
will this work for you? Here is what TPC says...
Although the underlying business model of TPC-E is a brokerage firm, the database schema, data population, transactions, and implementation rules have been designed to be broadly representative of modern OLTP systems.
From looking at the information available so far, seems it will be reasonable.
See a simple comparison to TPC-C at:
http://blogs.sun.com/bmseer/entry/tpc_e_new_oltp_benchmark
Monday Feb 26, 2007
Now that TPC-E is an official benchmark we can look at the public documentation
at http://www.tpc.org/tpce/default.asp to start to compare it with the overly-simplistic TPC-C benchmark.
| Characteristic |
TPC-E |
TPC-C |
| Age: |
New! |
very old: 14 years |
| Benchmark: |
OTLP database |
OTLP database |
| #tables: |
33 |
9 |
| #transaction: |
10 mixed |
5 lightweight |
| hyperoptimize-able?: |
new, too early to tell |
IBM over-optimised |
TPC has seemed to make some headway on this workload. We'll have to see how
useful it is to judging real-world performance. At least this is a step
in the right direction.
More info at www.tpc.org.
Friday Feb 23, 2007
TPC-E is official. I just saw this on www.tpc.org, fantastic! This is great news! Now maybe
we can get past the 14-year old simplistic TPC-C benchmark. If you've been
following this blog for the last 6-7 blog entries
I've been complaining about IBM and how they over-optimised for TPC-C.
TPC-C is way past its prime -- now hopefully TPC-E will pick up the slack and
provide realistic database performance comparisons.
Details seem to be at http://www.tpc.org/tpce/default.asp
|
|
|
|
