TPC Benchmarks Don't Matter Anymore (Forrester, March 6, 2009)

Thursday Mar 12, 2009

TPC Benchmarks Don't Matter Anymore by Noel Yuhanna with Mike Gilpin, David D'Silva (Forrester)

Forrester authors write:

Transaction Processing Performance Council (TPC) benchmarks, once widely accepted as the standard DBMS benchmark, are becoming obsolete. Why? First, all top-tier DBMS vendors such as IBM, Microsoft, Oracle, Sybase, and Teradata are delivering high performance and scalability to support most large workloads.

Second, TPC benchmarks no longer reflect the complex workloads of today's real-world deployments. Third, customers that need high-end performance often prefer internal benchmarks to TPC benchmarks.

Finally, virtualization, cloud computing, and database-as-a-service are changing the way customers deploy databases, and TPC does not address these architectures. Enterprise architects performing assessments should not waste their time on TPC benchmarks of top-tier DBMS products, and tech industry marketers and product managers should redirect the millions they spend on benchmark engineering toward automated tuning and performance optimization

I've been saying for years these things about the TPC-C benchmark!!!
http://blogs.sun.com/bmseer/tags/tpc-c

We even know that IBM has tuned it useless: IBM's TPC-C "tuning"(?) that won't apply to anything in the real world

June 2005 Interview with Bruce Lindsay (IBM Fellow) at http://www.sigmod.org/sigmod/record/issues/0506/p71-column-winslet.pdf

"And the good news is that about 40-70% of the stuff we do in performance tuning actually ends up helping end users."

Who will have the maturity to kill TPC-C. Sun has quite publishing a while back, remember when Sun had the world record TPC-C and in that announcement Sun said: "It's well-understood in the technical communities that TPC-C no longer represents current customer workloads since the transaction load that its models are made of are small, primitive and disconnected transactions. While this model was acceptable for the workloads of the late 1980s, it misses the mark..." Sun's World Record TPC-C Press release, August2000

Disclosure Statement

TPC-C results referenced above was the fastest overall performance world record at August 31, 2000. Sun Enterprise 10000 server (Starfire) running Sybase Adaptive Server Enterprise (ASE), 156,873.03 transactions per minute (tpmC), $48.81 price/tpmC, available February 28, 2001. A full disclosure report and executive summary are available through the TPC Web site located at www.tpc.org.

Like this post? | | | |

Posted at 10:20AM Mar 12, 2009 in Benchmark Issues
Tags: benchmarks tpc-c