• Java Performance Community
• Java Performance Tech Topic

Tuesday July 25, 2006

Java Benchmarking Folksonomy

One of the hot topics in the Java Libre crowd is around performance benchmarking and the tools and infrastructure to be successful at it.

I've thought about how to represent the different components of the story: from microbenchmarks to macrobenchmarks, load drivers, test harnesses, statistics tools and results visualization. Here is a fantastic space for Open and Free collaboration -- this is a common need and we often end up reinventing the wheel.

Of course those that know me know that I'm big on collaboration. A recent essay by Larry Sanger defines Strong Collaboration. This is a very important work in it's own right and merits the time to read it and grok it [ with a cup of Java ;-) ]. For my purpose here I'm looking for collaboration on the components, but also the meta information: how to find existing work and avoid duplication of effort.

That's why organizing the different facets of benchmarking cannot really live on a static web page and invite collaboration. My first instinct is to grab a Wiki... but many people are shy about editing Wikis and they are notoriously flat. So as I'm trying to develop a collaborative taxonomony it seems the logical approach is to hack del.icio.us as our Open Source Java Benchmarking Folksonomy.

So after a little research I decided that everything in this folksonomy will have the tags opensource java benchmarking. Then additional tags can point to specific areas, such as:

• benchmark a benchmark
• microbenchmark a microbenchmark
• macrobenchmark a macrobenchmark
• suite a set of many benchmarks
• loaddriver a synthetic load generator
• harness a benchmarking script/program
• stats statistics calculation tools for benchmarks
• visualization tools for viewing results
• results publication of specific results
• article a review of benchmarking and/or tools
• index a directory of tools

• benchmark (tmarble: benchmark)
  One challenge I had right away is around the exact definitions of microbenchmark and macrobenchmark. Another challenge is that the japex project is really an infrastructure for creating microbenchmarks. So right away we have some semantic challenges. At least for starters I encourage you to check out the Beware of Microbenchmarks! section of the Java Performance Tuning White Paper.
• suite (tmarble: suite)
  Well, I'm in trouble again because suite could mean, literally, that one benchmark is comprised of some very specific sub-benchmarks. Alternatively it could be just a grabbag of different benchmarks. I'll let this ambiguity remain for now.
• loaddriver (tmarble: loaddriver)
  Sun has contributed nicely to this space of synthetic load generation.
• harness (tmarble: harness)
  I'm sorry to say I haven't found a Free benchmarking Framework that is a standalone harness. I'm sure every one of the suites has some kind of harness, but I haven't found a generic one. The value of a harness is outlined in Use a benchmark harness.
• stats (tmarble: stats)
  Like the harness I haven't tagged anything here yet. This is tricky because I'm sure there are some statistics tools in Java -- don't know if there are any standalone packages (i.e. for calculating the student's T-Test). See also Use Statistics. While looking at other's stats tags I found this link which made me chuckle: Programmers Need To Learn Statistics Or I Will Kill Them All.
• visualization (tmarble: visualization)
  This is the fun part... actually using your inner Don Norman to show people what the data says.
• results (tmarble: results)
  As we get better at doing this benchmarking Libre I expect we will see many more publications.
• article (tmarble: article)
  Tag with article for whitepapers, reviews, columns and otherwise human readable discussion on Java benchmarking. we will see many more publications.
• index (tmarble: index)
  And, of course, the whole concept of open collaboration is that we are standing on the shoulders of giants. Use index to tag other taxonomies on Java benchmarking.