Eoin Lawless

Bad practices exposed

There is an interesting (and frightening!) study recently published in the ACM Transactions on Storage: A nine year study of file system and storage benchmarking together with a short summary of the results and a set of recommendations.

It surveys nine years worth of papers on file system and storage benchmarks and makes for sobering reading:

We found that most popular benchmarks are flawed, and many research papers used poor benchmarking practices and did not provide a clear indication of the system's true performance.

and:

Finally, only about 45% of the surveyed papers included any mention of statistical dispersion.

We can only hope that the paper will raise awareness of the important of rigor and thoroughness in performance benchmarking.

Posted at 09:17PM Sep 01, 2009 by eoin in Sun  |  Comments[0]