VM Benchmarking Wars

Several months ago I wrote about the coming virtualization benchmark wars.  It seems that Oracle has fired the first shots and now VMware is starting to fire back.  It's been widely reported that Oracle is claiming their VM is 3x faster than VMware.  These kinds of claims are sure to get a lot of press.  However, when you look at them closely, they're totally meaningless.

As I've mentioned before, one of my first jobs at Sun was doing benchmarking and performance for Java.  During that time, I became pretty familiar with all the dirty tricks vendors play with benchmarks.  This latest spat between VMware and Oracle looks to be a classic example.  Despite the reports about 3x faster, what Oracle said was actually, "Oracle consistently saw much better resource utilization with an average of three times less overhead using Oracle VM."  In benchmark-speak, that's a lot different than 3x faster.  In fact, I'm guessing that someone constructed a benchmark that shows Oracle VM running at 99% of native speed, while VMware ran at 97% of native speed -- 1% overhead instead of 3%.  Yes, "three times less overhead", but ultimately trivial.

Of course, the the long term, there will be better official benchmarks of VM performance, but the real test will be customer metrics like

• How fast can I roll out a new application?
• How many system admins do I need to run my data center?
• How much energy can I save by operating my data center in a new way?

Posted at 10:18AM Nov 21, 2007 by Stephen Wilson in Virtualization  |  Comments[2]