BM Seer Unofficial thoughts from an anonymous Sun employee

VMware/VMmark Sun Fire X4450 Best 24 Core Result and watts

Wednesday Jan 14, 2009

The Sun Fire X4450 powered by four 6-core Intel Xeon X7460 processors at 2.66 GHz, has achieved a score of 19.47 @14 tiles, supporting 84 fully fledged VM instances (14 Tiles). This result puts the SUN X4450 on the leading front of 4-socket/24 cores Intel-based VMware servers.

This results beats IBM x3850-M2 result to date of 19.10 @ 14 tiles and all the other Intel based submissions in the 24 cores category with similar architecture while using equal or smaller memory footprint (80GB) then the other systems.

The Sun Fire X4450 server, equipped with four 6C Xeon X7460, provides a highly scalable virtualization platform, that, in combination with the VMware ESX Server 3.5.0 hypervisor, takes full advantage of Intel Virtualization Technology.

To support 14 tiles per server , efficient and reliable Sun StorageTek 2540 Fibre Channel arrays with Sun StorageTek 2501 SAS expansion units were used. This family of arrays offers enterprise-class, reliable RAID-protected functionality in a high-density 2 RU package and is well suited for virtualization environments.

The electrical Power consumption measured for the 4-socket 6C 80GB server during the benchmark runs was on average: 933 Watts (Idle 750 Watts).

The electrical Power consumption for the data storage during the benchmark runs for each disk array and workload type was on average:

  • 1x STK2540 (database workloads): 304 Watts (Idle 296 Watts)

  • 1x STK2540 (Tile workloads except database): 306 Watts (Idle 296 Watts)

  • 1x STK2501 (Tile workloads except database): 261 Watts (Idle 253 Watts)

Sun is the first vendor (and only) to disclose actual power consumption measured while running VMmark, the industry's most popular virtualization workload.

VMmark Performance Results for 24 Cores, Benchmark Score (bigger is better)

System Socket/
Core/
Thrds		 GHz Type GB ESX ver Spindle Mirr Tiles Score Date Pub
Sun Fire X4450 4/24/24 2.66 Xeon X7460 80GB 3.5.0U2 192 n 14 19.47 13-Jan-09
IBM x3850 M2 4/24/24 2.66 Xeon X7460 80GB 3.5.0U2 166 n 14 19.10 17-Sep-08
Dell PE R900 4/24/24 2.66 Xeon X7460 128GB 3.5.0U2 140 n 14 18.96 02-Dec-08
HP PL DL580 G5 4/24/24 2.66 Xeon X7460 96GB 3.5.0U2 168 n 14 18.56 18-Aug-08

Complete benchmark results may be found at the VMware benchmark website http://www.vmware.com/products/vmmark/results.html.

Benchmark Description

VMmark is the first benchmark that was designed specifically to quantify and measure the performance of virtualized environments. It features a novel tile-based scheme for measuring the scalability of consolidated workloads and provides a consistent methodology that captures both the overall scalability and individual application performance.

The VMmark benchmark is built on VMware's expertise in virtualization performance and incorporates popular workloads from application categories most commonly represented in customer data centers.

The purpose of this benchmark is to measure performance and scalability of a pre-established mix of workloads (a Tile), which allows comparisons among similar configuration platforms.

A Tile consists of 6 fixed workload applications, each running in its own Virtual Machine (VM) (6 VMs per Tile) such as Mail, Java, Web, Database and File Serving plus a standby server which only purpose is to provide a spare Virtual Machine that does not do any work and is not accounted in the score.

VMmark benchmark provides two key performance metrics:

  1. The number of tiles supported by a system, which is an indication of how many systems/applications can be consolidated on one platform where the higher the number of tiles supported the higher the number of consolidated systems.
  2. The Score, which is an overall measure of the amount of work that is accomplished by all the Tiles in the system and summarizes the level of service of all the workloads during a benchmark run. The score or amount of work is a composition of Actions/minute(Mail server), New Orders/minute(Java server), Access/minute(web server), Commits/minute(Database), MB/second(file server). Thus, among systems with the same number of tiles, the system with the higher score is the system that is capable of producing the greater amount of work.

For detailed description of VMmark, tiles and score definition, please refer to VMmark Features

Disclosure Statement:

VMware(R) VMmark(R) is a product of VMware, an EMC Company. VMmark utilizes SPECjbb2005(TM) and SPECweb2005(TM), which are available from the Standard Performance Evaluation Corporation (SPEC). Results from http://www.vmware.com/products/vmmark/ as of January 13, 2009.

Result Information
Certified Results
Score: 19.47@14-tiles
Server: Sun Fire X4450
Processors: 4-socket 2.66 GHz 6-core Intel Xeon X7460
Memory: 80 GB
VMware ESX Server: 3.5.0 Update 2
VMmark: 1.1
Storage: 9 x STK2540, 7x STK2501, each storage array using 12x146GB 15K RPM disks

[9] Comments
Like this post? del.icio.us | furl | slashdot | technorati | digg
Posted at 07:40AM Jan 14, 2009 in Benchmark Issues
Tags:
Comments:

You should fix this incorrect statement:

"that, in combination with the VMware ESX Server 3.5.0 hypervisor, takes full advantage of AMD Virtualization Technology."

And if you read the disclosure, Intel Virt technology was disabled as well.

Posted by Ed Kacz on January 14, 2009 at 08:19 AM PST #

Correction: Intel VT was enabled

Posted by Ed Kacz on January 14, 2009 at 08:20 AM PST #

Very good to see the power consumption of the storage reported. Hopefully that will become the norm, particularly as benchmarks (however agonizingly slowly) officially add power to their disclosure.

From the disclosure it looks like some of the 2540's had more cache than the others - is the power measurement from the 1GB cache ones or the 512 MB ones? Under the 2540's the power was broken down for database vs non-database workloads, however the 2501 has only a non-database workload listed. Did none of the 2501's get used for database workloads?

Anyhow, if I've done my sums correctly that is 933 Watts for the SUT and ~9*305+7*251 or 2745 + 1757 or 4502 - ie the storage was nearly 5X (4.83) the power consumption of the system or 80% of the total power consumption. While we have no way of knowing unless they disclose it, given that IBM have used rather fewer spindles and the same processors and quantity of RAM, unless their arrays were piggier they may well have actually consumed less power.

Also of note is that the disclosure states that HW prefetch was disabled on the SUT along with adjacent cache line prefetch. Isn't the "hacking" (sic) of the BIOS against which you rail in the context of SPECpower?

"This family of arrays offers enterprise-class, reliable RAID-protected functionality in a high-density 2 RU package and is well suited for virtualization environments." I'm sure that is an accurate statement, however, it does not appear that this benchmark result demonstrates it as the RAID level is listed as RAID 0. I've not looked at the HP or Dell disclosure, but the IBM disclosure shows they used a mix of RAID 5 and RAID 1. All the more intriguing given their lower spindle count.

FWIW I also do not see any evidence of their "hacking" their BIOS :)

Posted by rick jones on January 14, 2009 at 09:13 AM PST #

Rick Jones: SUT? funny word. For the rest of you, "SUT" is part of the lingo often used in SPEC.org and by computer vendors. "SUT" means "system under test" or _server_.

What do other vendors server watts look like for this size memory? We need that answer before we get distracted with your other questions. Rick since you consider yourself a fair person can you plug in either IBM or or HP SUT configuration into a power calculator and tell us what it comes up as rough guess?

Here is the output from previous work:
http://blogs.sun.com/bmseer/entry/vmmark_performance_watt_performance

I hope our benchmark guys didn't *HACK* to turn off prefetch, I'll try to look into this.

Rick: would you recommend this BIOS change to any customer?

Posted by BM Seer on January 14, 2009 at 11:10 AM PST #

Looks like Sun and IBM are "cheating" by using "only" 80GB when they should have used a "realistic" memory size of 96-128GB. :-)

Posted by Wes Felter (IBM Research) on January 14, 2009 at 12:16 PM PST #

Hmm given 80GB to 96GB is only 20% more memory and 80 to 128GB is only 60% more memory -- really makes you think since 80GB is 5-TIMES (400%) larger than what is used in the largest configurations used in SPECpower.

Wes since you work for IBM and post around the web on power, do you have any power measurements on different size memory that you can share with the community?

SPEC and SPECpower benchmark name are registered trademarks of the Standard Performance Evaluation Corporation

Posted by BM Seer on January 14, 2009 at 01:13 PM PST #

I have some data, although our work is mostly on blades which don't support a lot of RAM. Maybe I can dig it up.

The more I think about it, the more I think SPECpower should be called an SMB benchmark; clearly it doesn't represent consolidated enterprise environments.

Posted by Wes Felter (IBM Research) on January 14, 2009 at 01:20 PM PST #

I think SPECpower really needs to be very explicit about memory size. I think your data even on blades would support this.

I really would like to see power reported on a broad base of different applications. That would go a long way to inform customers.

I agree it clearly is too small memory to represent consolidated enterprise environments. I'd say even enterprise environments given all of the other benchmarks.

I'd suggest SB not SMB.

SPEC and SPECpower benchmark name are registered trademarks of the Standard Performance Evaluation Corporation.

Posted by BM Seer on January 14, 2009 at 01:32 PM PST #

I think I've already commented in one or more of your prior blog entries that I don't believe in using power calculators for guesstimating these things, and your link to:

http://blogs.sun.com/bmseer/entry/vmmark_performance_watt_performance

only reinforces that belief since it shows the rather large discrepancy between a power calculator and actual measured power.

Posted by rick jones on January 15, 2009 at 07:11 PM PST #

Post a Comment:
Comments are closed for this entry.