Intel defaults and judging performance

Tuesday Sep 18, 2007

Intel non-default BIOS change results by 25%? Sure turning off prefetch is a technique but if you don't know if a priori if you should, then should you use it to judge performance?

Always interesting when you have more information. I guess our friends at AMD wanted everyone to see what our friends at Intel were doing so they submitted two SPEC results for them.

Case in point on Clovertown there are two AMD results on the same hardware that gives 25% difference.

Point: Normal mode = prefetch on
Gives 163,080 SPECjbb2005 bops
www.spec.org/jbb2005/results/res2007q2/jbb2005-20070326-00276.txt

Counter-point: Disable HW prefetcher in BIOS for benchmark imprv
Gives 203,754 SPECjbb2005 bops
www.spec.org/jbb2005/results/res2007q2/jbb2005-20070326-00275.txt

...both on the same hardware:
same: 2-socket SuperMicro X7DBE (Intel 2.66GHz Xeon quad-core X5355), 16 GB

Disclosure statement

SPECjbb2005 SuperMicro X7DBE (2 chips, 8 cores, 2.66 GHz) SPECjbb2005 bops=163080, SPECjbb2005 bops/JVM=81540 submitted by AMD; SuperMicro X7DBE (2 chips, 8 cores, 2.66 GHz) SPECjbb2005 bops=203754, SPECjbb2005 bops/JVM=101877 submitted by AMD; SPEC, SPECjbb are registered trademarks of Standard Performance Evaluation Corporation. Results 3/7/07 on www.spec.org.

AMD, Intel, TDP, ACP...way too much on CPU, what about the system?

Friday Sep 14, 2007

I know AMD & Intel love to focus on the power used by CPUs in their high-stakes battle to gain server chip dominance. Both started talking TDP (Thermal Design Power) and getting people to judge systems based on TDP.

...wait a minute, buying a system based on the power of a CPU is a bit of misdirection unless the CPU is most of the power. This isn't the case any more. So CPU TDP should be ignored, unless you are designing your own product and are just buying CPUs.

First of all system power is what datacenters care about, so Intel and AMD should be talking about system power in realistic memory configurations. Memory draws lots of power these days.

Second,TDP was created so the manufactures of servers manufacturers would know much power the chip consumes in worst-case maximum-power cases so they could design power supplies, cooling, etc. That just isn't useful to datacenter managers.

AMD's marketing only slightly improved the situation by telling customers of SYSTEMS to look at the processor's ACP (Average CPU Power).

Two problems:

• Focus on CPU power to avoid talking system power, but system power is what one needs to know, in average case to estimate electrical bills.
• Focus on average CPU power not server maximums that datacenters need to design cooling on (see: http://blogs.sun.com/bmseer/entry/watts_a_matter_with_their.

ACP of a CPU ...hmmm, do you know what a pain it is to just measure a CPU. AMD in their whitepaper , has to isolate the power consumed by the processor and that consumed by the motherboard -- this requires motherboard modifications and special instrumented server platforms!. way to much work to get a marketing message, all we want is server power on a variety of memory configuations and full-speed CPUs and actually running at good datacenter utilizations!

WARNING: Everyone loves to talk performance of high-GHZ CPUs and low-power of low-GHz CPUs, so watch for the confusing marketing and much worse "bait&switch." Also watts per core is useless marketing, it is the watt/perf for a system that counts. Also any vendor trying to sell power-efficiency on high-performance systems should report watt/performance along with their world record performance on that system.

The Register needs to ask better power questions

Thursday Sep 13, 2007

The register really needs to start asking tougher questions about power. A new article "Researchers: AMD less power-hungry than Intel" by Austin Modine.

This article just takes the results and ignores the really tough questions facing datacenters and power. I'll help here...

• What is the power draw of large memory configurations 16GB, 32GB, 64GB. The 2GB to 8GB used in this report are tiny for those making decisions in many datacenters.
• If you have racks and racks of idle servers (even servers at 30% utilisation) you are major problems and should be counting on new servers to save power!!!
• It is Watt/Performance (like $/perf) not perf/watt, as this highlights efficiency.
• CPU utilisation needs to be reported, if you are measuring at less than 60% you are wasting 2 TIMES to 5 TIMES more power per unit of work!
• Everyone needs to realize the variability of power measurements and not report differences as ##.#% - what? one simply doesn't have one-tenth of 1% reproducibility in power measurements of servers.

Focusing on small memory tries to focus on CPU power which is fine for chip vendors, but it is frankly no what customers I talk to care about. Customers care about full system ("car not piston performance").

The results show that under certain configurations and load levels, the Intel server was 2% to 12% per cent more power efficient. But in a majority of cases, the AMD server was 9% to 23% per cent more efficient.

These are tiny differences. Not enough testing was done to show if these results are consistent (power measurement has big run-to-run variations due to the complexity of most systems.

AMD server was 30% to 53% more power efficient. If accurate, it's a noteworthy figure, considering many servers spend the most of their time waiting for work.

WHAT?!?!? Datacenters idle? this is insane, maybe in extremely poorly run datacenters. Actually, if this the case, those people running the datacenter should find other jobs and not run datacenters.

On the whole, NN&A's tests showed that Intel's power efficiency decreases as memory size increases. Conversely, AMD's power efficiency increases as the memory is upped.

Intel, of course, disputes the results.

"The report doesn't measure our latest Xeons, or quad cores," said Intel rep Nick Knupffer in an email. We have 2 GHz quad cores in the market at 50 watts, 12.5 per core!"

I'll agree with Intel's criticism about latest CPUs and QCs, but even in the Intel statement they quickly talk about 2GHz QC and not the full GHz ones. Everyone loves to talk performance of high-GHZ CPUs and low-power of low-GHz CPUs, so watch for the confusing marketing and much worse "bait&switch." Also watts per core is useless marketing, it is the watt/perf for a system that counts.

NN&A's white paper http://www.worlds-fastest.com/d.pdf/wfw991.pdf

Also take a look at: http://blogs.sun.com/bmseer/entry/saving_the_planet_one_datacenter

If you want to see big power savings (not the small percentages talked about above!) take a look at: http://blogs.sun.com/ValdisFilks/entry/another_win_for_ecological_computing

...it is far too late, enough writing for today.

Performance of the new Sun UltraSPARC T2

Tuesday Aug 07, 2007

Sun UltraSPARC T2 is an amazing chip and very fast! The UltraSPARC T2 features several industry firsts:

• Eight cores and 64 threads
• Integrated 10 GbE networking and I/O
• Dedicated, cryptographic and floating point units per core
• 10 cryptographic functions supported with hardware
• open-source design: www.opensparc.net

Based upon preliminary runs, the Sun UltraSPARC T2 processor at 1.4 GHz, beat all single chip scores showing 78.3 est. SPECint_rate2006. How do these preliminary runs (we must use the term "estimated" by SPEC rules) compare to SPECint_rate2006 results.

• These Sun UltraSPARC T2 1.4GHz processor scores beat the best single-chip IBM POWER6 4.7GHz processor published result by 29%.
• These Sun UltraSPARC T2 1.4GHz processor scores beat the best single-chip estimated scores of the AMD Barcelona by 23%.
• These Sun UltraSPARC T2 1.4GHz processor scores beat the best single-chip published scores of the 2.66GHz Intel X5355 (Clovertown) by 48%.

Based upon preliminary runs, the Sun UltraSPARC T2 processor at 1.4 GHz, beat all single chip scores showing 62.3 est. SPECfp_rate2006. How do these preliminary runs (we must use the term "estimated" by SPEC rules) compare to SPECfp_rate2006 results.

• These Sun UltraSPARC T2 1.4GHz processor scores beat the best published single-chip IBM POWER6 4.7GHz processor result by 7%.
• These Sun UltraSPARC T2 1.4GHz processor scores beat the best single-chip estimated scores of the AMD Barcelona by 11%.
• These Sun UltraSPARC T2 1.4GHz processor scores beat the best single-chip published scores of the 2.66GHz Intel X5355 (Clovertown) by 66%.

Performance per core doesn't matter GHz doesn't matter, what matters is numbers of cores, efficiency, and design of the chip! Competitors are saying that UltraSPARC T2 is proprietary... this makes no sense. both UltraSPARC T1 and UltraSPARC T2 are open source designs (www.opensparc.net). You do not find the latest design of Intel, AMD, or IBM as open source designs.

Disclosure Statement:

All Sun UltraSPARC T2 SPEC CPU metrics quoted are from full “reportable” runs, but are nevertheless designated as “estimates” because they use preproduction systems. SPEC, SPECint, SPECfp registered trademarks of Standard Performance Evaluation Corporation. Sun UltraSPARC T2 1.4GHz (1 chip, 8 cores, 64 threads) 78.3 est. SPECint_rate2006, 62.3 est. SPECfp_rate2006. Competitive results from www.spec.org as of August 6, 2007. IBM POWER6 4.7GHz (1 chip, 2 cores, 4 threads) 60.9. SPECint_rate2006, 58.0 SPECfp_rate2006. AMD Barcelona 2.6 GHz (1 chip, 4 cores, 4 threads) 63.9 est SPECint_rate2006, 56.3 est. SPECfp_rate2006. Barcelona estimates based upon "The Register" article stating 2.6GHz quad is 21% and 50% faster than Intel 2.66 system. Fujitsu RX300 Intel X5355 2.66 GHz (1 chip, 4 cores, 4 threads) 52.8 SPECint_rate2006, 47.5 SPECfp_rate2006.

Reminder: The Niagara 2 score was obtained from a full "reportable" SPEC run, but is designated as an "estimate" because a pre-production system was used.

...more information on the UltraSPARC T2 later today.

Beware the Ides... of May

Tuesday May 15, 2007

Careful reading often required when vendors make claims. I've lumped some of the bad comparisons in the industry into some general classifications. I'll perfect these classifications at a later date.

"Pay no attention to the man behind the curtain."

Here vendors point to peak numbers even though it is easy to actually measure delivered bandwidths.

Action: Ask for delivered bandwidths on memory and IO.

"Don't accept analysis rotten to the core."

Here vendors avoid pointing at delivered performance results on modern benchmarks (you know the ones developed in the past 5 years) and looking at complete systems. They instead construct ratios to things such as cores, threads etc -- AND ignore that everyone has cores and threads that are implemented completely differently and at very different cost structures. Everyone's cores cost completely differently.
Note: My comments on the pre-jurassic TPC-C benchmark, if you think that Sun is avoiding current tests, TPC-C isn't current.

Action: Ask for system performance and system cost, for example: ignore all performance per core comparisons.

"Don't believe in paper tigers."

Rush out the prototype then deliver the world (with un-promised date). Take a long time to avoid showing on current projects

Action: Ask for certified delivery dates, if they get dodgy during the discussion then the don't have it.

"Don't think things in magnifing glass are as big as real things."

This comes in a couple of forms:
First, use small systems to deliver some performance and then imply undelivered bigger systems to be world beaters.
Second, claim a new feature that has huge benefit, show it on one test -- then make it sound like it is also a huge benefits for everything.

Action: Read carefully, don't make assumptions, ask questions, share your analysis with others. If you see a low level performance measurement asks what how it really effects real workloads -- get nervous if they say "your mileage will very." Press for numbers.

"Don't believe 'A' implies 'C'. 'Sea' implies 'B', therefore 'A' implies 'B'."

(ok not as poetic, I'll work on this)
Report on one small or huge configuration (which ever gives you the particular advantage), then talk about another configuration so you think that both are the same. An example of this was one vendor used a small configuration to measure watts, then in the next sentence talked about a big configuration so if you weren't careful you think the wattage rating applied to both.

Action Item: Listen and read very carefully. Ask for config details. Don't make assumptions.

Might be fun to use this "Sieve of BM Seer" to rate any product announcement.

I'm thinking of evolving this and having a regular "Beware the Ides of xxxx" article and see how it develops. Check back on the ides of each month for the next installment.
(ides = 15th)

Intel funny comparisons and calculations

Thursday Mar 22, 2007

Intel is really playing with information, ZDNet writes:

"...instead of comparing Intel's latest greatest chips to AMD's latest greatest chips (as Intel should be doing to legimately convince Wall St., the press, and customers of leadership and/or breakaway performance), more than half of the data points that show Intel leading or breaking away show it doing so against older AMD chips (in some cases, single-core chips or chips from an older generation of Opterons) and in some cases, with retired benchmarks"

SOURCE: "AMD’s no angel, but Intel’s public usage of benchmark data is feloniously misleading" Posted by David Berlind @ 3:33pm http://blogs.zdnet.com/Berlind/?p=366

David missed something we've blogged about here, some Intel systems with normal-size memory use more watts when compared to some systems. Watt & configuration data really needs to be shown somewhere in the Intel presentation mentioned above if a chart is to have validity. http://blogs.sun.com/bmseer/entry/woodcrest_memory_lacks_some_important

and one other thing, in this presentation Intel used perf/$ and perf/watt. As blogged yesterday, everyone needs to use $/perf and watt/perf to really help customers.

Many more reasons that people are paying attention to Solaris

Monday Jan 22, 2007

As I've mentioned here, using Solaris can really help your performance: (ex: http://blogs.sun.com/bmseer/entry/update_solaris_beating_linux_performance

Last night you probably also saw various leaks on yahoo about things that Sun and Intel are up to on Solaris. To get the straight scope go to: http://www.sun.com/2007-0122/feature/index.jsp?intcmp=hp2007jan22_intel_live

Intel chip power & details on Opteron vs. Woodcrest, etc

Friday Dec 08, 2006

Here is a processor wattage chart, but notice that Intel has the memory controller off chip so you need to add 30-35 watts to this figures (opteron includes this on chip) http://www.intel.com/products/processor_number/chart/xeon.htm

For more details on power budget breakdown I got pointed to these two pages for an AMD comparison (looks like it is part of a bigger preso?, no confidential statement): http://www.amd.com/us-en/assets/content_type/DownloadableAssets/opt_vs_wc_8_dimms.pps
http://www.amd.com/us-en/assets/content_type/DownloadableAssets/4_opt_vs_4_pax.pps