Scoring IBM... didn't come out well for POWER6

Monday Jun 04, 2007

IBM does weird things to make power6 look good. Before the IBM power6 announcement we gave all consumers a big warning of how some corporate marketing games could be played to confuse people.
http://blogs.sun.com/bmseer/entry/beware_the_ides_of_may

Let's see how IBM did... I see lots of red

IBM and the "Sieve of the BM Seer"

"Pay no attention to the man behind the curtain." IBM failed

Here vendors point to peak numbers....

IBM adds random bandwidths inside chip to make it looks like system bandwidth

http://blogs.sun.com/bmseer/entry/ibm_power6_numbers_don_t
http://blogs.sun.com/bmseer/entry/huge_truthiness_in_ibm_marketing

"Don't accept analysis rotten to the core." IBM failed

Here vendors avoid pointing at delivered performance...and system wide perf

IBM cores cost 3 to 4x more than other cores, which means even if it is slightly faster on a per core basis it has worse $/perf

http://blogs.sun.com/bmseer/entry/part2_ibm_power6_cores_extremely
http://blogs.sun.com/bmseer/entry/ibm_power6_cores_extremely_expensive

"Don't believe in paper tigers." Jury Still Out on IBM

Rush out the prototype...

IBM promises to ship on time, we'll also have to see how they do on the rest of the product line

"Don't think things in magnifing glass are as big as real things." Jury Still Out on IBM

Use small systems... claim a new feature that has huge benefit...

Lots of hype of decimal math acceleration, yet to see clear results to directly show benefits on benchmarks on normal workloads

"Don't believe 'A' implies 'C'. 'Sea' implies 'B', therefore 'A' implies 'B'." IBM failed in several claims

Report on one small ...then talk about another configuration

Actually IBM reported on a very old Sun system, and didn't specify except in the fine print that it was 2 years old...

IBM compared Power6 to 2-year old Sun system, with slow CPUs

http://blogs.sun.com/bmseer/entry/ibm_had_to_use_old

They didn't really estimate performance on larger systems from what they introduced, but they did some VERY FUNNY MATH!

IBM derated Sun to 20% utilisation and IBM to 60% utilisation to get fake 3x performance comparison.

http://blogs.sun.com/bmseer/entry/ibm_power6_readers_must_always
http://blogs.sun.com/bmseer/entry/ibm_using_bad_math_to

"Avoid details because someone might use Facts against us" IBM failed, had to add new category

Wow, I had to add a new category to the "Sieve of the BM Seer"

IBM doesn't show details but make watts claims (but no config details, so likely underconfiged to reduce power? We can't tell!) http://blogs.sun.com/bmseer/entry/ibm_perfecting_comparisons_without_details

IBM hides that POWER6 p570 is cabled together cluster

http://blogs.sun.com/bmseer/entry/ibm_power6_really_an_expensive

By the way, this blog isn't any official voice of Sun. It is the voice of an employee at Sun that is a whistle-blower when things just don't sound right.

Since comments are closed, I will put into this text a response to "Anonymouse" here in the text

Have you really looked at the Oracle E-Business Suite benchmark, it is tough to compare results in a meaningful way since there are so many variables.

I want LMbench results to prove that there is no problem with scaling, it is so easy to run that I don't know why some comments keep pointing at other benchmarks instead of answering the question (classic FUD technique). Also interesting that none of the comments say, "yes it is silly that IBM compared against and OLD Sun platform and they used a silly 3x, but BM Seer isn't whole application performance more important than micro benchmarks like LMbench?". To which I would have responded. Yes whole application performance is the most important thing! Knowing latencies and using experience one can really determine how scalable applications are on a wide variety of workloads. I just want to make sure the cables don't get in the way and LMbench would clearly show that that isn't an issue.

I'm really trying to assess the system. All I'm seeing from IBM is hype and meaningless over-hype comparisons (derating Sun by 3x based on utilisation is just silly marketing!).

But no, instead we get FUD about "p570 is a cluster of 4-ways".

It is not FUD it is an easy to answer question of asking for measured latency. Well it is clearly cabled together, LMBench would very easily prove those cables aren't adding latency.

Sun publishes lots of commercial benchmarks and has lots more coming. But you chose to look at none of those. You just point to IBM's latest and avoided the mention of all of the benchmarks that IBM doesn't publish.