Beware the Ides... of May
Tuesday May 15, 2007
Careful reading often required when vendors make claims. I've lumped some of the bad comparisons in the industry into some general classifications. I'll perfect these classifications at a later date.
"Pay no attention to the man behind the curtain."
-
Here vendors point to peak numbers even though it is easy
to actually measure delivered bandwidths.
Action: Ask for delivered bandwidths on memory and IO.
"Don't accept analysis rotten to the core."
-
Here vendors avoid pointing at delivered performance results on
modern benchmarks (you know the ones developed in the past 5 years)
and looking at complete systems. They instead construct ratios
to things such as cores, threads etc -- AND ignore that everyone has
cores and threads that are implemented completely differently and
at very different cost structures. Everyone's cores cost completely
differently.
Note: My comments on the pre-jurassic TPC-C benchmark, if you think that Sun is avoiding current tests, TPC-C isn't current.
Action: Ask for system performance and system cost, for example: ignore all performance per core comparisons.
"Don't believe in paper tigers."
-
Rush out the prototype then deliver the world (with un-promised date).
Take a long time to avoid showing on current projects
Action: Ask for certified delivery dates, if they get dodgy during the discussion then the don't have it.
"Don't think things in magnifing glass are as big as real things."
-
This comes in a couple of forms:
First, use small systems to deliver some performance and then imply undelivered bigger systems to be world beaters.
Second, claim a new feature that has huge benefit, show it on one test -- then make it sound like it is also a huge benefits for everything.
Action: Read carefully, don't make assumptions, ask questions, share your analysis with others. If you see a low level performance measurement asks what how it really effects real workloads -- get nervous if they say "your mileage will very." Press for numbers.
"Don't believe 'A' implies 'C'. 'Sea' implies 'B', therefore 'A' implies 'B'."
-
(ok not as poetic, I'll work on this)
Report on one small or huge configuration (which ever gives you the particular advantage), then talk about another configuration so you think that both are the same. An example of this was one vendor used a small configuration to measure watts, then in the next sentence talked about a big configuration so if you weren't careful you think the wattage rating applied to both.
Action Item: Listen and read very carefully. Ask for config details. Don't make assumptions.
Might be fun to use this "Sieve of BM Seer" to rate any product announcement.
I'm thinking of evolving this and having a regular "Beware the Ides of xxxx"
article and see how it develops. Check back on the ides of each month
for the next installment.
(ides = 15th)
|
|
|
|
