Thursday May 26, 2005

Suns Performance Lifestyle : Buzzword Bingo and High Level Bug Evaluation
This post falls out of a converstation I had with my boss earlier this week. We were discussing a (non) issue which had emerged in a conference call that I had attended the week before, where I was asked are there any peformance problems in the latest builds of a new product, my answer was "not directly in the product, but one of the benchmarks has highlighted a problem in build blah of another product", to which I got an answer of "oh thats okay, its not in our test matrix anyway".

Now I'll skip how infuriating that comment was [1] (suffice to say I am of the opinion if there are free cycles available on a rig and you have benchmarked everything that needs to be benchmarked, well add another combination and benchmark that as well, and if its a combination that you have no doubt that a customer is eventually going to deploy it should be in your matrix - idle machines are a waste of valuable resources. Needless to say this applies to QA work as well), and instead use some buzzwords - leveraging synergies [ yes the dilbert link is intentional ].

Synergy is one of these words that upper management in every company loves, and as for leveraging, yes the MBA's are going wild at that one. Anyway what does this "leveraging synergies" mean in reality? In our case it means joining all the dots of what we are doing already, and highlighting this information. Okay, that sounds like MBA speak as well (hey, I thought of doing one at one stage, but then I reflected on Ross Perots famous comment (paraphrasing here) that he "never willingly hired an MBA" [2]). Our initial goal is focused on Solaris, so we run a Solaris matrix that looks like the following

Now the thing is that you as an end user will generally be much more interested in the performance of the app that is running on Solaris, and in our case we need these userland apps to effectively measure the kernel performance. So say we add in a specific version of these apps into our matrix.

Okay, so this just looks like slightly more combinations. Where does the "leveraging synergies" come into this? Within Sun we are not just interested in kernel performance, we are interested in what is known as "complete system" performance, so this means looking at the application as well. The real benefits here (or "leveraging synergies" for the buzzword bingo players) begins when we start revving the versions of the userland applications that we are using in our benchmarks, giving us a matrix like the following...

Now we have a much fuller picture of complete system performance, for relatively little more investment. For the purposes of a practical example lets break this down a bit, say we end up with the following matrix of results after a set of runs of application X on a set of OS images.

So here it is quite obvious that a new patch on Solaris 9 is causing the problem, so this is where we starting looking. Whereas if we saw the following matrix

It becomes pretty obvious that build n+1 of App X is causing our problem in combination with some resource that it is using in Solaris 9 that has evolved in Solaris 10, and we begin our bug hunt by focusing our search there.

The benefit of creating these large combinatorial matrices for Sun, and obviously for the members of our group, is a more worthwhile, and higher value add, usage of our time (and Sun's resources). Narrowing down our inital search field for a bug saves us a huge amount of time, and also allows us to assist the developers in a much more detailed manner by providing a greater amount of pertinent information, and the actual cost is maybe another day or two of benchmarking, which in turn may also highlight other potential problems. And as I mentioned before this all happens automatically, so we don't even have to think about it.