"Performance" or "Price/Performance" consistently rates as one of the top reasons that customers choose one technology vendor over another (Top Reason: "Relationship with Current Vendor"). Sun's product offerings are currently in very good performance position in the industry. The M9000 series holds the world record for SAP and Lotus Notes performance and Sun consistently outperforms the competition in the popular SPECjAppServer benchmark. This is the public face of application performance benchmarks that usually gets all of the press and are hotly contested among all of the major systems manufacturers.
Not surprisingly, ISV
Engineering is often engaging with our partners to perform benchmark
tests, performance tuning, and capacity planning (sizing studies)
projects. Despite all of the headlines, most of our partners do not have a public face on their performance tests and provide them only to make sure that the customers buy the right system for the right purpose. Even though these tests are usually derived from actual customer experiences, they are "artificial" in the sense that they are done in a lab under somewhat ideal conditions.
When a potential customer talks with Sun, they are very naturally interested in how their applications perform with the systems they are intent on acquiring. They are rightly concerned that the benchmark offered as proof of adequate or superior performance actually represents the reality that they will see after deployment. The answer is that although the test is as close as possible, it still cannot give precise information. However, this is no reason to invalidate the testing or throw out all of the effort put into ISV application performance testing (and it can be considerable).
I offer the customer these additional things to think about :
- The benchmark test that is run on the SUT (Systems under Test) will put unusual stress the system and configuration. This is especially true in benchmarks or sizing tests that are so called "moon shots" which seek to test the largest configuration possible. This stress can cause the setup to develop a bottleneck or even to break entirely. Oftentimes what is discovered is that the system will perform the workload, but only utilize a fraction of the compute power that is in the SUT.
- The bottleneck, breakage, or low utilization will be subjected to root cause analysis and the system/software configuration adjusted or parts of the application code rearchitected. The system will be tested again until a measure of success is achieved. This improves the inherent stability and robustness of the ISV software and platform vendor solution in the long run.
- By its cyclical nature, this process forces the application vendor and Sun Microsystems to work together to overcome the obstacles presented by the test.
- The working relationship that is developed during these tests is instrumental in providing the foundation for future cooperation during real customer support escalations. That is, if and when an escalation comes to the performance team on either side, we know who to call and have confidence that we can work together towards a solution.
