ISV/IHV Engineering News ISV Engineering

On the heels of the ISV Sun T5440 performance announcements, we've released an impressive SAP benchmark on the Sun Fire X4600M2 system.  This is the best 8 processor result¹ in the business and shines in a number of interesting areas:

• Surpasses the HP DL 785 result using the same basic configuration (5800 users versus 5230)
  
• Uses SAP's bundled "no aquisition cost" MAX-DB (who needs those SQL Server or Oracle licenses?)
  
• Solaris 10 once again shows superior performance compared to Microsoft Windows Server 2003
• An apples-to-apples comparison? Not exactly.  This result used the more difficult Unicode version of the benchmark versus the non-Unicode (2005) version
  

Expanding on the last very important point, as BM Seer and Heise Online Magazine [German original] have pointed out, the superior Sun benchmark result was achieved with the Unicode 6.0 version of the benchmark.   According to SAP documentation, the Unicode version requires up to an estimated 30% more cpu to execute (as well as other greater resource requirements).  This is like having a 160 lbs (72.5 kg) swimmer strap on another 50 pounds before a race and still finish first by a comfortable margin.   I particularly liked the comments from to the Heise article.   One reader complained that the unicode thing as an arcane point that no one would catch or care.  The response to that was something like "hey, I'm blogging about it aren't I?" and "this is on purpose - Sun is an engineering company" (I'm paraphrasing).

Disclosure Statement:

¹SAP and all SAP logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries. For more details, see http://www.sap.com/benchmark. Two-tier SAP SD Standard Application Benchmark - Configurations:
Sun Fire X4600M2, 8 Proc. / 32 Cores / 32 Threads running SAP ERP 6.0 (Unicode) and MaxDB 7.6 on Solaris 10. Result 5,800 SD Benchmark users, 29,670 SAPS, certification number 2008061
HP ProLiant DL785, 8 Proc. / 32 Cores / 32 Threads running SAP ERP 6.0 and SQL Server 2008 on Windows Server 2003 EE. Result: 5,230 SD Benchmark users, 26,180 SAPS, certification number 2008026

This week, Sun announced a new 4-chip UltraSPARC T2 Plus system - delivered in a compact 4U, energy efficient, and low cost package. We expect this systems to be at the center of many ISV applications that have heavy scalability requirements   Here's how our colleague Allan Packer describes the system.

Sun today announced the 4-chip variant of its UltraSPARC T2 Plus system, the Sun SPARC Enterprise T5440.   Each UltraSPARC T2 Plus chip offers 8 hardware strands in each of 8 cores. With up to four UltraSPARC T2 Plus chips delivering a total of 32 cores and 256 hardware threads and up to 512Gbytes of memory in a compact 4U package, the T5440 raises the bar for server performance, price-performance, energy efficiency, and compactness. And with Logical Domains (LDoms) and Solaris Containers, the potential for server consolidation is compelling.

Our team in ISV Engineering has had access to early production systems to test ISV application workloads.   We always have more applications queued up for testing than there is time and equipment available, but here is what we were able to highlight by the announcement time:

• Since the mid-1990's SAP AG has worked closely with its hardware partners on developing reliable and consistent benchmarks to measure the performance of their leading ERP and business application software.   The most popular of these has been the SAP SD benchmark which mimics a sales and distribution workload with simulated users booking and shipping orders.   The T5440 delivers the best 4 processor performance in the industry at 7510 concurrent users.¹  I worked with the Sun team that produced some of the first world-record SAP benchmarks back in 1996.   Today, if you dropped a T5440 on the ground, it might just chip the pavement.   If you dropped the "kit" we used to produce the 600 user benchmark in 1996, it would have made a crater.
  
• We've been working with the Siebel PSPP workload since long before they joined forces with Oracle.   The T5440 turned in an impressive 14,000 users besting our own previous T5120 performance of 10,000 users and absolutely leaving behind entries from IBM & HP.   Check out Giri Mandalika's great post on the results and comparsons to other vendors.
  
• Our involvement with SugarCRM dates to the beginning of the Open Source revolution.  Sun's CMT servers have been a natural fit for their workload that tests hardware platform capability.  Our result of 3200 users with the T5440 showed near linear scalability from our previous work on the T5440's younger brother, the T5120 (900 users).    Furthermore, this benchmark featured the use of Logical Domains (LDOMs) which I've mentioned on an earlier post concerning the S10 Binary Guarantee Program.
• The SPECjAppserver2004 benchmark stresses both the Java application server and the database.   The T5440 turned in the best single application server performance² for 4 chip systems using Oracle's Weblogic 10.3 and the Oracle Database 11g DB 11.1.0.7.
  
• Not every ISV has a standard benchmark kit- nor do we have enough equipment or manpower to test them all.  Nevertheless, we were able to do some good performance analysis of IBM Websphere & DB2 running on the T5220.   In addition, we were able to secure the endorsements of 19 ISVs who were able to see the compelling value of the T5440.
  

We'll be releasing more great performance results based on the T5440 system in the near future.   The price-performance, low energy, low footprint aspects of the system have made it a favorite topic in discussions and project proposals with our ISV partners.

Update!  Sun has a page dedicated to benchmark excellence. 

Please check out: http://www.sun.com/benchmarks/

¹Disclosure Statement for SAP SD benchmark:

Two-tier SAP Standard Sales and Distribution (SD) standard SAP ERP 6.0 (2005) application benchmark: SPARC Enterprise Model T5440 (4-way, 4 processors, 32 cores, 256 threads) 4 x 1.4 GHz UltraSPARC T2 Plus, 128GB memory, 7520 SD Benchmark users, 1.98 sec avg response time, Cert#2008058, Oracle 10g, Solaris 10; SPARC Enterprise Model T5240 (2-way, 2 processors, 16 cores, 128 threads) 2 x 1.4 GHz UltraSPARC T2 Plus, 128GB memory, 4170 SD Benchmark users, 1.97 sec avg response time, Cert#2008021, Oracle 10g, Solaris 10; SPARC Enterprise Model T5120 (1-way, 1 proc, 8 cores, 64 threads) 1 x 1.4 GHz UltraSPARC T2, 64GB memory, 2175 SD Benchmark users, 1.91 sec avg response time, Cert#2007059, Oracle 10g, Solaris 10; IBM System p 570 (8-way, 8 processors, 16 cores, 32 threads) 8 x 4.7 GHz POWER6+, 128GB memory, 8000 SD Benchmark users, 1.98s avg resp time, Cert#2007039, DB2 9, AIX 5L Version 5.3; IBM System p 570 (4-way, 4 processors, 8 cores, 16 threads) 4 x 4.7 GHz POWER6+, 64GB memory, 4010 SD Benchmark users, 1.96s avg resp time, Cert#2007038, Oracle 10g, AIX 5L Version 5.3; IBM System p 550 (4-way, 4 processors, 8 cores, 16 threads) 4 x 4.2 GHz POWER6+, 64GB memory, 3104 SD Benchmark users, 1.91s avg resp time, Cert#2008002, DB2 9, Redhat Enterprise Linux 5; HP ProLiant DL580 G5 (4-way, 4 processors, 24 cores, 24 threads) 4 x 2.66 GHz Intel Xeon, 64GB memory, 5155 SAP SD Benchmark users, 1.97 sec avg response time, Cert#2008050, SQL Server 2005, Windows Server 2003 Enterprise Edition; HP ProLiant BL680c G5 (4-way, 4 processors, 24 cores, 24 threads) 4 x 2.4 GHz Intel Xeon, 64GB memory, 4432 SAP SD Benchmark users, 1.99 sec avg response time, Cert#2008049, SQL Server 2005, Windows Server 2003 Enterprise Edition; HP ProLiant DL585 G5 (4-way, 4 processors, 16 cores, 16 threads) 4 x 2.5 GHz Quad-Core AMD Opteron, 64GB memory, 3801 SAP SD Benchmark users, 1.99 sec avg response time, Cert#2008041, SQL Server 2005, Windows Server 2003 Enterprise Edition; HP ProLiant BL685c G5 (4-way, 4 processors, 16 cores, 16 threads) 4 x 2.3 GHz Quad-Core AMD Opteron, 64GB memory, 3524 SAP SD Benchmark users, 1.98 sec avg response time, Cert#2008016, SQL Server 2005, Windows Server 2003 Enterprise Edition; HP ProLiant DL385 G5 (2-way, 2 processors, 8 cores, 8 threads) 2 x 2.3 GHz Quad-Core AMD Opteron, 32GB memory, 2102 SAP SD Benchmark users, 1.99 sec avg response time, Cert#2008015, SQL Server 2005, Windows Server 2003 Enterprise Edition; HP Integrity BL860C (2-way, 2 processors, 4 cores, 8 threads) 2 x 1.66 GHz Dual-Core Intel Itanium, 24GB memory, 1165 SAP SD Benchmark users, 1.94 sec avg response time, Cert#2008052, Oracle 10g, HP-UX 11i V3; SAP, R/3, mySAP reg TM of SAP AG in Germany and other countries. More info www.sap.com/solutions/benchmark.

Results Summary

Certified Results
	Performance:		7520 benchmark users
	Server:		Sun SPARC Enterprise T5440
	Processors:		4 x 1.4 GHz UltraSPARC T2 Plus
	Memory:		128 GB
	Operating system:		Solaris 10
	Database S/W:		Oracle 10g
	SAP S/W:		SAP ECC 6.0
	SAP Certification:		2008058
	Storage:		3 x Sun StorageTek(tm) 2510 3 x Sun StorageTek(tm) 2540 2 x Sun StorEdge 6120

²SPECjAppServer2004
Sun SPARC Enterprise T5440 (4 chips, 32 cores) 5836.15 SPECjAppServer2004 JOPS@Standard. HP DL580 G5 (4 chips, 24 cores) 4410.07 SPECjAppServer2004 JOPS@Standard. HP DL580 G5 (4 chips, 16 cores) 3339.94 SPECjAppServer2004 JOPS@Standard. Two Dell PowerEdge 2950 (4 chips, 16 cores) 4794.33 SPECjAppServer2004 JOPS@Standard. Two Dell PowerEdge 1950 (4 chips, 16 cores) 3593.68 SPECjAppServer2004 JOPS@Standard. SPEC, SPECjAppServer reg tm of Standard Performance Evaluation Corporation. Results from www.spec.org as of 10/13/08

"Performance" or "Price/Performance" consistently rates as one of the top reasons that customers choose one technology vendor over another (Top Reason: "Relationship with Current Vendor").    Sun's product offerings are currently in very good performance position in the industry.   The M9000 series holds the world record for SAP and Lotus Notes performance and Sun consistently outperforms the competition in the popular SPECjAppServer benchmark.   This is the public face of application performance benchmarks that usually gets all of the press and are hotly contested among all of the major systems manufacturers.  

Not surprisingly, ISV Engineering is often engaging with our partners to perform benchmark tests, performance tuning, and capacity planning (sizing studies) projects.   Despite all of the headlines, most of our partners do not have a public face on their performance tests and provide them only to make sure that the customers buy the right system for the right purpose.   Even though these tests are usually derived from actual customer experiences, they are "artificial" in the sense that they are done in a lab under somewhat ideal conditions.

When a potential customer talks with Sun, they are very naturally interested in how their applications perform with the systems they are intent on acquiring.   They are rightly concerned that the benchmark offered as proof of adequate or superior performance actually represents the reality that they will see after deployment.   The answer is that although the test is as close as possible, it still cannot give precise information.   However, this is no reason to invalidate the testing or throw out all of the effort put into ISV application performance testing (and it can be considerable). 

I offer the customer these additional things to think about :

• The benchmark test that is run on the SUT (Systems under Test) will put unusual stress the system and configuration.  This is especially true in benchmarks or sizing tests that are so called "moon shots" which seek to test the largest configuration possible.  This stress can cause the setup to develop a bottleneck or even to break entirely.  Oftentimes what is discovered is that the system will perform the workload, but only utilize a fraction of the compute power that is in the SUT.
• The bottleneck, breakage, or low utilization will be subjected to root cause analysis and the system/software configuration adjusted or parts of the application code rearchitected.  The system will be tested again until a measure of success is achieved.   This improves the inherent stability and robustness of the ISV software and platform vendor solution in the long run.
  
• By its cyclical nature, this process forces the application vendor and Sun Microsystems to work together to overcome the obstacles presented by the test.
• The working relationship that is developed during these tests is instrumental in providing the foundation for future cooperation during real customer support escalations.  That is, if and when an escalation comes to the performance team on either side, we know who to call and have confidence that we can work together towards a solution.

I have seen these points proven true for countless ISVs that we have worked with over the years.   Those ISVs that are most successful end up treating their performance/sizing benchmark as a kind of special product.  Complete with versions, documentation, packaging, and bug reporting.   These vendors have figured out that investment in a good performance/sizing workload is not only a wise investment for their customers, but a great way to develop a close engineering relationship with a technology provider like Sun Microsystems.

As ISVs increasingly find their applications deployed into virtualized environments, the concept of supported Operating System compatibility becomes blurred.   For example, if an ISV tests their application on Solaris 10 update 5 on x64 and lists the compatibility on their website, what if the target customer installs it as a guest on a virtualized platform like xVM or VMware?   Will the application still be supported and behave as if it was on the actual OS next to the metal?

Sun has just provided a little bit of peace-of-mind in this area by updating the Solaris 10 Binary Application Guarantee Program to include the guarantee for virtualized platform environments.   The guarantee has been updated to say:

The Binary Program applies to both Solaris native host operating system environments, as well as Solaris 10 OS running as a guest operating system in a virtualized platform environment.

The list of supported virtual platform environments is kept on the Solaris OS: Hardware Compatibility List.   The specific virtual environments that the guarantee applies to are a little hard to find on this big list, but here are the popular ones: LDOMs, xVM, & VMware.

Sun has done a pretty good job over the years (I've been at Sun 21 of those years) at protecting ISV binary compatibility and ensuring forward guarantees as new technology and OS releases (including virtualization) become available and popular.

I lead a group named "ISV Engineering" within Sun Microsystems.   Our team makes sure that key ISV applications, IHV Hardware, Open Source, and startup customers work great with Sun technologies.  We do projects like these:

• Technology adoption (e.g. adoption of OpenSolaris)
• Technology Education (e.g. Mobile Ajax at SunTech Days)
• Work with Open Source communities to get the most out of Sun (e.g. SugarCRM)
  
• Capacity & Configuration Planning (e.g. Helios UB+ Sizing & Tuning)
• Application Developer News & Insights (e.g. Accelerate Application Builds with DMake)
  
• ISV related Benchmarks (e.g. SAP world records)
• Sun Partner Advantage Program (SPA) Developer Support (e.g. ISVs can get no-cost developer support here or simply send your developer related question to isvsupport@sun.com)
  
• Supporting ISVs & startups who want to get on Sun through hosting (e.g. Amazon EC2 for OpenSolaris) 
• Lots of blogs and articles on interesting technology that we work with (e.g. Herrington ... Rajadurai ... Startup Team ... Kumar ... Zhu(中文) ... Drapeau ... van den Berg ... Mandalika ... Hasham ... India Open Source ... Glore ... Chen ... Jignesh ... Langston ... Shankar ... MShetty ... Mithun ... Reid ... van den Bogaard ... Hosanee ... Praneet ... Prashant ... Lor ... Nair ... Sidharth ... Schneider et. al. ...Daly ... Vanga ... ChrisZhu ... )
  

We've also been attending and presenting at a lot of start up events (e.g. Ruby, LAMP, Facebook Meetups, Mashable, Proto.in, DrupalCon, etc).  I'll be pointing out the interesting and significant work that we're doing.   Please point me to other relevant links and news!