ISV/IHV Engineering News ISV Engineering

As mentioned earlier, Oracle's Real Application Cluster (RAC) 10g is fully supported and certified on Sun's Logical Domains (LDOMs) feature.    This feature, which allows and controls the dynamic allocation of systems resources, is available on Sun's CoolThreads Servers running Solaris.   This line of products is commonly known as "CMT" (Chip Multi-Thread) and provide up to 256 threads per system.

A comprehensive Sun BluePrint™, "Running Oracle Real Application Clusters (RAC) on Sun Logical Domains)", has just been released that describes (in 40 page detail) how to understand, install, and deploy the Oracle RAC software into the Solaris LDOMs environment.   This is divided into four parts:

1. An introduction which provides background information on RAC & LDOMs and discusses the various options
2. Requirements in getting the correct software and hardware environment
3. Actual install & configuration of the software including how to handle network and storage considerations.
4. Finally, an additional section which discusses RAC performance in this environment and goes over known issues.

The blueprint was a joint cooperation between Oracle and Sun and was written by Daniel Dibbets (Oracle), Alexandre Chartre (Sun), and Roman Ivanov (Sun, ISV Engineering).

ISV Engineering performs hundreds of small and large projects every month with our ISV application partners.   To track the dispatch, status, delivery, and reporting of projects, we use the popular TWiki^® Open Source Enterprise Wiki and Web 2.0 Application Platform-  which we run within Solaris Containers on a shared Sun Fire T2000 Server.   We've been able to implement a full workflow using this tool.  Here's an example of one of the tool views (heavily simplified).

Projects In Execution Phase

Project Update/Edit	Status	Phase	Project Type / Metric	Project Name	ISV/Community	Product & Version	Etc...
Project Link 1	Green	Execution	Benchmark	ERP  Benchmark	MyERP Inc.	ERP 5.0	More columns describing the projects...
Project Link 2	Green	Execution	Sizing Study	Open Source Sizing Study	Open Source	Open Source 1.0 or OSS 6.x

Our experience with implementing this system has also lead to greater understanding of how to tune and scale social & Web 2.0 applications like TWiki on highly threaded systems like the T2000.  

Nagendra Nagarajayya in ISV Engineering has written a white paper in the popular Sun Blueprints™ series detailing the best practices in deploying TWiki software on Sun's open source GlassFish application server.   He describes how utilizing the Java long-running web process (LRWP) protocol and the the Solaris™ Operating System enables the platform to execute twice the speed of an equivalent Apache implementation which results in reduced system overhead and the ability to accommodate more users.

Oracle 10g certification on Solaris SPARC using the ZFS file system is now complete.   Oracle has a support note posted in the certification section on metalink (support registration required).   This brings the huge advantages of ZFS (simple administration, transactional semantics, end-to-end data integrity, and immense scalability) to implementations of Oracle Database 10gR2 (10.2.0.3 and higher patches).   This applies to single-instance (non-RAC) deployments on Solaris 10 SPARC.   Two guides will come in handy when setting this up:

• Solaris Internals - ZFS for Databases
• Solaris Internals - ZFS Best Practices Guide
  

We'll be working on more best practices documents for the ZFS & 10g combination, in particular, support for more advanced ZFS features.   As mentioned in earlier posts, LDOMs and Containers also have certification with Oracle 10g.

Continuing the expansion of Oracle products certified on Sun's Logical Domain (LDoms) feature, we've added Oracle Weblogic Server 10gR3 (10.3) to the list.

Working on a Sun's CoolThreads Server with Solaris, LDOMs technology allows the allocation of a system's resources into logical groupings.    Each group is a discrete system with its own operating system, resources, and network identity operating within a single physical computer system.  This allows an administrators to protect and isolate the groups from each other using a hardware/firmware combination.

With this certification, Oracle will fully support customer installations that deploy the Oracle Weblogic Server 10.3 in an LDom partition.   This will allow multiple Oracle Weblogic instances to operate independently on a single Sun CoolThreads (CMT) Server in full hardware level isolation.

James Liu has an in-depth posting on how to convert a $299 Acer AspireOne into a dual boot Solaris/Windows netbook.   Since I had acquried this netbook recently for a trip to China, I thought it would be a perfect opportunity to jump back into the OpenSolaris/Solaris desktop world.  It is certainly a better solution than dragging around a huge & expensive 17" Macbook.

The netbook conversion took about 3hrs with only a couple of false starts.   Thanks to James for getting everything setup and ready to go.  Here are the system features that are working with no problems:

• Startup time - about 1 minute
• Suspend/Resume - upon close of the lid
• Video Camera - works with Ekiga
• Wireless Networking
• USB stuff (keyboard, mouse, sticks, etc.)
• SD cards (but need to boot up with them already inserted)
• Windows on a dual boot partition in 40GB
• rsync command for keeping the Mac & Acer files synchronized
  

Here are the applications that are working great:

• Thunderbird 2.0.0.19 with Lightning Calendar plugin
• Firefox 3.0.8 with ScrapBook, ColorfulTabs, GooglePreview, FlagFox, & RSS Ticker plugins
• StarOffice 8  (OpenOffice.org)
  
• GIMP Image Editor
• Punchin VPN for corporate connectivity
  
• Ekiga Video Conference
  
• Pidgin IM Client
• mplayer for music & video
  

Here are the things that will need some more work to get going:

• Network Printer.   HP Officejet C6150
• Freeciv - a free simulation game.   If someone already has this compiled & working on Solaris/OpenSolaris, please provide a pointer.
  
• Play nice with ViewSonic 21" monitor
• Support for xD cards.   This is coming soon in an OpenSolaris update.

The Solaris platform on small devices has really come a long way.   Solaris/OpenSolaris on this device does about everything needed for a mobile device.   Most of the software is available at: http://www.sunfreeware.com/

Sun announced is vision for Cloud Computing this week at the CommunityOne conference.   One of the key demonstrations (video) by Dave Douglas and Lew Tucker was the ability for cloud users to create a "Virtual Data Center" (VDC) by dragging and dropping system components from a palatte and linking them together into a scalable and robust service.

The demo showed a conceptual "Mediawiki" architecture implementation based on the same type of workload as one might expect from a service similar to the popular Wikipedia.  It showed:

• It's very simple to bring in components (switches, load balancing servers, web servers, dbms servers, etc) and link them together to create a complete, robust, and scalable service.   The demo dynamically added load balancing servers and showed how web accesses would cycle through these servers.
  
• Management of the VDC environment is very holistic.   It's easy to see the whole implementation as one unit rather than have to deal with each individual (physical) machine.
• A great advantage of being in the cloud, as opposed to an actual physical implementation, is that it is very straightforward  to replicate the VDC and package it up for use somewhere else (try that in an actual data center)

You can sign up for more information and the latest developments on the Sun Cloud here.   More information on the architecture of the Wikipedia/Mediawiki is available from Alka of ISV Engineering.

In my last post, I forgot to mention another MySQL related Sun Blueprints™ that was recently published: MySQL Guide for Sun Storage 7000 Unified Storage Systems by Bill Aiken and Krishnan Shankar.  MySQL and the Sun Storage 7000 is a great marriage because it delivers good performance, low cost, combined with ease of management.   Configuring system I/O can be a complex and frustrating task even for experienced systems administrators.   Start ups and smaller companies using MySQL typically just don't have the skill or time to tune the I/O interface of their database.   Here the Sun Storage 7000 really shines by delivering great performance, ease of management, and lower cost that can take away a lot of the worry about managing the I/O performance of MySQL.

Here is the table of contents:

• MySQL Guide for Sun Storage 7000 Unified Storage System
• Sun Storage 7000 Unified Storage System
• Setup and configuration
• Installing and configuring MySQL software
• Configuring NFS access
• Configuring iSCSI access
• Database snapshots and cloning
• Test results
• Summary

The ISV Engineering team at Sun has released 3 new documents giving important MySQL implementation and performance advice on Solaris Containers, Java technology, and SugarCRM.

The first is another great publication in the Sun BluePrint™ Series called Running MySQL Database in Solaris Containers by Ritu Kamboj and Giri Mandalika.  Solaris Containers are a Sun unique virtualization option that let applications run on the same server in isolation from each other.    This allows easier workload resource management, maintenance of service levels, and superior performance compared to other virtualization methodologies. The paper discusses the various requirements & options when installing MySQL into this environment.

The second paper is an article on the Sun Developer Network (SDN) called Using MySQL With Java Technology by Giri Mandalika (a prolific ISV Engineering writer).    He covers these four key areas:

• Accessing MySQL With Java/JDBC
• MySQL Connector/J JDBC Driver
• Installing MySQL Connector/J
• Testing the MySQL Database Connectivity Using the Connector/J JDBC Driver

While not specifically concentrating on MySQL, Satish Vanga's presentation to SugarCon 2009 presents best practices for getting the best performance out of the SugarCRM open source relationship management application.   After showing where the bottlenecks in the implementation architecture can occur, Satish lays out the Solaris, PhP, and SugarCRM considerations before starting spending some time on MySQL InnoDB and DTrace around page 10.  There is also a pointer to the Sun BluePrint™ Optimizing MySQL Database Application Performance with Solaris Dynamic Tracing. by Jenny Chen of ISV Engineering.

Oracle 10g is now certified  & supported on Sun's Logical Domain (LDoms) feature for both single instances and RAC implementations.   Working on a Sun's CoolThreads Server with Solaris, this technology allows the allocation of a system's resources into logical groupings.    Each group is a discrete system with its own operating system, resources, and network identity operating within a single physical computer system.  This allows an administrators to protect and isolate the groups from each other using a hardware/firmware combination.

With this certification, Oracle will fully support customer installations that deploy the Oracle 10g DBMS in an LDom partition in either a single instance or a RAC configuration.   This will allow multiple Oracle 10g instances to operate independently on a single Sun CoolThreads (CMT) Server in full hardware level isolation.

Another great publication from ISV Engineering in the Sun Blueprints™ series.   This time, Amanda Waite & Marina Fisher explain the "rest of the story" that goes beyond the explicit features & functions of an application.  Their article, called Addressing Systemic Qualities in SAMP Architectures, lays out the details of what developers using a SAMP (Solaris, Apache, MySQL, PhP) need to consider when delivering a complete application that meets both user and administrator expectations.   Here are some of these systemic issues that they detail:

• Performance — how long it takes to execute aset of particular actions. Users expect a snappy response from Web-based applications.  Prolonged waits for page loads can ultimately lead to user discontent, which can cause users to migrate to competitive sites.
  
• Scalability — refers to the ability to accommodate greater workloads and larger user populations. Today it is not uncommon for a site to experience sudden, massive peaks in the number of page accesses due to a mention on popular blogs or social networking sites such as Digg or Slashdot.org.
  
• Availablity — a measure of a user’s ability to access an application or service. Designing for high availability generally means protecting against unplanned system downtime.  In addition, application data must be protected and is also subject to requirements for consistency and integrity.
  
• Extensibility — how quickly new features can be added to an application service and made available to users. The ability to extend the functionality of an application in response to user demand can provide significant competitive advantage.
  
• Interoperability — defines how well an application works with other applications. Web 2.0 applications frequently offer services to other applications through APIs. While most APIs are built on top of interoperability standards, these standards often exist only to define the syntax of inter-application communications.
  
• Security — A Web site must be protected from attacks that intend to compromise the site’s ability to service its users.  At a minimum, applications must be designed to protect against common attacks such as Denial-of-Service (DoS) and Cross-Site Scripting (XSS) attacks.

Stefan Schneider also talks about this blueprint here and suggests developers to post questions about how to implement the suggestions.

ISV Engineering's Luojia (Jenny) Chen weighs in on the subject of MySQL performance with a new document in the  Sun Blueprints™ series titled "Optimizing MySQL™ Database Application Performance with Solaris™ Dynamic Tracing"^*.  She sets out this summary:

... this Sun BluePrints™ article describes how taking advantage of Solaris Dynamic Tracing (DTrace) probes can help simplify MySQL database application tuning. Through examples, this document shows some of the specific aspects of MySQL database server operation that can be observed through DTrace probes. ...

DTrace is indeed the "killer tool" available in Solaris & OpenSolaris that allows deep and unobtrusive observation about what is happening during application execution.   In this document, she explains and gives examples on how the probes can help answer questions such as:

• Which queries impose the heaviest load and can be targeted for optimization?
• Are any queries executing with enough frequency to generate a sizeable cumulative load?
• How much time is a particular query spending on file sorting?
• Do any stored procedures generate excessive load? If so, which SQL statements are executing within that stored procedure
• Which queries contribute the most to replication lag?
• Can the application take better advantage of cache and how?

This is a great reference for Web 2.0 applications built on the solid SAMP stack (Solaris, Apache, MySQL, Perl).  Check out the other great Sun Blueprints™  at the site: 

^* (if you are shy about logging in, you can get it here).   

ISV Engineering's Dr. Christoph Brune has written an excellent summary of Sun's virtualization offerings that help to increase datacenter utilization for SAP deployments.   He points out that surveys have consistently shown that most datacenters only achieve about 10-15% average server utilization.   Then he describes the broad and deep set of offerings from Sun that helps to maximize datacenter efficiency and utilization for SAP installations.   Sun's virtualization platform for SAP Systems includes five key areas.   From the article:

1. Virtual Desktops: Replace traditional PCs with virtual machines that you can manage from the data center.
2. Consolidated SAP Server landscape: Multisocket, multicore, multithreaded servers from Sun can handle the workloads of multiple SAP applications and their tiers, consolidating the SAP environment so you need fewer physical machines.
3. Partitioned servers: Sun presents a comprehensive offering of virtualization technologies, such as Dynamic Domains, Logical Domains, Solaris Containers, and the Sun xVM Server, providing highly isolated and secure environments at the hardware and OS levels.
   
4. Cross-platform virtualization management: Technologies such as Sun xVM Ops Center provision and manage pools of resources that users can share and allocate to many SAP instances.
5. Cost-effective datastorage & management: Products like Solaris ZFS and Sun StorageTek Virtual Storage pool resources to manage storage as a single resource, which decreases the burden of managing large tape libraries, increases system usage and efficiency, and reduces the overall cost of protecting SAP data through improved tape utilization,
   shared tape resources, and reduced complexity.
   

Read the full article in the SAP NetWeaver Magazine or the pdf link here.   Even more about SAP and Sun Microsystems virtualization solutions can be found here.

We've opened two new community sites for HPC and Webscale ISV Developers:

• HPC Community

You've come to the right place if you're interested in building the best HPC applications you can with the Sun platform. HPC applications are used, now more than ever, across a wide range of industries and touch our live in innumerable ways. Join us and let's build something amazing together.

• Web Community

Sun leads in open and scalable technologies and solutions to meet the needs of the web, today and for the future. So you've come to the right place if you're interested in building the best web applications you can with the Sun platform. Web infrastructure is anything your browser touches - software, systems, and storage - and we've got the tools and resources you need to build better solutions, along with examples of how we've worked with partners.

The ISV Engineering team at Sun Microsystems will be contributing to the resources section, forums, and blogs of each of these sites.   Let us know if you crave any specific information.   Your general Sun web login works here.

Although OpenSolaris is initially targeted as a premium developer platform, IHVs (Independent Hardware Vendors) are busily making their solutions available on it.   The Solaris Ready Program certifies and verifies these products on the different OS platforms on offer from Sun.    The Solaris Ready program allows Sun customers and developers to have multiple choices of third party I/O technology products for use with Solaris 10 or OpenSolaris platforms.

The Solaris Ready Program allows developers to display the Solaris Ready logo, enabling customers to readily identify third-party solutions having proven interoperability with the Solaris Operating System on SPARC and x86/x64 based server platforms. The Solaris Ready logo informs customers that these products have met the testing requirements of the Solaris Ready Program for system compatibility, interoperability, ease-of-installation, functionality, and network interoperability as defined and controlled by Sun. Participants in the Solaris Ready Program have also committed to provide long-term support for their products of the Solaris Ready Program leading to greater customer confidence. 

You can find the cards and devices compatible with OpenSolaris 05.11 here and even the recently released OpenSolaris 11.08 already has a bunch of entries here.   Some of the  I/O technologies and product types in the program are:

USB 2.0 --   USB 2.0 Hubs, KVM switches, keyboards , USB to Serial devices, CD ROM & DVD RW devices
PCI, PCI-X, PCI Express  -- Network adapters, USB 2.0 adapters, Graphics adapters  & expansion boxes
SATA and SAS -- hard disk drives and their adapters
Fiber - Fiber optic switchesFiber Channel  - SAN Arrays
iSCSI - IO Controllers and adapters
10 Gig Ethernet - copper and fiber network cards

IHVs who'd like to make their solutions available for OpenSolaris can do so by registering for the Solaris Ready Program.

ISV Engineering's sister group in Sun called "Performance Application Engineering" has decades of experience in optimizing and tuning industry standard, internal and customer-based application workloads.   They have recently open-sourced a powerful new developer, deployer, and performance tool as an Apache incubator project named Olio.   Shanti Subramanyam who was one of its main developers wrote a great post introducing Olio:

I view Olio as a tool to aid developers, deployers as well as performance engineers. For developers, Olio provides 3 different implementations of the exact same application using three different languages and their associated frameworks - PHP, Java EE and Rails. (At this time, the Java EE version is still not in the Olio repository but will soon be). Developers can browse the source code and understand how to design and code a complex web2.0 application. Even experienced PHP developers may gain by looking at the Olio PHP application as we've tried to design the application using object-oriented principles and well-known design patterns - typically not seen much in the PHP world ! In fact, a couple of fairly large companies in China are already using Olio as a training tool for their new hires/interns. If you've been considering rails but have been hesitant, here's your chance to check out a full-blown app and see what it will take to develop yours.

Olio has done all of the hard work using best practices and patterns to integrate popular open source pieces and lets you mix and match different combinations.  Nick Kloski of our Technical Marketing group also has a good explanation here. High Scalability weighs in here.

ISV Engineering sees Olio as being a great platform for POCs (proof of concept) when working with startup and Web 2.0 companies developing web applications.

In the early days of workstations and servers, the size and weight (aka thump factor) of a hardware vendor's application catalog was a frequently used and effective marketing tool.   More applications = More Solutions = More Sales.   These days, vendors (including Sun) continue to tout the size of their application catalogs.   Matt Asay from Alfresco wrote an article in CNET recently commenting on the virtual-thump-factor wars between Novell and Red Hat.  

Solaris has had a long history of working hard to make sure that its application catalog is both broad and deep.   The listing available here shows a total application footprint of 11,225 applications available for the Solaris platform and of those 7850 are for the x86/x64 platform.   This is significantly more "thump" than the current Red Hat catalog. Red Hat argues here that there is an 80/20 rule in which 80% of customers use the same 20% of applications.   This is true only to a point - our customers know that there exists a long tail of applications that may not be big names, but are nonetheless critical to operations.   Although most of the applications are common to most customers, each customer usually has a couple of special applications that have crept their way into use and are indispensible.  In the end, *thump* still wins.

Sun has also constructed a software catalog community which is gaining publishers by the day.

Wikimedia's Wikipedia.org is one of the most visited websites approaching 250 million unique visitors per month.  New content is pushed in by the over 100,000 volunteers who edit the articles.  This announcement "Wikimedia Selects Sun Microsystems to Enhance Multimedia Experience..." describes how users can now upload large video files using the servers and open storage environment provided by Sun and other open source organizations.

The solution is built up in an open source paradise:  MySQL, Apache, Sun's Open Storage infrastructure all on Sun's X4500 and X4150 Servers connected to Sun StorageTek arrays.

An article by Rafe Needleman in Webware further describes that the the Ogg Theora open source video compression technology will be used in a Firefox browser (more open source!).  It is interesting that he says the current Wikipedia storage footprint is currently only 5TB growing to 15TB by 2009E.

Alka Gupta on our team has a great blog about Wikipedia's use of the rest of the LAMP stack to achieve the 7 billion page views that the 250 million visitors generate each month.

There are a couple of easy ways to test your Solaris application with MySQL ("the world's most popular opensource database").   Of course, you can always setup a Solaris or OpenSolaris environment and download it (70-80MB) and then install/configure locally.   However, if you don't want to go through the hassle of setting up the System, OS, & database yourself, Sun has made it easy to get your hands on pre-configured bits in a couple of other efficient ways.

Use the EZQual Virtual Lab.  Sun Partner's can get access to secure, no-cost, test environments (within Solaris containers/zones).  Here's the information directly from the program:

We've set up a secure, remote, on-line test facility designed to make it easier for ISVs to develop, test and qualify applications on the Solaris 10 or OpenSolaris operating system with the MySQL database for FREE!  There are no servers to install and no OS installations - really doesn't get much easier than this.  The Lab features pre-installed SPARC or x86 processor-based Sun servers with Sun Studio Development Tools, Java SE platform, Sun's Cool Stack, the MySQL database, Solaris 10 and OpenSolaris OS, and more and are accessed conveniently with Sun's Secure Global Desktop Software.

Use Amazon EC2 with OpenSolaris.  We also have a pre-configured MySQL and SysBench AMI that has embedded DTrace probes you can use through the Amazon EC2 service:

This 32-bit AMI is based on the OpenSolaris 2008.05 AMI with updated software revisions to build 91. It contains MySQL 5.1.25 binaries, and SysBench. ... There are also sample Dtrace scripts which utilize the probes in MySQL to collect information about SQL calls.

Stay tuned for vdi/vmdk images of OpenSolaris that will have pre-configured MySQL available.   Also, keep an eye on Jennifer Glore's regular posts for ISV partners using MySQL.

The rise of the "Open Source Software Stack" reached another milestone today.  

Sun released a SPECjAppServer2004 performance result using top-to-bottom Open Source Software stack components:  OpenSolaris/MySQL/Glassfish.   At 1197.10 JOPS, the performance is highly respectable - not setting any land speed records, but it is in the ball park with other results using the same class of hardware.   The difference (of course) is in the price/performance.

The other contenders were two [1][2] results submitted by Oracle using a stack of (surprise!) Oracle DBMS (proprietary)/Oracle Appserver (proprietary)/Oracle Linux & Red Hat (open source).  On the face of it, the Oracle results (using Dell & HP gear) currently can achieve performance above a result with all Open Source stack components.  The difference is $ & €.   After all, very rarely does a customer choose a system based only on absolute performance advantage.  Price/Performance remains consistently the #1 or #2 reason that customers choose one solution over another.  In this case, even a conservative estimate of the two Oracle results place their $/JOPS at nearly 10x the price of the Sun result.  It is this "10 times"  advantage that is usually quoted by marketeers as being the tipping point that is required to move from an old way of doing something to a new cheaper/better way.   Tom Daly of ISV Engineering at Sun has all of the details here.

It will only be a matter of time before completely Open Source stack components are competing head-to-head with proprietary software solutions even on the absolute performance front.   I remember back in the 90's when Linux was struggling to get a foot hold.  In those days, many customers did say that "Linux performance just isn't there yet" (compared to Unix solutions).   The keyword there is "yet".   Everyone had the full expectation that Linux would one day compete with everyone else in the same raw/absolute performance space.   So it does today, and so will complete Open Source stacks in the near future.

BMSeer has even more comprehensive information on the benchmark here.

SPEC required disclosure : - SPEC, SPECjAppServer reg tm of Standard Performance Evaluation Corporation. Results from www.spec.org as of 5th Nov 2008 All comparisons are based on the SPEC SPECjAppServer2004JOPS@Standard metric from www.spec.org or on pricing made using the bill of materials included in each SPECjAppServer2004 result

Find Free & Open Source Software that "runs best on Sun" here.

Sun has invested and positioned itself in the market as a huge provider and supporter of Free and Open Source applications & software platform stacks.   We are also making sure that our stacks and other open source applications "run best" on the hardware and software that we sell (See Jignesh Shah's excellent summary of what we mean by "run best").

We've created a new landing page where you can see the  news, blueprints, and featured open source community partners who are making the vision of "run's best on Sun" a reality.   The web site is also connected to the newly released beta of the Sun Software Library.  There is a Sun Software Library view of Open Source Applications and Tools as well as a specialized view for Enterprise Open Source Applications.