News Update Center and Multi-platform pkg(5)

The Update Center team is pleased to announce the availability of release 2.3 of the Update Center Toolkit. The focus for this release has been to improve the overall quality of the toolkit by reducing the bug backlog. This release also includes the following new features:

• Support for the AIX operating system (command line only) and the Windows 7 operating system (user folder only)
  
• A new version of pkg(5) that is aligned with OpenSolaris build 122. This version includes a new transport library based on libcURL.
  
• Support for package icons in updatetool (see pkg.icon.24px)
  
• Ability for updatetool to show only a few recent versions rather than all or one.
• Support for an "excludedirs" capability in the makepkgs tool
• Support for specifying a default ownership for files when using makepkgs
• The copypkgs tool can copy packages to/from a pkg.depotd server
• The HTML templates for pkg.depotd support showing only the most recent version of packages
• Improved on-line help for the software update interface
• Improved startup time for the software update interface when started from the notifier
• Support for promoting new software applications via the software update interface
  

Downloads and release notes for the 2.3 release are now available as are updated screenshots of the graphical user interfaces.

This release will make its debut in the GlassFish v3 release that is scheduled for later this year.

GlassFish v3 Prelude is the highest volume adopter of Update Center and the pkg(5) system on multiple OS platforms, but has been running with an older version of the Update Center Toolkit (2.0) since Prelude shipped a year ago.  Last week the GlassFish team pushed Update Center Toolkit 2.2 Update 2 to the Prelude repositories and we are starting to see significant uptake. With UC 2.2 Update 2 customers will see a more streamlined update experience as well as a number of other improvements.

In addition to updating the UC toolkit packages the Prelude team upgraded their repository runtimes resulting in an improved web UI for their repositories.

The Update Center 2 development team is pleased to announce the availability of build 38 (download) of the Update Center Toolkit. This build resolves 25 issues beyond the previous build. This build is expected to be the final build for the 2.3 release.  When quality testing is completed and the go is given for the release, we'll rename this (or a respin if necessary) the 2.3 final release.

As the Update Center Toolkit team wraps up the development of the 2.3 release, planning for the 2.4 release has started. There is now a 2.4 planning page on our development wiki with a draft schedule and ideas about major themes and features. If as a user you would like to provide input to this planning process, that input would be most welcome. Just send email to dev@updatecenter.dev.java.net.

One of the big changes that will be in 2.4 is the version of pkg(5) that will be shipped with OpenSolaris 2010.02, which includes changes in the way that the meta data is stored for an image.  These changes are expected to have a big improvement on performance along with better update planning through the SAT solver feature.

If you have filed issues against the toolkit and would like to express the need to have the issue resolved, please update the issue with that information on Issue Tracker. We will take that information into consideration when planning the next release.

We don't offer a standalone distribution (download bundles + dedicated repos) for GlassFish v2.1, but we do produce and maintain v2.1 packages in support of GlassFish Web Space Server deployments (see my earlier post for background).   Although Web Space Server is currently the only distribution that embeds these packages, you can easily experiment with the packages by installing them from our layered collection development repository.  The beauty of the packaging system and our repositories is that it's quite simple to try out these and other packages:

1. Download a copy of the Update Center Toolkit. Alternatively, if you're already on OpenSolaris or you already have the toolkit available with the pkg(1) CLI, move on to the next step.
2. pkg image-create --user -p layered.sun.com=http://pkg.sun.com/layered/collection/dev/   ~/gfv2.1
3. cd ~/gfv2.1
4. pkg install glassfish2
5. cd glassfish2
6. ../ant/bin/ant -f setup.xml 
7. bin/asadmin start-domain
8. Access http://localhost:4848 and log in with admin / adminadmin

Over the past several months our Update Center project has been rounding out our overall pkg(5)-based content delivery solution by providing usage reports to adopting project teams: GlassFish Application Server v3, Web Space Server, Web Stack and soon, Message Queue.

You may be familiar with the term web analytics, defined as the "analysis and reporting of internet data for purposes of understanding and optimizing web usage" by Wikipedia. In the context of delivering content via network package repositories, the Update Center team has deployed the beginnings of a reporting solution that helps product teams improve their offerings by providing greater insight into how add-ons and updates are being adopted by end users.

Over the past several months, our toolkit project has:

• Defined a database schema and deployed a MySQL database to manage the usage data
  
• Developed and deployed scripts to tranform raw repository access log data and to load it into the database
• Developed an overall dashboard and an initial series of product-specific reports
• Started tweaking reports based on feedback from users
  

Project teams can currently obtain the following information via the reports:

• Active unique installations (i.e. "active users")
  
• Add-on (new) packages installed
• Updated packages
• Operating system usage
• Client tool usage (Update Tool GUI, Software Update GUI, pkg(1) CLI, pkg(5) Java API)
  
• Add-on activity by registered installations
  

Additionally, we report on the versions of the toolkit that are in use across all installations.  This information is a real boon to our toolkit development project in that it lets us make informed decisions about how much we invest in older versions.  It also helps us monitor the roll out of new releases of the toolkit.

Over the next several weeks we'll publish new blog entries that explain the reports and how projects are using them to improve their products and ultimately the experience of their users.

The Update Center 2 development team is pleased to announce the availability of build 37 (download) of the Update Center Toolkit. This build resolves 18 defects and contains 1 enhancements beyond the previous build.  The pkg(5) drop for this build is based on build 122 of OpenSolaris with some additional fixes. 

The next build, 38, is expected to be the final build for the 2.3 release.

The Update Center 2 development team is pleased to announce the availability of build 36 (download) of the Update Center Toolkit. This build resolves 21 defects and contains 13 enhancements beyond the previous build.  The pkg(5) drop for this build is based on build 121 of OpenSolaris with some additional fixes. 

This was the last open bug fix build for the 2.3 release.  The remaining builds are for stoppers for the release. 

Yesterday, August 11, 2009, the Image Packaging System, pkg(5), was compiled and run for the first time on IBM's AIX® 6.1 operating system. All of the API unit tests ran successfully, and 69 of the CLI tests passed. Since I don't have root access on the AIX system being used, the other CLI tests were not expected to pass. AIX support for the command line interfaces is one of the features in the Update Center 2.3 toolkit.

The hardest part about porting pkg(5) to AIX has been to assemble the build environment that is needed to compile the software, especially without root access.  Although there are some RPMs available with some of the tools for AIX, these require root access to install.  That's one of the nice things about pkg(5) packaging is that it supports user images that do not require root access. Compilation of pkg(5) requires Python, Mercurial, a C compiler, setuputils, pyXML, simplejson, patch, cURL,  OpenSSL, and tidy. This list and the instructions for performing the build have been recorded on a project wiki page.

Once the build environment was assembled, modifications to only four files (nine lines) were needed to accomplish the initial port. These changes consisted of modifying the build and test scripts and the utility for getting a canonical operating system name to recognize 'aix' as a valid platform. Work still remains to complete more thorough testing and debugging, packaging of pkg(5) into the toolkit for AIX, and making sure other parts of the toolkit such as the Java API are supported on AIX, but this is a big first step.

With the addition of AIX in the 2.3 release, this expands the multi-platform support for the Image Package System to 5 major platforms (Solaris, Linux, Mac/OS, Windows, and AIX).

® AIX is a registered trademark of IBM. 

The release of Web Stack 1.5 was recently announced.  One of the unique features of the 1.5 release is support for pkg(5) and the Update Center Toolkit.      

As Jeff Trawick demonstrates in his blog installing Web Stack components such as Apache is greatly simplified using the tools provided by the Update Center Toolkit.   The Update Center will also provide desktop notification to users when updates to the installed Web Stack components become available as well as provide a straightforward method to update those components via the Software Update tool.

If you are a new or existing Web Stack user now is a great time to try out the new pkg(5) based version of the product.

The Open MQ team has been busy preparing the initial promoted builds of their upcoming new minor feature release, Open MQ 4.4.  It's impressive that without any real prodding from our Update Center Toolkit team, the MQ team has pretty much of their own accord begun adopting pkg(5) and the Update Center Toolkit as the main delivery format for this next release.  The MQ team has produced a series of installer wrapped download bundles containing pre-installed pkg(5) images for their supported platforms and they are in the process of delivering a companion set of raw zip downloads using virtually the same underlying pre-installed pkg(5) images.  You can obtain their preliminary promoted builds of the installer-wrapped bundles from their MQ 4.4 release site.

Shape of the pkg(5)-based Open MQ Distribution

The Open MQ 4.4 builds currently consists of at least 9 pkg(5) packages containing content ranging from portable Java components of the core product to platform-specific components such as the optional C language client API. Platforms addressed by these preliminary builds include: Windows, Linux, Mac OS X, Solaris and OpenSolaris.  Once AIX is supported by the Update Center project's port of the pkg(5) system in v2.3 of the toolkit, the Open MQ AIX bundle will likely migrate to using pkg(5) packages of their AIX install bundles.

Benefits of Sustainable, Modular Binary Packages

In addition to forming the basis of the MQ 4.4 distribution, these same pkg(5) packages are integrated into builds of the GlassFish v3 distribution. Thereby demonstrating the value of reusing the same modular binary packages across multiple product distributions.  Historically, the lack of formal packaging of raw zip and tar.gz bundles has made it difficult for development and sustaining teams to deliver and share the same updated binaries.  With the adoption of the pkg(5) system and the Update Center Toolkit, development and sustaining teams only need to deliver one set of updated packages that can be shared across multiple product distributions.