Patch Corner

Training for Solaris 10 Patching

Sun Learning Services are in the process of creating a number of patch related training lessons.

They've launched a blog, which contains the initial introductory videos.

Future lessons will be much more detailed, concentrating for example on Live Upgrade.   These lessons will be available on the Sun Open Learning Center (SOLC) website: https://learning.sun.com/solc/smartstart.

Posted at 10:06PM Jun 25, 2009 by Gerry Haskins in Sun  |  Comments[0]

Heads up on Kernel patch installation issues with jumpstart or ZFS Root

I'd like to give you a heads-up on a couple of Kernel patch installation issues which we're currently investigating.

mv: cannot rename /var/run/.patchSafeMode/root/lib/libc.so.1.20102 to /lib/libc.so.1: Device busy
ERROR: Move of /var/run/.patchSafeMode/root/lib/libc.so.1.20102 to dstActual failed
usage: puttext [-r rmarg] [-l lmarg] string
pkgadd: ERROR: class action script did not complete successfully

Installation of <SUNWcslr> failed.

There are several workarounds to avoid the issue:

1. Apply KU 139556-08 to an inactive boot environment, either using LiveUpgrade or via boot net/cdrom
2. Manually umount /lib/libc.so.1 BEFORE applying KU 139556-08
3. One more way to apply to inactive boot environment is to boot into failsafe mode, mount root pool on /a and apply KU to /a

2. There are reproducible issues using jumpstart finish scripts and other scenarios to install Kernel patch 137137-09 followed by Kernel patch 139555-08.   Here's the gist of the issue which I've pulled from an engineering email thread on the subject:

Issue 1: I have a customer whose system is not booting after applying the patch cluster with Live Upgrade (LU).

Solution 1: If using 'luupgrade -t', then you must ensure that latest version of LU patch is installed first, currently 121430-36 is currently the latest revision on SPARC, 121431-37 on x86. Once thse patches are installed, LU will automatically handle the build of the bootarchive when 'luactivate' is called, thus avoiding the problem.

Issue 2: There are other ways to get oneself into situations where a boot-archive is out of sync: e.g. If using jumsptart finish scripts to apply patches that include 137137-09.  Basically any operation that invovles patching to an ABE outside of 'luupgrade' will invovle a manual build of boot-archive.

Solution 2: One must manually rebuild the boot-archive on the /a partiton after applying the patches.  Otherwise once the system boots, the boot-archive will be out of sync.

Here's some more detail on the jumpstart finish script version of this: 

We've seen the same panic a few times when the latest patch cluster is applied via a finish script to a boot environment prior to  s10u6 via a jumpstart installation. It appears that the boot archive is out of sync with the kernel on the system. The boot archive was created from the 137137-09 patch and not updated after the 139555-08 kernel was applied, therefore the mismatch between the kernel and the boot archive.

In these instances updating the boot archive allows the system to boot successfully. Boot failsafe (ok boot -F failsafe) will detect an out of sync boot archive.  Execute the automated update then reboot. This will now boot from the later kernel (139555-08) which successfully installed from the finish script.

I reproduced the problem in a jumpstart installation environment applying the latest 10_Recommended patch cluster from a finish script. The initial installation was s10u5 which is deployed from a miniroot that has no knowledge of a boot archive (my theory anyway).  This is similar to a live upgrade environment if the boot environment doing the patching is also boot archive unaware (meaning the kernel is pre 137137-09).

In the jumpstart scenario the immediate problem was solved by updating the boot archive by booting failsafe as previously described.  The Solution was to update the boot archive from the finish script after the patch cluster installation completed.  BTW, all patches in the patch cluster installed successfully per the /var/sadm/system/logs.finish.log.

In a standard jumpstart the boot device (install target) is mounted to /a, therefore adding the following entry to the finish script solved the problem:

/a/boot/solaris/bin/create_ramdisk -R /a

Depending on the finish script configuration, and variables the following would also work:

$ROOTDIR/boot/solaris/bin/create_ramdisk -R $ROOTDIR

Issue 3: This above issues are sometimes misdiagnozed as CR 6850202: "bootadm fails to build bootarchive in certain configurations leading to unbootable system".

But CR 6850202 will only be encountered in very specific circumstances, all of which must occur in order to hit this specific bug, namely:

1. Install u6 SUNWCreq - there's  no mkisofs so we build ufs boot archive

2. Limit /tmp to 512M - thus forcing the ufs build to happen in /var/run

3. Have a separate /var - bootadm.c only lofs nosub mounts "/" when creating the alt root for DAP patching build of boot archive

4. Install 139555-08

You must have all 4 of above in order to hit this, i.e. step 4 must be installing a DAP patch such as a Kernel patch associated with a Solaris 10 Update such as 139555-08. 

Solution 3: Removing the 512MB limit (or whatever limit has been imposed) to /tmp in /etc/vfstab and/or adding SUNWmkcd (and probably SUNWmkcdS) so that mkisofs is available on the system is sufficient to avoid the code path that fails this way.

Booting failsafe and recreating the boot archive will successfully recreate the boot archive.

Best Wishes,

Gerry.

Posted at 07:48PM Jun 25, 2009 by Gerry Haskins in Sun  |  Comments[4]

Zones Parallel Patching versus Update On Attach: When to use which one ?

The Zones Parallel Patching enhancement for the Solaris 10 patch utilities was released this week giving customers a choice of how to improve zones patching performance.

In the Zones "Update On Attach" section of a previous blog posting, I mentioned that the Zones "Update On Attach" feature could also be used to improve Zones patching perfomance.

Zones Parallel Patching is a true patching solution utilizing the 'patchadd' utility.  

Whereas Zones "Update On Attach" uses zones functionality similar to that used during zones creation to provide a pseudo-patching solution that does not utilize 'patchadd'. 

So which one to choose ?

Let's look at the two options in more detail:

Zones Parallel Patching

Zones Parallel Patching is an enhancement to the standard Solaris 10 patch utilities and is delivered in the patch utilities patch, 119254-66 (SPARC) and 119255-66 (x86).

Simply install this patch, set the maximum number of non-global zones to be patched in parallel in the config file /etc/patch/pdo.conf, and away you go.

It works for all Solaris 10 systems. 

It also works well in conjunction with higher level patch automation tools such as xVM Ops Center. 

It can dramatically improve zones patching performance by patching non-global zones in parallel.  The global zone is still patched first.

While the performance gain is dependent on a number of factors, including the number of non-global zones, the number of on-line CPUs, the speed of the system, the I/O configuration of the system, etc., a performance gain of ca. 300% can typically be expected for patching the non-global zones - e.g. On a T2000 with 5 sparse root non-global zones.

See my previous Zones Parallel Patching blog entry for further information.

Since it's a pure enhancement to 'patchadd', it's normal 'patchadd' functionality.  You can subsequently remove patches using 'patchrm', etc.  Nothing has changed except that it's now much faster to patch non global Zones with Zones Parallel Patching invoked.

Zones "Update On Attach"

The primary purpose of Zones "Update on Attach" is Zones migration from one server to another.  

For example, a database instance in a non-global zone hosted on a server has grown to the extent that the Sys Admin wants to transfer it to a better spec'd server which can better handle the workload.   The Sys Admin can detach it from the old server (e.g. a Sun4u) and reattach it to the new server (e.g. a Sun4v) using Zones "Update On Attach".   This will bring the OS Software level on the non-global zone up to the same level as the new server's global zone.

Zones "Update On Attach" can certainly be used for patching but there are limitations you need to be aware of as outlined below.

For example, detach the non-global zones from a system, apply a bunch of patches to the global zone, reattach the non-global zones using "Update On Attach" and viola, the non-global zones will be brought up to the same software level as the global zone (for OS type packages), effectively patching the non-global zones without using 'patchadd' at all.   This is typically even faster than using Zones Parallel Patching.  But there are limitations to this approach which users must be aware of (see below).

My senior engineer, Enda O'Connor, has just published an interesting article on The Zones Update on Attach Feature and Patching in the Solaris 10 OS. 

Zones "Update On Attach" limitations as a patching aid

Zones "Update On Attach" only works for packages which are SUNW_PKG_ALLZONES=true - i.e. typically OS level packages, and not application packages.

So when to use Zones Parallel Patching in 'patchadd' and when to use Zones "Update On Attach" ?

Here's what my senior engineer, Enda O'Connor, says:

"The Zones Update on Attach Feature and Patching in the Solaris 10 OS document may help customers understand how the technology works, applying a cluster via patching and via zones Update On Attach is not quite the same really.

It really depends on the patches being applied, i.e. applying a firefox patch via Update On Attach would not work if you wanted it to apply to the global zone and all non-global zones as well.

One has to understand how Update On Attach works and then apply that to the list of patches to see if it gets them to a desirable state.

There is no black or white answer here.

I'd recommend Zones Parallel Patching using 'patchadd' as it has a known outcome all the time, whereas Update On Attach makes it's own internal determination based on a number of things, that can vary from system to system ( e.g. inherited directories ).

But if time to patch is critical then if the customer does proper testing to validate things, and are happy with the results, then by all means use Update On Attach.

But using Update On Attach without:

1. Understanding how it determines what packages to update

2. Not inspecting the patches being applied.

...will most likely lead to grief at some point."

And my other senior engineer, Ed Clark, says:

"In terms of giving guidance on which technology to use, there are a number of considerations -- two of these considerations are:

1. Using Update On Attach to update sparse zones can require significantly more disk storage space than would be needed by applying patches with 'patchadd' (3-4 times as much space would not be uncommon i think), due to Update On Attach copying fully populated global zone 'undo' files into the non-global zones, as opposed to having patchadd build sparsely populated 'undo' files in the non-global zones.

2. If a customer is really concerned about the ability to back out patches reliably, then 'patchadd' is a lower risk option than Update On Attach -- 'patchrm' of a patch from a non-global zone that has a copy of the global zones 'undo' pkg data (as is the case after Update On Attach) may potentially have unexpected side effects." [although we have yet to see any actual cases of negative results from this.]

Conclusion

In general, we recommend using the Zones Parallel Patching enhancement in the patch utilities rather than the Zones "Update On Attach" feature as Zones Parallel Patching is standard patching functionality, only faster, whereas Zones "Update On Attach" is really designed for migrating zones from one server as another and was not primarily designed to speed up patching.  

Because Zones "Update On Attach" uses Zones functionality similar to the zone creation functionality, rather than 'patchadd' functionality, limitations exist on what will be patched (typically the OS but not applications) and there's the potential for anomalies around things like the "undo" files which would be used by 'patchrm' if patches applied using Zones "Update On Attach" were subsequently removed from the non-global zones using 'patchrm' (although we have yet to see any actual cases of serious issues resulting from this).

So in patching situations where time is absolutely critical, Zones "Update On Attach" may provide a good option, as long as it's well tested in the customer environment prior to deployment on production systems.

Remember too, Live Upgrade is also your friend in such situations, enabling you to patch an inactive boot environment while the system is still in production.   So a combination of Live Upgrade and Zones Parallel Patching would be ideal.

I hope you find this helpful!

Best Wishes,

Gerry.

Posted at 06:06PM Jun 19, 2009 by Gerry Haskins in Sun  |  Comments[0]

Solaris 10 5/09 (Update 7) patch bundle now available!

The Solaris 10 5/09 (Update 7) patch bundle is now available for download from the SunSolve Patch Cluster & Patch Bundle Download Page.  Click on the "Solaris Update Patch Bundles" link.

As with previous patch bundles, it contains the patches which are included in the corresponding Solaris Update, in this case Solaris 10 5/09 (Update 7).

This is useful for Sys Admins who wish to bring all their systems up to the same patch level as the Solaris Update without wanting to upgrade to the release - for example, due to change control policy restrictions in their organizations.

See previous blog entries for previous Solaris Update patch bundles for further information.

Posted at 02:19PM Jun 18, 2009 by Gerry Haskins in Sun  |  Comments[0]

Zones Parallel Patching feature now available!

The Zones Parallel Patching feature is now available in the latest Solaris 10 patch utilities patch, 119254-66 (SPARC) and 119255-66 (x86).

This is available for use on all Solaris 10 systems. 

Simply install this patch, set the maximum number of non-global zones to be patched in parallel in the config file /etc/patch/pdo.conf, and away you go.

Prior to this feature, each non-global zone was patched sequentially, leading to unnecessarily long patching times for zones systems.  (Sequential patching remains the default behavior unless the config file is edited to enable Zones Parallel Patching.)

With this feature invoked, the global zone continues to be patched first, but then the non-global zones can be patched in parallel, leading to significant performance gains in patching operations on Zones systems.

While the performance gain is dependent on a number of factors, including the number of non-global zones, the number of on-line CPUs, the speed of the system, the I/O configuration of the system, etc., a performance gain of ca. 300% can typically be expected for patching the non-global zones - e.g. On a T2000 with 5 sparse root non-global zones.

Here's the relevant note from the patch README file:

NOTE 10: 119255-66 is the first revision of the patch utilities to deliver "zones parallel patching".
          This new functionality allows multiple non-global zones to be patched in parallel by patchadd.
          Prior to revision 66, patchadd would patch all applicable non-global zones sequentially,
          that is one after another. With zones parallel patching, a sysadmin can now set the number
          of zones to patch in parallel in a new configuration file for patchadd called /etc/patch/pdo.conf.

         The two factors that affect the number of non-global zones that can be patched in parallel are
         1. Number of on-line CPUs
         2. The value of num_proc in /etc/patch/pdo.conf

          If the value of num_proc is less than or equal to 1.5 times the number of on line CPUs,
          then patchadd limits the maximum number of non-global zones that will be patched in
          parallel to num_proc. If the value of num_proc is greater than 1.5 times the number of on line CPUs,
          then patchadd limits the maximum number of non-global zones that will be patched in parallel
          to 1.5 times the number of on line CPUs. Note that patchadd will patch all applicable non-global
          zones on a system, the above description outlines only how patchaadd determines the
          maximum number of job slots to be used during parallel patching of non-global zones.

          An example of this in operation would be where:
          num_proc=8
          and number of on line CPU's is 4

          In this case the maximum setting for num_proc would be 6, that is the maximum number
          of zones that could be patched in parallel is 6.  If there are more than this number of non-global zones on the
          system, the first 6 will be patched in parallel, then the remaining non-global zones will be patched
          as processes finish patching the first 6 non-global zones.   Only one patch process will be used for each
          non-global zone, so if there are less than 6 non-global zones on the system, then only the number of processes
          equal to the number of non-global zones will be initiated.

          Please see comments in /etc/patch/pdo.conf for more details on setting num_proc.

I would like to thank Ed Clark and Enda O'Connor from my own team for all their work in developing and testing Zones Parallel Patching.

I would also like to thank Jon Bowman, Arindam Sarkar, and the rest of the RPE (Sustaining) Install team for all their work in getting this feature integrated into the patch utilities and delivered to production.

I would also like to thank our selected key customers who kindly Beta tested the feature for us.

I believe this feature is an important milestone in improving our customers' patching experience in a Zones environment as it addresses a long standing customer complaint on Zones patching performance.

Enjoy!

Posted at 07:11PM Jun 17, 2009 by Gerry Haskins in Sun  |  Comments[11]

New PatchFinder tool now available

The initial version of the new PatchFinder tool is now available on http://sunsolve.sun.com/patchfinder/. 

It's linked off the main SunSolve Patch page, http://sunsolve.sun.com/show.do?target=patchpage.  Look for the following link immediately under the old PatchFinder search box:

Preview The New PatchFinder

Why a new PatchFinder tool ?

The old PatchFinder tool was a pet peeve of mine.  You needed to know at least the 6 digit base PatchID of the patch you were trying to find in order to find it.   Rather self defeating IMHO.

The new PatchFinder tool directly leverages Sun's internal Patch Metadata Web Services to provide a much richer search experience.

Features of the new PatchFinder tool

You can still search by PatchID if you want.  This will override all other search options.

But you can also search for all Recommended or Security patches, and restrict that search, for example, to Solaris 10 SPARC.

By the way, "Recommended" means it's part of the Solaris Recommended Patch Cluster, which contains the latest revision of all Solaris OS patches which fix Security, Data Corruption, or System Availability issues.  See the cluster inclusion criteria definitions by clicking the appropriate heading on the Patch Clusters & Patch Bundles download page, http://sunsolve.sun.com/show.do?target=patch-access.

"Security" includes all patches which address Security issues, including Solaris OS patches and application and middleware patches for other products.

If you click the "OS Patches Only" box, the search results can be restricted to patches for the Solaris OS only, which will exclude application and middleware patches which are not bundled as part of the Solaris OS.  

Caveat: Note, the interpretation of what is a Solaris OS patch used by the tool is currently about 97% accurate.  This is due to difficulties interpreting patches for applications and middleware which may be bundled with the Solaris OS.  (The old PatchFinder tool and SunSolve patch reports have the same issue.) Hence you can see anomalies between search results returned for "Recommended" and "Recommended & OS Patches Only" which should be identical.  A subsequent version will have a 100% accurate interpretation of what is a Solaris OS patch.

Advanced Search Capabilities

Click on "Show Advanced Search" for more options.

This gives you options such as searching by CR (Change Request, a.k.a. BugID) number, so if you suspect you've hit a particular bug, you can check whether a patch for that CR is available yet.

Or you can search for patches with particular words in the patch synopsis or keywords fields - e.g. ldap, "patch util", "package util", "pkg util", etc.  These options have limited value as it's difficult to guess the values.

The "Released Before" option is handy if your company has a policy of waiting for patches to "age" a specified number of days after release before you consider applying them.

The "Released After" option is useful to restrict the search to patches released since the last time you checked for patches.

The "README Modified After" option is subtly different to the "Released After" field and is a superset of the "Released After" results in that is also shows patches whose README or patchinfo metadata files have been updated since the patch was initially released - for example, Special Install Instructions may have been added to the README to specify workarounds for issues found post-release which do not warrant the patch being withdrawn from SunSolve (i.e. the patch still does more good than harm for the majority of customers).

You can filter the search further to see only those patches whose README file was modified since you last downloaded patches by using the following search filter combination: For example, if you downloaded patches 30 days ago, you can see which patches which were release 30 or more days ago have had their READMEs modified since then by using the combination: "Released Before" == 30 && "README Modified After" == 30

In all of these time related fields, you can specify actual dates instead of a specified number of days.

If you don't have a support contract, click on the "Public Patches Only" box to restrict the search to patches which are available for download without a support contract.

The "Patch Property" field enables you to search for things like "Interactive" patches which require manual intervention during installation, "NonStandard" which means they aren't applied using the standard 'patchadd' utility (e.g. firmware patches), or patches which require downtime (Single User Mode, Reboot*) if applied to the live boot environment.  (Remember, Live Upgrade can be used to minimize the downtime and risk associated with applying patches by applying the patches to an inactive boot environment, thereby avoiding such downtime requirements during or immediately after patch installation.  You can reboot to set the inactive boot environment live at a time that suits you.)

By default, only patches which are currently available for download (i.e. patches which haven't been withdrawn due to issues) are returned in the search results.    You can select "Withdrawn" patches instead to get a list of patches which have been withdrawn from SunSolve due to serious issues.   This is useful to ensure you don't have any withdrawn patches installed on your systems.  I recommend you also select "Show Obsoletes" along with "Withdrawn" so withdrawn patches which have been superseded by replacement good patches aren't masked.  (Note, a Sun Alert is issued whenever a patch is withdrawn, so if you keep abreast of Sun Alert notifications as is advisable, this step is simply a check and balance.)

Fields such as "OS Release", "State", etc., allow multiple options to be selected concurrently from the drop down menu.

Patch Metrics Gathering 

The new PatchFinder tool is also useful for helping you to calculate patch metrics - e.g. the number of Solaris 10 SPARC OS patches released in the last year.

Or, if you feel so inclined, you can use the new PatchFinder tool to calculate the percentage of Solaris OS patches which are publicly available.   Select an OS Version, select "OS Patches only", and search with and without the "Public Patches Only" box selected to get the number of publicly available patches and the total number of available patches respectively.   To save you the trouble, the percentage of Solaris OS patches which are publicly available without a support contract today (May 27th 2009) are 25% for Solaris 10 OS patches, 28% for Solaris 9 OS patches, and 31% for Solaris 8 OS patches.

Display and Bookmarking Options

You can also select the number of patches to display in each page of search results returned (default 20), hide the search form so that only the results are displayed (the option is in the top right hand corner of the tool), and order the results by PatchID, Released date, or Synopsis, in either ascending or descending order (by clicking on the appropriate column heading of the results returned).

You can click on a PatchID in the search results returned to display the Patch README.

You can also bookmark the search results returned for future reference.  This is handy if you wish to run the same query regularly. 

Help! 

There's a "Help" summary in the top right hand corner and each search field has it's own help summary marked "?".

What's next ? 

I hope you find this initial version of the new PatchFinder tool useful.

This is a start, not the finished article.   In future versions we plan to provide options to resolve patch dependencies and patch installation order, enable patch download, etc.  

Feedback - what else would you like to see ?

Feel free to provide feedback on features which you'd like to see to the software-update-finder-feedback@sun.com alias or directly to me, Gerry.Haskins@sun.com .  

Our goal is to improve your patching experience.

Posted at 03:11PM May 27, 2009 by Gerry Haskins in Sun  |  Comments[4]

How to analyze patch failures and other useful info

My colleague, Enda O'Connor, has published 3 more patching articles on Big Admin which I hope you will find useful:

• Analyzing a patchadd or patchrm Failure in the Solaris OS

  How to gather and analyze the necessary information to determine why a 'patchadd' or 'patchrm' session failed.
  

• How to Split a Root Mirrored With Solaris Volume Manager Prior to Updating Software

  How to split a Solaris Volume Manager mirror prior to updating the software through upgrades or patches.

• How to Remove a Solaris Patch While Booted From a Network or CD-ROM

  How to run 'patchrm' from a network or CD-ROM against a non-live system that is mounted from such media, and how to mount file systems under the mount point after the system is booted from the media.

Posted at 03:55PM May 08, 2009 by Gerry Haskins in Sun  |  Comments[0]

Patch News Update

Solaris 10 5/09 (Update 7) and subsequent Kernel PatchIDs 

The Kernel patch included in Solaris 10 5/09 (Update 7) has now been released to SunSolve.  The PatchIDs are 139555-08 (SPARC) and 139556-08 (x86).  The rest of the patches included in Solaris 10 5/09 are either released or in the process of being released over the next week or so. 

I've updated the Solaris Kernel PatchID sequence listed in http://blogs.sun.com/patch/date/20080416 to reflect this, including the PatchIDs of the post Solaris 10 5/09 (Update 7) sustaining Kernel PatchID and the Solaris 10 Update 8 Kernel PatchID.

We will be releasing a patch bundle containing the set of patches included in Solaris 10 5/09 in the next couple of weeks.  This will be available from the "Solaris Updates Patch Bundle" section on the new look SunSolve Patch Cluster and Patch Bundle page, http://sunsolve.sun.com/show.do?target=patches/patch-access, which now includes a description of the purpose, contents, and update frequency of each patch cluster and bundle.

As always, customers need to have a valid support contract in order to download Solaris patch clusters and bundles.

New PatchFinder tool coming

We plan to release a new PatchFinder tool on SunSolve at the end of May.   This leverages the patch metadata in our release database to provide a much richer customer experience to search for patches.  Further enhancements are planned after the initial deployment.  The old PatchFinder tool will remain available for a transition period.

Zones Parallel Patching Performance Enhancement

The Zones Parallel Patching performance enhancement continues on schedule.  It has been successfully beta tested by a number of key customers who confirm a 3x performance improvement patching zones.   It is on schedule to be released in a revision of the patch utilities patch (119254 SPARC / 119255 x86) in June.

Solaris 10 "Recommended" and Sun Alert Patch Clusters

Improvements to the Solaris 10 "Recommended" and Sun Alert Patch Clusters are on schedule to be deployed before the end of June.  The improvements include an improved install_cluster script (currently available in the Solaris 10 Live Upgrade Zones Starter Patch Bundle and the Solaris 10 Updates Patch Bundles) and other process improvements designed to improve quality.

Solaris 8 Vintage Patch Support Program

The Solaris 8 Vintage Patch Support Program is now up and running.  The first Vintage Solaris 8 patches have been released.

Patches which fix SunAlerts which were issued prior to the April 1 start date of the Solaris 8 Vintage Patch Support program will be released as "normal" (i.e. non-vintage) Solaris 8 patches. 

But all other Solaris 8 patches created after April 1, including patches which fix security issues, require customers to have a Solaris 8 Vintage Patch Support contract to use them.

See http://www.sun.com/bigadmin/topics/vintagepatch/  for further information.

Posted at 02:40PM May 08, 2009 by Gerry Haskins in Sun  |  Comments[0]

Ooops! Please ignore empty feature kernel patches accidentally released

A software glitch caused a number of empty patches to be accidentally released last week.   The PatchIDs are:

• 139555, revisions -01 to -07: Solaris 10: Kernel Patch
• 139556, revisions -01 to -07: Solaris 10_x86: Kernel Patch
• 139981-01: Solaris 10_x86: md patch

The above patches have been removed from SunSolve.   All of these patches were empty (i.e. they contained no payload packages), so they are incapable of being installed or causing any problems on a customer system.   This notice is simply to clear up any confusion.

The correct Kernel patch revisions are now available.  These are:

• 139555-08, which is the Solaris 10 Kernel patch included in Solaris 10 5/09 (Update 7)
• 139556-08, which is the Solaris 10 Kernel patch included in Solaris 10_x86 5/09 (Update 7)

The correct "md" patch revision will be released shortly.  This is:

• 139981-03: Solaris 10_x86: md patch

On behalf of Sun, I apologize most sincerely for any confusion or inconvenience caused. 

Posted at 01:50PM May 08, 2009 by Gerry Haskins in Sun  |  Comments[0]

Patching Zones goes Zoom!

My colleague, Jeff Victor, has written an excellent and informative blog posting on how to improve patching performance, "patching zones goes zoom!".   Enjoy!

Posted at 05:34PM Apr 02, 2009 by Gerry Haskins in Sun  |  Comments[0]

Patch Presentation for Customers

Here's a Solaris patch presentation which I hope will be of use to customers, partners, and field engineers.

It distills much of the information contained in this blog, including the recommended patching strategy, best practices, news on patching enhancements, background information on Solaris and the bug fix process, etc.

Patching isn't a one-size-fits-all operation, but I hope you find the information provided a useful guideline.

Posted at 01:07PM Mar 26, 2009 by Gerry Haskins in Sun  |  Comments[15]

Improvements to Patch Cluster pages on SunSolve

My team and I have been working with the SunSolve team on improvements to the Patch Cluster pages on SunSolve.  These improvements went live on April 20, 2009.

The old "Recommended Patch Clusters" and "Recommended and Security Patches" pages have been combined into a single Patch Cluster & Patch Bundle Downloads page.

A Notice Board section at the top of the page will be used to alert customers to current issues.

Click on the cluster headings to see a brief description of the purpose of the cluster, with links to view the cluster README as well as a download link.   The date the cluster was last updated and the size of the cluster are also shown.

No change has been made to the underlying cluster file names, so scripts using 'wget' to access the patch clusters should be unaffected.

This is part of an ongoing effort to improve our patch presentation to customers.

As before, customers need a valid support contract in order to be able to access patch clusters. 

If you are not registered in Member Support Center, simply log into SunSolve and associated one or more support contracts with your Sun Online Account using the "Change Contract" option in the top right hand menu.

If you are registered in Member Support Center, your contracts will be automatically associated with your account (and the "Change Contract" option will not be shown when you log into SunSolve).

Posted at 03:02PM Mar 10, 2009 by Gerry Haskins in Sun  |  Comments[3]

Need unzip fix available in patch utilities patch to unzip Solaris 10 Clusters

A fix to the unzip utility is available in recent patch utility patch revisions.  This fix is required in order to be able to successfully unzip very large files such as the Solaris 10 Recommended and Sun Alert Patch Clusters.

Please download the latest revision of the patch utilities patch first and install it, before attempting to unzip the Solaris 10 Recommended or Sun Alert Patch Clusters.

The fix was incorporated in the putback to CRs 6344676 and 6464056.

The following are the earliest revisions of the patch utilities containing the fix:

• Solaris 10 SPARC: 119254-46 or above
• Solaris 10 x86:        119255-46 or above
• Solaris 9 SPARC:   112951-14 or above
  
• Solaris 9 x86:          114194-11 or above
  
• Solaris 8 SPARC:   108987-19 or above
  
• Solaris 8 x86:          108988-19 or above
  

Without the fix to unzip provided by the above patches, the following error will be seen when attempting to unzip the Solaris 10 Patch Clusters:

# unzip -q 10_Recommended.zip

note:  didn't find end-of-central-dir signature at end of central dir.
  (please check that you have transferred or created the zipfile in the
  appropriate BINARY mode and that you have compiled UnZip properly) 

In addition, do not unzip Solaris patch clusters on Windows. Solaris patch clusters, and solaris patches more generally, can contain case-sensitive file names. Consequently clusters and patches must be unzipped on a case-sensitive filesystem (corruption can occur if unzipping on filesystems that are not case-sensitive). 

Posted at 03:51PM Mar 05, 2009 by Gerry Haskins in Sun  |  Comments[1]

Cannot patch Solaris 10 from Solaris 8 or 9

I've been asked to post a clarification: 

You cannot patch Solaris 10 from Solaris 8 or 9 as the version of 'patchadd' in Solaris 8 and 9 is totally unaware of how to handle Zones and other Solaris 10 specific features.

If using Live Upgrade to upgrade an inactive boot environment from Solaris 8 or 9 to Solaris 10, you must activate and boot into the Solaris 10 boot environment before patching it.  For example, activate and boot into the Solaris 10 boot environment, and either patch the live boot environment or create another inactive boot environment, and then apply patches to the inactive boot environment.

See http://www.sun.com/bigadmin/features/articles/live_upgrade_patch.jsp for further information.

Posted at 12:00PM Feb 18, 2009 by Gerry Haskins in Sun  |  Comments[0]

Now possible to upgrade directly from Solaris 8 SPARC to latest Solaris 10 release

Thanks to my colleague Enda O'Connor, who has made p7zip available for Solaris 8 SPARC, it's now possible to upgrade directly from Solaris 8 SPARC to the latest Solaris 10 Update releases such as Solaris 10 5/08 and Solaris 10 10/08.  See http://sunsolve.sun.com/search/document.do?assetkey=1-9-250526-1 and http://sunsolve.sun.com/search/document.do?assetkey=1-61-72099-1 for details.

Previously, due to the lack of p7zip on Solaris 8, customers needed to perform an interim upgrade to Solaris 9 or an earlier Solaris 10 release before upgrading to the latest Solaris 10 release.

Posted at 08:00PM Feb 11, 2009 by Gerry Haskins in Sun  |  Comments[2]

Freeing up space in /var

The following is now published as Infodoc 208057:

To whom it may concern,

Over time, customers may experience space issues in /var as patch "undo" and "obsolete" data, which is stored for use during patch backout, builds up as more patches are applied to the system.

It is best to plan for this expansion of /var in advance,  and allocate plenty of free space to this partition. 

How much space to allocate to /var depends on a number of variables, such as how many patches are likely to be applied to the system, which is dependent on frequency of patching, strategy of which patches will be applied (all, sun alert, security only, etc.), products patched; whether the system has zones which implies 'pspool' space requirements, etc.  A figure of 10Gb to 20Gb or even more is not unreasonable.   In my Patch System Test lab, we currently have Solaris 10 systems with >7GB used in /var and this will continue to grow over the lifetime of Solaris 10.

If you have insufficient space in /var of an existing system, the recommended solution is to extend the size of the /var partition.  This can be accomplished by backing up /var, creating a bigger /var partition on another disk, restoring the /var backup onto that partition and then updating the /var entry in the /etc/vfstab file.  If /var is part of the "/" (root) partition, then relocating /var to a separate partition would be a good solution.  The relocation process is documented in Infodoc 215988.

Sun strongly discourages manual modifications to the /var/sadm/pkg directory and its file structure.  That directory structure should only be modified by the Solaris[TM] patching and packaging utilities.  Corruption in this directory structure will prevent any future packaging or patching operations.

Below is information on how to free up space in /var by deleting certain patch related files.  This should only be used when there are no other ways of gaining space in /var.

Having a full and current backup of the OS is highly recommended before deleting files from /var.

On behalf of Sun Microsystems, I hereby confirm that unneeded "obsolete" and "undo" files may be removed from the /var/sadm/pkg/<pkgname>/save/<PatchID> and "undo" files from the /var/sadm/pkg/<pkgname>/save/pspool/<pkgname>/save/<PatchID> directories in order to free up space in /var.

This will not invalidate support contracts.

The "undo" files are used during the patch removal process. Removing the "undo" files associated with a patch means that it will no longer be possible to uninstall the patch.

Once a later revision of a patch is applied, or a patch which obsoletes a patch is applied, the "undo" files in /var/sadm/pkg/<pkgname>/save/<PatchID> are renamed "obsolete".  These "obsolete" files may also be removed.

It is strongly recommended to only remove the "obsolete" and "undo" files for patches which the customer is confident will not need to be backed out - for example rejuvenated Kernel patches associated with Solaris 10 Update releases such as 118833-36 (SPARC) / 118855-36 (x86), 120011-14 (SPARC) / 120012-14 (x86), etc. and other older Kernel patch revisions.  See the sequence of Solaris 10 Kernel PatchIDs listed on http://blogs.sun.com/patch/date/20080416

As patches containing subsequent fixes to the objects contained in these rejuvenated patches will have a SUNW_REQUIRES dependency on the these rejuvenated patches, it's extremely unlikely that older versions of these rejuventated Kernel patches will ever be backed out.   Therefore, their "undo" files are prime candidates for removal to free up space in /var if required and it's not possible to extend /var.

All the "undo" or "obsolete" files for a particular patch should be removed from the /var/sadm/pkg/<pkgname>/save/<PatchID> directories. For example:

rm /var/sadm/pkg/*/save/118833-36/undo*

The system also keeps a copy of "undo" files for use during creation of new non-global zones. These may also be removed if the customer is confident that the patches will not need to be backed out. For example:

rm /var/sadm/pkg/*/save/pspool/*/save/118833-36/undo*

The "undo" files in the pspool directory are not renamed to "obsolete" when a later revision is installed.

Further Information:

1. The "undo" files are specific to the zone of the system on which they were created during application of a patch by patchadd. Therefore, do not copy the "undo" file from one zone to another or from one system to another.

2. One could potentially archive the "undo" files for each zone of a system and restore it to that zone if desired.

Best Wishes,

Gerry Haskins
Director, Software Patch Services

Posted at 03:15PM Jan 21, 2009 by Gerry Haskins in Sun  |  Comments[5]

Stricter Solaris patch entitlement implementation roll-out commencing this week

This blog entry expands on a previous blog entry regarding Solaris patch entitlement.  

The Solaris patch entitlement policy is available on http://sunsolve.sun.com/search/document.do?assetkey=1-61-203648-1. "Entitlement" refers to patches which require you to have a valid support contract to access them.

Solaris changed its business model a few years ago from selling Solaris and providing patches for free to a model of giving away the Solaris releases for free and charging for patches.

The Solaris patch entitlement policy applies to all Solaris Operating System patches.  It does not necessarily apply to middleware or application layer product patches which may be installed on top of Solaris, such as SunStudio, Java, etc.

The Solaris patch entitlement policy is that the following Solaris OS patches will remain available irrespective of whether or not you have a valid support contract:

• the specific patch revisions which introduce all new security fixes
• the specific patch revisions which introduce certain hardware support
• all revisions of Solaris patch utility, smpatch, and Update Manager patches to ensure correct patch application
  
• the specific pre-requisite patch revisions for Live Upgrade
  
• the specific pre-requisite patch revisions for certain Sun software application products
• all revisions of patches which patch products which are both bundled as part of Solaris and also released as separate products which don't enforce patch entitlement
  
• a small number of other specific patch revisons at the discretion of Sun
• any patch revision explicitly required by any of the above patches
  

Other Solaris OS patches require that you have a valid support contract to access them.

All fixes will all be available for free in the next Solaris 10 Update release, so if you are not willing to pay for a support contract, you can still get the fixes by installing or upgrading to the next Solaris 10 Update release.  You'll just need to wait for it to be released.

The key point is that if you may need timely access to a patch which fixes a critical non-security issue, then you need to have a valid support contract for each system you may wish to patch.  You also need to have a valid support contract in order to get telephone support or fixes coded for any issues which are unique to your environment.

So it's highly advisable for you to have a valid support contract in place for each production system.

If you are a home user for example, and don't want to go to the expense of buying a support contract, using OpenSolaris or waiting for the next Solaris 10 Update release are valid options.

This policy is not changing.

What is changing is the implementation of patch entitlement to ensure it matches the policy.  Currently, circa 60% of Solaris OS patches are available without a support contract, including most of the key patches.  Under the new entitlement implementation, 18% of Solaris OS patches will remain available without a support contract.  The rest will require a valid support contract to access. 

Any of the following support contracts will provide you with access to all Solaris OS patches and patch clusters: a Solaris subscription, a Software Support Contract, a Sun System Service Plan for Solaris, a Sun Spectrum Storage Plan, or a Sun Spectrum Enterprise Service Plan.  Since the names of the support contracts change from time-to-time, this list may change.

If you are running Solaris on Sun Hardware, I suggest you consider purchasing a SunSpectrum System Plan.  This will cover both your HW and OS with one simple support contract.

If you are running Solaris on non-Sun hardware, you should consider a Solaris Subscription Support Plan, which is available on-line from just $324 per year.

Remember, you need a support contract for each system you wish to patch, so if you need more of a site-wide support plan, Solaris Everywhere is a good choice. 

BTW: It's important to remember that hardware warranties do not cover software support or access to Solaris patches.

The new implementation will roll out in phases, starting this week.

You should check that you have valid support contracts in place for each system you may need to patch.  Please do not wait until you need a patch to put the support contract in place. There is a latency of several days between subscribing for a support contract and patch access being granted.  Support for your production Operating System really isn't something you should play "chicken" with.

The new Solaris OS patch entitlement implementation roll-out should be completely transparent if you have a valid support contract for each system you wish to patch.

A PodCast talking about the above and the Solaris 8 Vintage program which commences April 1, 2009 is available here

Posted at 02:52PM Jan 05, 2009 by Gerry Haskins in Sun  |  Comments[39]

OpenSolaris code swarms

Nothing directly to do with patches but a colleague of mine, Albert White, has produced some code swarm videos of OpenSoalirs code changes which I find way cool.

I've no idea what's it's telling me, but I'm sure it's telling me something useful if I learn to listen.  For example, Area of code change and amount of code change => Risk => Areas on which to concentrate test coverage.

I've asked Albert if he can do something similar for Solaris 10. 

BTW: I like his choice of music too!

From Albert to <pkg-discuss@opensolaris.org>, opensolaris-discuss@opensolaris.org :

A bit OT but this might be of interest to some of you.

You may have seen code_swarm videos on vimeo or youtube showing a visualisation of code development (from the files changed in the putback logs) over time for various projects.

I've put up videos of the [OpenSolaris] codeswarms of IPS [Image Packaging System] and ON [core Solaris Operating System and Network] here http://blogs.sun.com/albertw/entry/opensolaris_code_swarms

Cheers,

~Albert

Posted at 03:48PM Dec 17, 2008 by Gerry Haskins in Sun  |  Comments[3]

Definitive interpretation of the "rebootimmediate" and "reconfigimmediate" patch flags

The following is now available as Infodoc 249046:

What follows is an open letter to customers in response to customer confusion over how to handle the "rebootimmediate" and "reconfigimmediate" flags specified in some patches.

Despite the READMEs of patch clusters which contain such patches clearly stating that during a patching session, a reboot is only required in exceptional and documented circumstances, it has come to my attention that some customers are initiating reboots after applying every single patch in a patch set which specifies such flags.  Not surprisingly, such customers are concerned at the length of time this takes.

Open Letter with definitive interpretation of the "rebootimmediate" and "reconfigimmediate" patch flags

To whom it may concern,

Summary: When patching a live boot environment, it is usually OK to apply any number of patches before performing a single reboot at the end, even if multiple patches specify "rebootimmediate" or "reconfigimmediate".  On the rare occasion when it is found that this is not possible, specifically for 118833-36 (SPARC) and 118855-36 (x86) and 118844-14+ (x86), code will typically be inserted into the relevant patches to prevent the application of further patches which could cause problems.  Use of Live Upgrade to patch an inactive boot environment is recommended as it avoids the need for interim reboots for even these atypical patches.  Details below.

The "reboot" metadata flags which may be contained in the patch 'pkginfo' file(s) have the following meaning:

rebootafter - a reboot is required to activate some of the content delivered in the patch, but the system remains in a consistent state until the reboot is performed.

reconfigafter - a reconfiguration reboot is required to activate some of the content in the patch, but the system remains in a consistent state until the reconfiguration reboot is performed.

rebootimmediate - the system is in a potentially inconsistent state until the system is rebooted.  The objects applied in the patch are potentially inconsistent with processes running in memory.  Normal production must not be resumed until a reboot takes place to bring the system back into a fully consistent state.  However, since the footprint of the patch utilities is relatively small, it is normally OK to continue to apply further patches before initiating the reboot.   In cases where this is not OK, the patch in question will typically contain additional code to prevent further patches from being applied until the reboot takes place*.  Since the system is in a potentially inconsistent state, it's advisable to avoid running any additional processes until the reboot takes place.  If patch automation tools are being used to apply "rebootimmediate" or "reconfigimmediate" patches, it's up to the automation tools' QA to ensure that their additional code footprint does not hit the potential inconsistent system state when applying such patches.

reconfigimmediate - exactly the same as rebootimmediate, except a reconfiguration reboot is required.

*This is the case with Kernel patch 118833-36 (SPARC) / 118855-36 (x86), whose patch scripts replace 'patchadd' with a no-op telling the user to reboot the system.  The only other known reboot required before further patching can be done is specific to x86, and only if the system is running at a Kernel patch level below 118844-14.  A later revision of 118844, e.g. 118844-20, needs to be applied and the system rebooted to ensure the Kernel running in memory is compatible with library changes supplied in the libc patch 121208-02.  The prepatch script in 121208-02 and -03, and 118855-xx which obsoletes it, contains code to ensure 118844-14 or later is installed and active on the system.  (BTW, 118844-14 wasn't released. 118844-20 is recommended to fulfill the libc compatibility requirement.)

UPDATE, Jan 20, 2009: Murphy's Law strikes again!.  There's currently an issue, CR 6704883, with the "Sun Fibre Channel Device Drivers" patches 125184-05, -06, -07, and -08 (SPARC) and 125185-05, -06, -07, and -08 (x86) as described in Sun Alert 238630.  The fix for this issue is in rev-09 of the patches which is currently available as a T-Patch and will be released shortly.  Rev-09 of the patches uses modloading in its prepatch script to avoid the issue.  In the meantime, a workaround is to apply the affected patches last, immediately prior to rebooting the system.  The patches in the Solaris 10 10/08 patch bundle were specifically ordered to avoid this issue.  Where such issues are found, SunAlerts are published and the issue fixed.

Remember, patches can be downloaded and installed individually.  Therefore, each patch which requires a reboot must specify the reboot requirements.  But if patches are installed collectively in the same patching session, for example, as part of a patch cluster, then the install instructions contained in the cluster README file take precedence - e.g. that reboots are only required *during* patching sessions for the specific cases mentioned above.

Since the above patches were created, a significant enhancement has been made to the Solaris patch utilities called Deferred Activation Patching.  This enhancement is not retrospective, so the above historical problematic patches remain.

Deferred Activation Patching

The problem with the above atypical patches is that the new code they deliver may be invoked by the original patchadd code and the utilities it calls *during* patch installation.  A patch may patch many packages.  The packages are applied in alphabetic order.  In a Zones environment, the patch is applied to the global zone first, then to each non-global zone.

In the case of 118833-36 (SPARC) / 118855-36 (x86), the new versions of the libdevinfo.so.1 and libsec.so.1 libraries delivered in the patch could be invoked by patchadd and are potentially incompatible with the processes running in memory.

The solution devised in the patch scripts contained in 118833-36 (SPARC) / 118855-36 (x86) is to overlay mount the old objects on top of the newly laid down objects using the loopback filesystem (lofs).  This ensures that the system remains in a consistent state *during* the patch process as the old library versions which are compatible with what's running in memory will be called.

To avoid the application of further patches, which patch the same objects as 118833-36 (SPARC) / 118855-36 (x86), from patching the overlay mounted objects instead of the patched objects, 118833-36 (SPARC) / 118855-36 (x86) replace 'patchadd' with a no-op telling the customer to reboot the system before applying any further patches.

During reboot, the loopback filesystem mounts are torn down exposing the patched objects.  Further patching can now continue as the system is in a fully consistent state.

This loopback filesystem mount solution is the basis of Deferred Activation Patching.  After patch 118833-36 (SPARC) / 118855-36 (x86) was released, the solution was perfected and moved to the patch utilities.  The few patches which require application using Deferred Activation Patching specify the SUNW_PATCH_SAFE_MODE=true flag in their pkginfo files.  The solution was enhanced so that any subsequent patch applied prior to a reboot of the system, which patches the same objects as a patch explicitly specifying Deferred Activation Patching, will itself be automatically applied in Deferred Activation Patching mode.   This is known as implicit Deferred Activation Patching and enables other patches to be applied on top of a patch applied using Deferred Activation Patching without the need for an intervening reboot.  When a patch specifying Deferred Activation Patching mode is applied to a system, the user will see lots of loopback filesystem mounts on the system until such time as the reboot takes place.  Upon reboot, the loopback filesystem mounts are torn down, exposing the newly patched objects.

Kernel patch 12001[12]-14 which is included in Solaris 10 8/07 (Update 4), Kernel patch 12712[78]-11 which is included in Solaris 10 5/08 (Update 5), and Kernel patch 13713[78]-09 which is included in Solaris 10 10/08 (Update 6), are currently the only patches which specify application in Deferred Activation Patching mode.  Future Kernel patch included in future Solaris 10 Update releases are the likely candidates requiring application using Deferred Activation Patching.

With the introduction of Deferred Activation Patching, it is highly unlikely that future patches will require an interim reboot before further patches can be applied.

The problems with the system getting into an inconsistent state *during* patching (which Deferred Activation Patching resolves) could only occur when patching a live boot environment as it's due to the interaction between newly patched objects which are incompatible with processes running in memory being invoked prior to the system being rebooted.

To avoid this and other issues, Sun strongly recommends the use of Live Upgrade to patch (or upgrade) an inactive boot environment, which dramatically reduces the risk and downtime associated with patching.  For example, even though Deferred Activation Patching resolves the inconsistency issue, patching a live boot environment takes time and the system is out of production.

Using Live Upgrade, the inactive boot environment is patched, potentially while the system is still in production.  Issues such as those described above with Kernel patch 118833-36 (SPARC) / 118855-36 (x86), and 118844-20 (x86) simply don't apply when patching an inactive boot environment as there is no interaction between the objects being patched and the processes running in memory, as all the calls patchadd makes will be to the objects on the live partition, not the patched objects on the inactive partition.  A single reboot is required to boot into the new boot environment.

Another advantage of Live Upgrade is that if a problem arises with the new boot environment for whatever reason, the user can simply reboot back into the old boot environment to enable production to resume and the issues with the now inactive boot environment can be resolved later.

Best Wishes,

Gerry Haskins
Director, Software Patch Services

Posted at 09:14AM Dec 17, 2008 by Gerry Haskins in Sun  |  Comments[3]

Patching enhancements and other stuff

New title, same role, same me

I was promoted to Director, Software Patch Services in September.  The last couple of months have been quite hectic, as I've suddenly got a whole new bunch of buddies in Marketing and elsewhere who want some of my time.  That's a good thing, and I believe it will help me to drive and co-ordinate improvements for you, our customers, patching experience. 

Resources are limited and, as always, I'm interested in getting your thoughts as to what areas I should concentrate on next.  

Some of the stuff we're currently working on is outlined below as well as other information which I hope you will find useful.

Solaris 10 10/08 Patch Bundle

The Solaris 10 10/08 Patch Bundle, which delivers the equivalent set of patches to the Solaris 10 10/08 (Update 6) release image, is now available from SunSolve.  See my blog entry below on the Solaris 10 5/08 (Update 5) Patch Bundle for further information on why we produce it, what it contains, why you might wish to use it, how to download it, etc.

Recommended and Sun Alert patch cluster contents updated

I discussed the purpose of, and difference between, the Solaris Recommended and Sun Alert patch clusters in a previous blog posting. To recap:

The "Recommended" Cluster contains the latest revision of any Solaris OS patch which addresses a Sun Alert issue.  That is, a fix for a Security, Data Corruption, or System Availability issue.  The cluster also contains the latest revision of the patch utility patches to ensure correct patch application and any patch required by any other patch in the cluster.

The Sun Alert Cluster is newer, and contains the minimum revision of any Solaris OS patch which addresses a Sun Alert issue. The cluster also contains the latest revision of the patch utility patches to ensure correct patch application and any patch required by any other patch in the cluster.  Therefore, the Sun Alert Cluster provides the minimum amount of change to fix all Solaris OS Sun Alert issues. 

Both clusters are updated whenever a new patch meeting their inclusion criteria is released.  The Sun Alert Cluster changes less frequently than the "Recommended" Cluster as it contains only what is really needed to address Sun Alert issues and apply the patches.

One of my team members has been reconciling the cluster contents against the Sun Alert reports and the cluster contents have been updated as a result.  Some issues where found, largely to do with patches for things like GNOME which are also part of the Solaris OS.  A process has been put in place to ensure the cluster contents match the patches specified in the Sun Alert reports.   

Keeping as up to date as possible with the SunAlert or Recommended Cluster contents is advisable.   Remember also to keep firmware up to date.

BTW: The monthly EIS (Enterprise Installation Standards) patch baseline is based upon the Recommended Cluster contents but also includes ca. 150 additional patches to address irritants which are not Sun Alert fixes and includes patches for SunCluster, SunVTS, etc.  The monthly EIS patch baselines are available through xVM Ops Center and Sun Proactive Services.

I am planning to merge the Recommended and Sun Alert patch clusters into a single cluster using the Sun Alert cluster criteria as having two very similar clusters tends to confuse customers unnecessarily.  

I also intend to merge the two cluster pages on SunSolve as one is essentially a better formated subset of the other. 

ZFS and Zones features fully contained in patches

As I've mentioned previously, there's effectively a single customer visible code branch for each Solaris named release.  That means that there's one set of patches for all of Solaris 10, a separate set for Solaris 9, and a separate set for Solaris 8.  Within a named release, e.g. Solaris 10, the same set of patches will apply to any of the Solaris 10 releases, from the original Solaris 10 3/05 release right up to the current Solaris 10 10/08 (Update 6) release.  This simplifies System Administration and enables Sun to provide very long term support at reasonable cost for each Solaris named release. 

A consequence of effectively having a single code branch for each Solaris named release is that any change to pre-existing packages will be delivered in patch format.

New features are typically only added to the current Solaris named release, which is currently Solaris 10.  (They are also available via OpenSolaris.)

This means that if new features don't add any new packages, then the entire feature functionality is fully available in patches.  Customers can utilize the new features by simply applying the appropriate patches to their existing Solaris 10 system.  This is the case with all current Zones and ZFS* functionality, including neat features like ZFS Root, ZFS Boot, and Zones "Update on Attach".

Other features which deliver new packages are only available from the Solaris Update release in which they were first included.  So, for example, if a new package was first delivered in Solaris 10 8/07 (Update 4), then a customer wishing to use that feature would need to install or upgrade to the Solaris 10 8/07 (Update 4) or subsequent update release image.   Such features are not available in patches.

*OK, we cheated with ZFS.  ZFS does deliver new packages, but they are streamed into existence from a patch.  This type of patch is called a "genesis" patch, but they are hard to perfect, so we don't intend to release any more "genesis" patches.

Improving Zones Patching Performance

Zones Parallel Patching

My team has been working with those awfully nice folks in the Sustaining organization to deliver a Zones Parallel Patching enhancement to the patch utilities to dramatically improve Zones patching performance.  We have a fully stable prototype which has been given to selected Beta customers to trial. 

For a simple T2000 with 5 sparse non-global zones, the performance improvement is >3x.  On systems with optimized I/O (as Zones patching is primarily I/O bound), we expect the performance improvement to be even better.  A configuration file will allow users to select how many Zones to patch in parallel.  This will typically equate to the number of processors or threads available on the target system.

The general release of this feature is planned for April 2009.

Zones "Update on Attach" 

The Kernel patch associated with Solaris 10 10/08 (Update 6), 137137-09 (SPARC) / 137138-09 (x86) contains some cool new features, such as ZFS Root, ZFS Boot, and Zones "Update on Attach".  Beware, installing this patch requires significant free disk space to install!  See Sun Alert http://sunsolve.sun.com/search/document.do?assetkey=1-66-246207-1

Zones "Update on Attach" is a very cool feature indeed.

For example, if the patch level of non-global Zones is out-of-sync with respect to the global Zone, e.g. because the non-global Zones ran out of disk space during patch application, Zones "Update on Attach" provides a very neat way to bring the Zones back into sync.  Simply detach the affected non-global Zones, apply Kernel patch 137137-09 (SPARC) / 137138-09 (x86) to the global zones, and reattach the affected non-global Zones using 'zoneadm -z <zone-name> attach -u'.  The non-global Zones will be automagically updated to the same patch level as the global Zone.  Neat!

There are other interesting possibilities.  For example, detach all non-global Zones, apply an arbitrary set of patches to the global Zone (including 13713[78]-09), and reattach the non-global Zones using 'zoneadm -z <zone-name> attach -u'.  Viola!, the non-global Zones will be automagically updated with all of the patches applied to the global Zone.  Way neat!  And more importantly, way faster than even the Zones Parallel Patching solution we're working on.  And even better, it's available now!  This could be a key solution for customers having difficulty completing patching updates on Zones systems during tight maintenance windows.

We are working to explore potential caveats.  For example, when a patch is applied using 'patchadd' to a non-global zone, an "Undo.Z" file containing the data necessary to back out the patch is created specifically for each non-global zone to which the patch is applied.   Using Zones "Update on Attach" to patch non-global Zones will cause the "Undo.Z" file from the global Zone to be propagated to the non-global Zones.  This could theoretically cause issues if the patch is subsequently backed out (e.g. data from global Zone config files could potentially be merged into non-global Zone config files during patch backout which could potentially cause issues), although we've never actually encountered such an issue.  BTW: The same caveat applies to creating non-global Zones after the global Zone has been patched.  Again, we have yet to see this causing an actual issue, so it appears to be more of a theoretically caveat than a practical issue.

Improvements to 'smpatch' and Update Manager

The way the PatchPro analysis engine for 'smpatch' and Update Manager used to work was fine in theory, but in practice was what I call "a process with too many moving parts".   Too many steps had to happen correctly for the overall result to be correct.  In Six Sigma terms, there was too much error opportunity.  Occasionally, it would end up recommending a SPARC patch for an x86 system or a Solaris 8 patch for a Solaris 10 system.  Not surprisingly, its reputation suffered.

I'm pleased to say that a major overhaul to dramatically simplify the back end processing of 'smpatch' and Update Manager has just been rolled out by their engineering team.  The way 'smpatch' and Update Manager work is that Realization Detector(s) are associated with each patch.  These Realization Detectors determine whether it's appropriate to recommend a patch for application on a target system.  In the vast majority of cases, the Realization Detectors are simply comparing the packages contained in the patch to the packages installed on the system to see if the patch is applicable.  The enhancement is to replace these myriad Realization Detectors, which could potentially contain coding bugs, with a single Generic Realization Detector to map patch packages to packages on the target system.  It looks at the package name, package version, and package architecture fields (in pkginfo) for each package in the patch, and compares them to the same values for the packages installed on the target system.  If they match, the patch is recommended, else not.  Guess what, this is exactly how 'patchadd' decides whether a patch is applicable or not when installing a patch.  It's also how 'pca' works too in determining which patches to apply.

A few specialist Realization Detectors remain for a small number of patches which require special handling.

The changes to 'smpatch' and Update Manager should dramatically improve the reliability of these tools and the accuracy of their patching recommendations.

One remaining distinction between 'smpatch' / Update Manager and 'pca' is that 'pca' "knows" about all current Sun patches via the patchdiag.xref file, whereas 'smpatch' / Update Manager "knows" about all patches containing a 'patchinfo' file, including older patch revisions.  All Solaris OS and Java Enterprise System (middleware) patches contain a 'patchinfo' file.  These account for 49% of patches.  For patching the Solaris OS, the tools should produce similar results.  A decision was made not to "auto-include" all other patches for 'smpatch' and Update Manager, as it was felt that the explicit step of the patch creator including a non-blank PATCH_CORRECTS realization detector specification line in the 'patchinfo' file to signal that the patch was suitable for patch automation was potentially useful.  (Don't worry about what value the PATCH_CORRECTS field has.  This is overriden by the Generic Realization Detector in the vast majority of cases.  It has no meaning from a customer perspective.)

This enhancement is not an attempt to undermine 'pca'.  It's simply to improve 'smpatch' and Update Manager.  I will continue to work closely with Martin Paul to give him heads-ups on any initiative which may impact 'pca' and resolve any issues with patchdiag.xref.

One thing I want to do when I can free up some resources, is a comparative study of the patching recommendations of the various available patch automation tools, 'smpatch' / Update Manager, 'pca', UCE (a.k.a Sun Connection Satellite),  xVM Ops Center*, and TLP (Traffic Light Patching) which is used by Sun Proactive Services to provide tailored patching solutions for customers in conjunction with SRAS (Sun Risk Analysis Service) and the EIS (Enterprise Installation Standards) methodology, with a view to ensuring that the patching recommendations of the various tools are coherent and consistent, with the higher value tools providing more sophisticated analysis.  It's part of my efforts to co-ordinate patching improvements to improve our customers' patching experience.

*xVM OC also utilitizes the monthly EIS patch "baselines".

Same Patch Entitlement policy, new Patch Entitlement implementation

Solaris changed its business model a few years ago from selling Solaris and providing patches for free to a model of giving away the software releases for free and charging for patches. 

The policy is that patches delivering new security fixes will remain free to all customers, irrespective of whether or not they have a support contract, but most other patches require that customers have a valid support contract to access them.  (See my earlier blog entry on the subject.)

All fixes will all be available for free in the next Solaris Update release (and OpenSolaris), so customers not willing to pay for a support contract can still get the fixes by installing or upgrading to the next Solaris Update release.  They'll just need to wait for it to ship.  Alternatively, they can use OpenSolaris.

This policy is not changing.

What is changing is the implementation of patch entitlement to ensure it matches the policy.  Currently, circa 60% of Solaris patches are free, including most of the key patches.  Under the new entitlement implementation, 18% of Solaris patches will remain free, including the specific revision of all Solaris patches which include new security fixes.  The rest will require a valid support contract to access. 

Any of the following support contracts will provide access to all Solaris patches and patch clusters: a Solaris subscription, a Software Support Contract, a Sun System Service Plan for Solaris, a Sun Spectrum Storage Plan, or a Sun Spectrum Enterprise Service Plan.  Since the names of the support contracts change from time-to-time, this list may change.

The new implementation will roll out in Phases, starting this month.  The roll-out should be transparent to customers with valid support contracts.

Patch signing certificate renewal

The signing certificate used to sign Sun patches expires shortly.  A new signing certificate will be rolled out in January and instructions provided on how to adopt it.

Customers who download the unsigned patch versions will not need to take any action.

"Accumulation-only" patches

The "SplitGate" source code management model we first introduced in Solaris 10 8/07 (Update 4) has dramatically improved Solaris 10 patch quality.  A side-effect of the "SplitGate" model is that base PatchIDs (the first 6 digits) change at the end of each Update release.  See my earlier Solaris 10 Kernel PatchID Sequence posting.

In the "SplitGate" model, when building an Update release, we effectively have two parallel source code gates, one called the Sustaining Gate containing just the bug fixes we need to release to customers in patches asynchronous to the Update release, and the other called the Update Gate containing a superset of the the Sustaining Gate and as well as new features and less critical bug fixes which will be released as part of the Update release. 

The two gates remain separate (split) for the duration of the Update release build process.  Once the Update release has reached release quality, the Update Gate is promoted to become the new Sustaining Gate and the process repeats.  Since the Update Gate is always a strict superset of the Sustaining Gate, no regressions should result from the promotion of the Update Gate to become the new Sustaining Gate.  Each patch in the old Sustaining Gate is obsoleted by a corresponding patch from the Update Gate which has accumulated its contents.  When the Update is released, these new PatchIDs are released to SunSolve.  This is why you see the base PatchIDs changing after each Update release. 

If the Update Gate patch doesn't contain any additional code changes over the corresponding Sustaining Gate patch, then there's no need for customers to install the new Update Gate patch.  Such patches are called "accumulation-only" patches and can be identified as they have a different base PatchID (the first 6 digits) but don't contain any additional CR numbers over the Sustaining patch which they obsolete.

The reason Sun releases these "accumulation-only" patches is because some customers insist that all of the PatchIDs pre-applied into a Solaris Update release image be also available from SunSolve.

Posted at 06:29PM Dec 04, 2008 by Gerry Haskins in Sun  |  Comments[6]

Why it is now more important to keep your SPARC and x86 Firmware up to date

My colleague, Damien Farnham, from the Performance Regression Benchmarking team, has written an informative blog posting on why it is increasingly important to keep SPARC firmware up to date (similar to the importance of keeping x86/x64 firmware up to date).  A lot has changed since the humble OBP!

Please see Best Practise In Firmware Patching for the T1000/T2000/T5X20 platforms.

And he's now added an entry for x86 Firmware too.

See also the Firmware section on the Big Admin Patching Hub, especially the FAQ.

One area you may not be aware of is that upgrading Firmware can lead to significant performance improvements in certain circumstances.

Posted at 06:06PM Sep 10, 2008 by Gerry Haskins in Sun  |  Comments[0]

Solaris 10 Live Upgrade Zones Starter Patch Bundle

The Solaris 10 Live Upgrade Zones Starter Patch Bundle has been released.  It is designed to make it simpler for customers running on systems below Solaris 10 5/08 (Update 5) to apply the pre-requisite patch level needed to be able to utilize basic Live Upgrade functionality in a Zones environment.  These patches need to be applied to the live boot environment to enable Live Upgrade to work correctly in a Zones environment.

Aside: Customers with systems running Solaris 10 5/08 (Update 5) or later already have all the  pre-requisite patches pre-installed on the live boot environment and hence do not need to apply this patch bundle.

After this, Live Upgrade itself can be used to create an inactive boot environment and apply any additional patches referenced in SunSolve document 206844 'Solaris[TM] Live Upgrade Software: Minimum Patch Requirements' (formerly Infodoc 72099) to provide advanced Live Upgrade functionality such as support for ZFS Root. The document is available from: http://sunsolve.sun.com/search/document.do?assetkey=1-61-206844-1

The Solaris 10 Live Upgrade Patch Bundle is available from the normal patch cluster download center on SunSolve.  To download the patch bundle, login to SunSolve, http://sunsolve.sun.com , click on the Patches and Updates link, click on Recommended Patch Clusters, and scroll down the window under the heading "Recommended Solaris Patch Clusters, J2SE and Java Enterprise System Clusters" to find the "Solaris 10 SPARC Live Upgrade Zones Starter Patch Bundle" or "Solaris 10 x86 Live Upgrade Zones Starter Patch Bundle".  As always, you need a valid support contract to access patch clusters.  See previous postings for further information on support contracts.

Posted at 05:04PM Jul 04, 2008 by Gerry Haskins in Sun  |  Comments[4]

More info on patching using Live Upgrade

My colleague, Enda O'Connor, has written another useful article on Big Admin about patching using Live Upgrade, restrictions, and how-to use Live Ugrade to upgrade/patch from Solaris 8 or Solaris 9 to Solaris 10.  See http://www.sun.com/bigadmin/features/articles/live_upgrade_patch.jsp

Posted at 03:27PM Jun 19, 2008 by Gerry Haskins in Sun  |  Comments[0]

Solaris 10 05/08 (Update 5) Patch Bundle

Last week, the Solaris 10 05/08 (Update 5) Patch Bundle was released on SunSolve.  The patch bundle provides another option to customers when deciding their patching strategy to maintain their Solaris systems.

What is the Solaris 10 05/08 Patch Bundle ?

The Solaris 10 05/08 Patch Bundle contains the equivalent set of patches contained in the Solaris 10 05/08 (Update 5) release image.

Why use the Solaris 10 05/08 Patch Bundle ?

The Solaris 10 05/08 Patch Bundle was created as a result of direct customer feedback after the Solaris 10 08/07 (Update 4) release.  New hardware may require a specific minimum Solaris 10 Update release such as the Solaris 10 05/08 (Update 5) release.  Some customers may wish to bring their other existing Solaris 10 systems up to the same patch level as the new hardware running Solaris 10 05/08.  The recommended way to do this is to upgrade the existing systems to the Solaris 10 05/08 release using either regular Solaris Upgrade or Solaris Live Upgrade.  But some customers may have policies in place which make it difficult to upgrade but OK to patch a system.  The Solaris 10 05/08 Patch Bundle facilitates such customers to bring their existing systems up to the equivalent patch level to the Solaris 10 05/08 (Update 5) Release.  In theory, this should mean that pre-existing functionality on all of the customers' systems should react the same, warts and all. This makes for a more homogeneous environment which may help lower support costs.

The Solaris 10 Update releases are very intensely tested by a wide variety of QA teams within Sun.  Therefore, the functionality contained in the patches within the Solaris 10 05/08 Patch Bundle have been intensely tested as a unit through the testing performed on the Solaris 10 05/08 (Update 5) release image.  Additional testing of the Solaris 10 05/08 Patch Bundle has also been performed by the Patch System Test team.  Therefore, the Solaris 10 05/08 Patch Bundle provides a well tested "baseline" option on which to standardize systems.

So while the patch bundle may deliver more change than some other patching strategies, that change has been well tested as a unit and hence may actually reduce the risk of introducing regressions when compared to "dim sum" patching (i.e. choosing an arbitrary combination of patches).  Note that intensive processes are also in place to ensure "dim sum" patching works, and it's rare to encounter a problem caused by "dim sum" patching.

How does the Patch Bundle differ from the Solaris 10 05/08 (Update 5) release image ?

The Solaris 10 05/08 (Update 5) release is a complete Solaris release image.  It contains new packages to support new features in the Solaris 10 05/08 (Update 5) release as well as all Solaris patches which were available when the Update was built.  The patches are pre-applied into the Solaris 10 05/08 release image.  This means that one doesn't have to spend time adding the patches using 'patchadd'.  On the flipside, since the patches are pre-applied into the release image, they cannot be backed out using 'patchrm'.  This isn't generally a problem as the Solaris Update release images are very intensely tested.  One can do a fresh install of the Solaris 10 05/08 (Update 5) release, or upgrade to it from an earlier Solaris release.

The Solaris 10 05/08 Patch Bundle contains the equivalent set of patches to the Solaris 10 05/08 (Update 5) release. The patch bundle does not include the new packages contained in the Solaris 10 05/08 (Update 5) release.  Therefore, new features in Update 5 which depend upon new packages introduced in that release will not be available in the patch bundle.  However, as discussed in a previous blog entry, any change to pre-existing code is delivered in a patch.  This includes features as well as bug fixes.  Therefore some feature enhancements will be available in the patch bundle.  ZFS, for example, is typically self-contained in patches and hence ZFS enhancements will typically be available via the patch bundle as well as via the Update release image. So will most Zones enhancements.  The patch bundle is simply a collection of patches with an install order file (patch_order) and an install script wrapper (installbundle.sh) around 'patchadd'.  Patches in the patch bundle can be backed out using 'patchrm', so long as the '-d' (no save) option wasn't used when applying the patch bundle.

There are a number of "special" or "script" patches included in each Solaris Update release.  These patches are used to correct issues in how patches are pre-applied into the Solaris Update release image and have no purpose whatsoever outside of the Solaris Update release process.  Therefore, these "special" or "script" patches are not released to SunSolve and are not included in the patch bundle.  See the Solaris 10 05/08 Patch Bundle README file for further information on these and other minor differences between the patch set pre-applied in the Solaris 10 05/08 release image and the patch set included in the Solaris 10 05/08 Patch Bundle.

Access 

The Solaris 10 05/08 Patch Bundle is available from the usual patch cluster location. 

Log onto Sunsolve, click on Patches and Updates, then Recommended Patch Clusters and scroll down the box under "Recommended Solaris Patch Clusters, J2SE and Java Enterprise System Clusters" to the Solaris 10 SPARC 05/08 Patch Bundle and Solaris 10 x86 05/08 Patch Bundle entries.  

The cluster is chunked to aid download.  There are 2 chunks for x86 and 3 chunks for SPARC. 

Follow the download instructions to the right of the scroll-down box or read the README file for any chunk.

As with all patch clusters, you need a valid support contract to download the cluster.   The following support contracts include access entitlement to Solaris patches and Patch Clusters (BTW: Software Update = patch), plus a wide range of additional support services:  Solaris Subscriptions, which includes Basic, Standard, Premium, and Solaris Everywhere Service Plans (compare here); Sun Software Service Plans, including Basic, Standard, Premium, and Premium Plus; Sun System Service Plans for Solaris, which includes Bronze, Silver, Gold, or Platinum options (compare here); or a Sun Spectrum Enterprise Service Plan.  See also http://www.sun.com/servicelist/ for country specific details.

Installation

Read the Patch Bundle README file for full installation instructions.

The patch bundle can be installed either on the active boot environment (i.e. the live system) or an inactive boot environment. 

Patching an inactive boot environment is recommended as, depending on the starting patch level of the target system, it may involve less system downtime as only a single reboot is required at the end to activate the boot environment. 

If you patch the active boot environment (i.e. the live system), then depending on the starting patch level of the target system, you may need to reboot an x86 system up to three times (twice at specific points during the installation process and once at the end) and a SPARC system up to two times (once after installing Kernel patch 118833-36 and once at the end).  See the patch bundle README for details.

The Solaris 10 05/08 Patch Bundle includes a new install script, installbundle.sh, which guides users through the installation process. 

The patches are ordered in such a way as to process any reboots required when patching an active boot environment as near the start of the installation process as possible.  This is to facilitate System Administrators by allowing them to get over the interim reboots early in the process and kick off the final patching sequence and let the process complete. 

The screen output and logfiles produced are also designed to be as clear and self-explanatory as possible, providing both overview and drill-down capabilities.

Approximate Installation Time 

How long it will take to install the Solaris 10 05/08 Patch Bundle will depend upon a number of factors:

• The speed of the hardware and its I/O.
  

• Which Solaris 10 release is installed on the target system and what patch level the system is at.  The higher the Solaris 10 Update release or patch level, the quicker the patch bundle will apply.

• Whether Zones are installed on the system and which type of Zone.  Currently, the time to apply the cluster to each whole root non-global Zone will be approximately linear - i.e. multiple the install time by the number of whole root non-global Zones on the system.  Sparse root non-global Zones will be a little faster. (BTW: Sparse root non-global Zones is the recommended option when creating non-global Zones.)  As mentioned in a previous blog posting, there is a project in development to improve Zones patching performance.
  

For example, I installed the Solaris 10 x86 05/08 Patch Bundle on a v65x running the original Solaris 10 3/05 "FCS" (First Customer Shipment) release with no additional patch applied (worst case) and no non-global Zones.  I applied the patch bundle to the active boot environment.  Installation took a total of 3 hours and 58 minutes plus 3 reboots (see the Patch Bundle README for an explanation of the reboots when patching an active boot environment).

Conclusion

The Solaris 10 05/08 Patch Bundle will not suit everyone.  It is a large collection of patches and hence is slow to download and install.

As described in a previous blog posting, the Sun Alert patch clusters (available from the same location on SunSolve - see above) provide the minimum amount of change to address the most critical Solaris issues.  The Sun Alert cluster contains all available Solaris patch fixes for Security, Data Corruption, and System Availability issues. New versions of the Sun Alert cluster are posted whenever a new patch to fix a Sun Alert issue becomes available.  Customers should try to keep as current as possible with the contents of the Sun Alert clusters.

For customers who want to bring all their systems to the Solaris 10 05/08 (Update 5) release patch level, installing or upgrading to the Solaris 10 05/08 (Update 5) Release image remains the recommended option where feasible.  The Solaris 10 05/08 Patch Bundle was simply created in response to a demand from customers for an alternative option where upgrading was not feasible due to internal customer policies.

Since Solaris Update releases are intensely tested, the patch bundle provides a good quality patch "baseline" on which to standardize systems.

From customer feedback to date, the next Patch Bundle for the equivalent set of patches for Update 6 is likely to also be a complete set of patches from Solaris 10 3/05 "FCS" (First Customer Shipment - i.e. the original Solaris 10 release) and not an incremental bundle just containing the patch set delta between Updates 5 and 6 as I had previously suggested.  Feel free to post a comment with your preference.

Enjoy!

Posted at 12:36PM Jun 09, 2008 by Gerry Haskins in Sun  |  Comments[27]

Firmware patching

It's important to consider Firmware patches when considering an appropriate patching strategy for your systems.

In this regard, there's a new Firmware patching tab on BigAdmin which you may find useful, http://www.sun.com/bigadmin/patches/firmware/

Firmware issues are notoriously difficult to diagnose, so keeping up to date with firmware patches is a good preventative measure.  Also, firmware patches also occasionally deliver support for new features, such as LDoms (on appropriate hardware).

Due to the nature of what they deliver, firmware 'patches' are not standard patches applicable using patchadd.  Users must follow the install instructions in the patch README.

Posted at 12:46PM Jun 03, 2008 by Gerry Haskins in Sun  |  Comments[0]

Solaris 8 and Solaris 9 Kernel PatchID Sequence

As mentioned in a previous posting, the practice of patch "rejuvenation" to break out large complex patches (typically Kernel patches) into smaller, simpler components going forward has a side effect of making it difficult to follow the sequence of PatchIDs.  If you have the parent patch (e.g. an old Kernel patch), it's not obvious which child patches supercede the parent (e.g. what's the latest Kernel PatchID) as the parent isn't obsoleted by rejuvenation.  Instead, the children of the rejuvenation each specify a Requirement on the parent patch from which they were rejuvenated.

I've listed the Solaris 10 Kernel PatchID Sequence in a previous posting.  For the sake of completeness, here's the Solaris 8 and Solaris 9 Kernel PatchID Sequences (with the most current PatchID top of the list):

Solaris 8 Kernel PatchID Sequence

Solaris 9 Kernel PatchID Sequence

Posted at 10:49AM May 13, 2008 by Gerry Haskins in Sun  |  Comments[2]

Patch Management Best Practices

Please see the Patch Management Best Practices guide which my colleague, Enda O'Connor, has published on the BigAdmin Patching Hub.  I hope you'll find it useful.

Enda is a senior engineer in Patch System Test and he is far more technical than I am.

Enda has more practical experience of patching Solaris 10 Zones environments than anyone else in Sun.

Enjoy!

Posted at 07:17PM Apr 29, 2008 by Gerry Haskins in Sun  |  Comments[2]

Solaris 10 Kernel PatchID Sequence

Patch Rejuvenation

When a patch becomes complex and unwieldy, it is "Rejuvenated".   That is, no more revisions of the patch are created.  Instead, further code changes to objects contained in the patch are delivered in a series of smaller, simpler, new child PatchIDs, each of which declares a dependency upon (i.e. requires) the parent Patch.

This process is known as Patch Rejuvenation and is typically performed on the Kernel patches associated with Solaris Update releases.

Customers still need to install the large parent Patch once, but subsequent bug fixes are delivered in smaller, simpler patches.

The parent patches effectively provide "stepping-stones" to reach certain key functionality levels, with rejuvenation enabling smaller incremental change in between Update releases.

If a child patch itself becomes complex and unwieldy over time, it too may be Rejuvenated, so we end up with a "family tree" of PatchIDs providing a lineage of bug fixes for particular code areas such as the Kernel.

See http://sunsolve.sun.com/search/document.do?assetkey=1-9-86481-1 for further information.

Effect of Solaris Update "SplitGate" process on PatchIDs

Starting in February 2007, a new, improved, source code "Gate" management process was introduced for core Solaris.  This is known as the "SplitGate" process.  This replaced the old "Feature Foldback" Gate management model. "SplitGate" provides much better separation during the development process between feature code destined for release as part of a Solaris Update release and sustaining (bug fix) patches.  This addresses the problem with earlier Solaris 10 Update releases where issues with features destined for an Update release was adversely impacting the releasability of sustaining (bug fix) patches for customers in production on earlier Solaris 10 releases.

Note, as described in earlier blog postings, any change to pre-existing packages, whether as the result of new feature code or bug fixes, is always delivered in a patch.  Therefore, the Kernel patches released at the end of each Update do contain a mixture of feature and bug fix code.  What "SplitGate" provides is better separation of the feature code from bug fix patches until the Update is ready for release.

The improvement in Solaris 10 Kernel patch releasability has been dramatic:

Releasable Solaris 10 Kernel Patches using “Feature Foldback” model:

    SPARC:      21 out of 66 = 32%    (from March 2005 to February 2007)
    x86:           12 out of 66 = 18%    (from March 2005 to February 2007)

Releasable Solaris 10 Kernel Patches using “SplitGate” model:

    SPARC:      36 out of 41 =  88%    (from February 2007 to current [May 8th, 2009])
    x86:            39 out of 41 =  95%    (from February 2007 to current [May 8th, 2009])

A side effect of the "SplitGate" process, is that each Solaris 10 Update release, starting with Solaris 10 8/07 (Update 4) introduces a new set of PatchIDs which accumulate and obsolete the preceding set of PatchIDs. 

So, for example, a single new Kernel PatchID revision will appear at the end of each Solaris 10 Update release. For instance, 120011-14 (SPARC) and 120012-14 (x86) is the Kernel PatchID associated with Solaris 10 8/07 (Update 4).  Revisions -01 to -13 of this patch are not released to customers as these are purely for the interim internal builds of the Update.  Therefore, 120011-14 (SPARC) and 120012-14 (x86) are the only revisions of these PatchIDs to be released to customers.  This Kernel patch associated with the Update release is then Rejuvenated, so subsequent bug fixes will appear in a new set of PatchIDs, each of which will depend upon (i.e. require) the parent PatchID from which they were rejuvenated.

Solaris 10 Kernel Patch Lineage 

The impact of Patch Rejuvenation and the "SplitGate" process results in the following sequence of Solaris 10 Kernel PatchIDs, starting with the youngest (newest) child PatchID.  The install order of Kernel patches is starting from the bottom of the table upwards:

Posted at 12:40PM Apr 16, 2008 by Gerry Haskins in Sun  |  Comments[20]

Sun Alert Notifications

You can sign up to receive a weekly notification advising of new and updated SunAlerts .

Sun Alerts inform customers of the most critical issues affecting Sun's hardware and software.

They cover Security, Data Corruption, and System Availability issues.

Customers with a valid support contract will be able to access all Sun Alerts and patches which fix Sun Alert issues, including the Sun Alert patch clusters available on SunSolve which contain all Solaris OS patches which address Sun Alert issues.

Customer without a valid support contract will be able to access Sun Alerts and Patches only for Security related issues when they log onto SunSolve.

Posted at 04:13PM Mar 06, 2008 by Gerry Haskins in Sun  |  Comments[1]

Potentially Problematic Solaris 10 patches

As mentioned in previous blog postings, when applying patches to a live boot environment, the Solaris 'patchadd' utility may end up invoking objects which it has just patched during the installation of the remainder of the patch or patches.  This can cause the system to get into an inconsistent state during patching, as the new objects may be incompatible with objects already loaded in memory.  This is especially true in a Zones environment, where 'patchadd' calls the Zones utilties in order to patch the non-global Zone(s) after patching the Global Zone.

This wasn't a problem in Solaris 8 or Solaris 9, as the amount of code change delivered in patches was limited.  But due to the large features included in Solaris 10 Updates, this became a problem for Solaris 10.

For example, a new library may be invoked which makes a system call and passes 5 arguments, but the old Kernel is still running, not the newly applied Kernel, and if the old Kernel only expects to receive 3 arguments for the system call, problems are likely to result.

Deferred Activation Patching addresses this issue for new patches (see previous postings below), but customers must follow the Special Install Instructions listed in the patch READMEs of some older Solaris 10 patches to avoid such problems as it is not possible to retrofit Deferred Activation Patching into previously released patches.

118844 (x86) Library compatibility

When patching a Solaris 10 x86 live boot environment, Kernel patch 118844-19 or higher must be active to ensure compatibility with library changes provided in subsequent patches.

Therefore, if you are patching an old Solaris 10 x86 system which is below this Kernel patch level, you will need to reboot the system after applying 118844-19 or higher. 

For example, 118844-20 is included in the Solaris 10 x86 Recommended and Sun Alert Clusters on SunSolve to fulfill this purpose.  See the CLUSTER_README files for further information.

118844 (x86) NewBoot (GRUB)

Solaris 10 x86 Kernel patch 118844-21includes the GNU GRand Unified Bootloader (GRUB) architecture.

Please follow the appropriate system specific instructions specified in SunAlert http://sunsolve.sun.com/search/document.do?assetkey=1-26-102087-1 . 

Failure to follow these instructions might result in the system failing to boot.

118833-36 (SPARC) / 118855-36 (x86)

118833-36 (SPARC) and 118855-36 (x86) are the Solaris 10 Kernel patches associated with the Solaris 10 11/06 (Update 3) release.  They contain significant amounts of code change.

To avoid the system getting into an inconsistent state during patching, lofs loopback mounts are used in the patch's scripts to mount the original objects over the newly installed objects.  This ensures that any objects invoked by the patch utilities during patch application will be the old versions and will be consistent with processes loaded in memory.  Upon reboot, the lofs mounts get torn down, exposing the patched objects for use.  As a safety precaution, code in the patch's scripts overlay mount the patch utilities with a no-op object to ensure the system is rebooted before further patches can be applied.  This is to prevent subsequent patches from potentially patching the lofs mounted objects rather than the underlying patched objects.

This concept of utilizing lofs loopback file system mounts was later formalized as Deferred Activation Patching, which is now implemented in the Solaris patch utilities patch.  Deferred Activation Patching is described in a previous posting below.  In the Deferred Activation Patching implementation, the solution has been enhanced to enable further patches to be applied without having to first reboot.  This is achieved by applying all subsequent patches affecting the same objects in implicit Deferred Activation Patching mode (i.e. they are also applied utilitizing loopback mounts).

Customers are strongly advised to read and follow the Special Install Instructions in the README files in patches 118833-36 (SPARC) and 118855-36 (x86).  These are the most complex patches which Sun has ever released.

When is an obsoleted patch not fully obsolete ? 

Answer: When removing it from a patch set would introduce a circular dependency between the remaining patches. 

A circular dependency is where Patch A requires Patch B to be installed first and Patch B requires Patch A to be installed first.  Catch 22.  Neither patch can be installed.

The situation is pretty obvious where only 2 patches are involved, but more typically the situation can potentially arise where 3 or more patches are involved as newer patches accumulate and obsolete older patches.

For example, if Patch B requires Patch C, and Patch A requires Patch B and Patch A also obsoletes patch C, then if all three patches are present, the following is a valid install order:

   Patch C

   Patch B

   Patch A

However, if Patch C is removed from the patch set because it has been obsoleted and therefore considered no longer needed (it has been accumulated by Patch A so all dependencies on Patch C would normally be resolved by installing Patch A), then there's no valid install order as Patch A requires Patch B to be installed first and Patch B requires Patch A to be installed first.

Unfortunately, the first thing almost any patch tool does when processing patches is to discard obsolete patches.  This is true of 'patchadd -M' and most higher level patch automation tools.

Normally, this isn't a problem, as audits are in place to catch such issues during the patch creation and test processes.  Typically, if such a patch relationship exists, Patches A and B would be accumulated together into a single patch.

However, in Solaris 10, this wasn't possible with Zones patch 122660-10 (SPARC) / 122661-08 (x86) as outlined below. 

While this situation has only occurred once, and every effort will be made to prevent recurrences, there's no guarantee that similar complex patch relationships won't arise in the future.  Therefore, patch automation tools and home-grown customer patching processes may need to cope with situations where "obsoleted" patches may still need to be installed.

Zones patch 122660-10 (SPARC) / 122661-08 (x86) 

Zones patch 122660-10 (SPARC) and 122661-08 (x86) must be applied before Kernel patch 120011-14 (SPARC) and 120012-14 (x86) can be applied due to CR 6471974 "zoneadm mount mishandles shared file systems".

CR 6471974 is fixed in 122660-10 (SPARC) / 122661-08 (x86).

Customers with Zones environments where non-global zones include a non-IPD entry that references a file system shared between two boot environments are affected by CR 6471974 unless they install 122660-10 (SPARC) / 122661-08 (x86).

Such customers cannot alt mount their zones.

On SPARC, 122660-10 is also obsoleted by 120011-14 and on x86, 122661-08 is obsoleted by 120012-14.

The problem is that we need the fix for CR 6471974 in place before Kernel patch 120011-14 (SPARC) / 120012-14 (x86) can be applied to such systems.

So we need the fix which is contained in the Kernel patch which has accumulated and obsoleted the zones patch before we can add the Kernel patch.  Catch 22. 

We could not fix the problem by checking for the presence of the Zones patch in the prepatch script of Kernel patch 120011-14 (SPARC) / 120012-14 (x86) as patchadd would fail to alt mount the zones, and therefore never get as far as running the Kernel prepatch script.

Also, we could not have the Kernel patch require the Zones patch 122660-10 / 122661-08 directly as it already accumulated it.

To fix the problem, we had to take the unusual step of creating a "zones indirection" patch, 125547-02 (SPARC) and 125548-02 (x86) , which includes a prepatch script to check that the Zones patch is installed on the target system and exit gracefully if it is not.  120011-14 (SPARC) / 120012-14 (x86) require the respective "zones indirection" patch, which in turn ensures Zones patch 122660-10 (SPARC) / 122661-08 (x86) is installed.

122660-10 (SPARC) and 122661-08 (x86) are included in the Solaris 10 SPARC and x86 Recommended and Sun Alert patch clusters available from SunSolve and are automatically installed by the cluster_install script.

However, if a customer is not using the patch clusters and instead provides a patch list including 122660-10 (SPARC) or 122661-08 (x86) to 'patchadd -M' to install, it will be discarded by patchadd during its patch install ordering process as it recognizes that these patches are obsolete and hence assumes they are no longer needed.

Many higher level patch automation tools are likely to make the same assumption.  A number of them have been modified to correctly handle this situation.

If a customer encounters this issue, simply install 122660-10 (SPARC) or 122661-08 (x86) separately, and then apply the rest of the patch set as normal.

Posted at 03:57PM Mar 06, 2008 by Gerry Haskins in Sun  |  Comments[2]