Friday July 04, 2008
Friday July 04, 2008
Solaris 10 Live Upgrade Patch Bundle
In the next week or so, we'll be releasing the Solaris 10 Live Upgrade Patch Bundle, which is designed to make it simpler for customers to apply the pre-requisite patch level needed to be able to fully utilize Live Upgrade.
Applying the Solaris 10 Live Upgrade Patch Bundle will provide an alternative to following the manual steps described in SunSolve document 206844 'Solaris[TM] Live Upgrade Software: Minimum Patch Requirements' (formerly Infodoc 72099). The document is available from: http://sunsolve.sun.com/search/document.do?assetkey=1-61-206844-1
The Solaris 10 Live Upgrade Patch Bundle will be available from the normal patch cluster download center on SunSolve. ONCE IT IS RELEASED, to download the patch bundle, login to SunSolve, http://sunsolve.sun.com , click on the Patches and Updates link, click on Recommended Patch Clusters, and scroll down the window under the heading "Recommended Solaris Patch Clusters, J2SE and Java Enterprise System Clusters" to find the "Solaris 10 SPARC Live Upgrade Patch Bundle"
or "Solaris 10 x86 Live Upgrade Patch Bundle". As always, you need a valid support contract to access patch clusters. See previous postings for further information on support contracts.
Posted at 05:04PM Jul 04, 2008 by Gerry Haskins in Sun | Comments[0]
Thursday June 19, 2008
More info on patching using Live Upgrade
My colleague, Enda O'Connor, has written another useful article on Big Admin about patching using Live Upgrade, restrictions, and how-to use Live Ugrade to upgrade/patch from Solaris 8 or Solaris 9 to Solaris 10. See http://www.sun.com/bigadmin/features/articles/live_upgrade_patch.jsp
Posted at 03:27PM Jun 19, 2008 by Gerry Haskins in Sun | Comments[0]
Monday June 09, 2008
Solaris 10 05/08 (Update 5) Patch Bundle
Last week, the Solaris 10 05/08 (Update 5) Patch Bundle was released on SunSolve. The patch bundle provides another option to customers when deciding their patching strategy to maintain their Solaris systems.
What is the Solaris 10 05/08 Patch Bundle ?
The Solaris 10 05/08 Patch Bundle contains the equivalent set of patches contained in the Solaris 10 05/08 (Update 5) release image.
Why use the Solaris 10 05/08 Patch Bundle ?
The Solaris 10 05/08 Patch Bundle was created as a result of direct customer feedback after the Solaris 10 08/07 (Update 4) release. New hardware may require a specific minimum Solaris 10 Update release such as the Solaris 10 05/08 (Update 5) release. Some customers may wish to bring their other existing Solaris 10 systems up to the same patch level as the new hardware running Solaris 10 05/08. The recommended way to do this is to upgrade the existing systems to the Solaris 10 05/08 release using either regular Solaris Upgrade or Solaris Live Upgrade. But some customers may have policies in place which make it difficult to upgrade but OK to patch a system. The Solaris 10 05/08 Patch Bundle facilitates such customers to bring their existing systems up to the equivalent patch level to the Solaris 10 05/08 (Update 5) Release. In theory, this should mean that pre-existing functionality on all of the customers' systems should react the same, warts and all. This makes for a more homogeneous environment which may help lower support costs.
The Solaris 10 Update releases are very intensely tested by a wide variety of QA teams within Sun. Therefore, the functionality contained in the patches within the Solaris 10 05/08 Patch Bundle have been intensely tested as a unit through the testing performed on the Solaris 10 05/08 (Update 5) release image. Additional testing of the Solaris 10 05/08 Patch Bundle has also been performed by the Patch System Test team. Therefore, the Solaris 10 05/08 Patch Bundle provides a well tested "baseline" option on which to standardize systems.
So while the patch bundle may deliver more change than some other patching strategies, that change has been well tested as a unit and hence may actually reduce the risk of introducing regressions when compared to "dim sum" patching (i.e. choosing an arbitrary combination of patches). Note that intensive processes are also in place to ensure "dim sum" patching works, and it's rare to encounter a problem caused by "dim sum" patching.
How does the Patch Bundle differ from the Solaris 10 05/08 (Update 5) release image ?
The Solaris 10 05/08 (Update 5) release is a complete Solaris release image. It contains new packages to support new features in the Solaris 10 05/08 (Update 5) release as well as all Solaris patches which were available when the Update was built. The patches are pre-applied into the Solaris 10 05/08 release image. This means that one doesn't have to spend time adding the patches using 'patchadd'. On the flipside, since the patches are pre-applied into the release image, they cannot be backed out using 'patchrm'. This isn't generally a problem as the Solaris Update release images are very intensely tested. One can do a fresh install of the Solaris 10 05/08 (Update 5) release, or upgrade to it from an earlier Solaris release.
The Solaris 10 05/08 Patch Bundle contains the equivalent set of patches to the Solaris 10 05/08 (Update 5) release. The patch bundle does not include the new packages contained in the Solaris 10 05/08 (Update 5) release. Therefore, new features in Update 5 which depend upon new packages introduced in that release will not be available in the patch bundle. However, as discussed in a previous blog entry, any change to pre-existing code is delivered in a patch. This includes features as well as bug fixes. Therefore some feature enhancements will be available in the patch bundle. ZFS, for example, is typically self-contained in patches and hence ZFS enhancements will typically be available via the patch bundle as well as via the Update release image. So will most Zones enhancements. The patch bundle is simply a collection of patches with an install order file (patch_order) and an install script wrapper (installbundle.sh) around 'patchadd'. Patches in the patch bundle can be backed out using 'patchrm', so long as the '-d' (no save) option wasn't used when applying the patch bundle.
There are a number of "special" or "script" patches included in each Solaris Update release. These patches are used to correct issues in how patches are pre-applied into the Solaris Update release image and have no purpose whatsoever outside of the Solaris Update release process. Therefore, these "special" or "script" patches are not released to SunSolve and are not included in the patch bundle. See the Solaris 10 05/08 Patch Bundle README file for further information on these and other minor differences between the patch set pre-applied in the Solaris 10 05/08 release image and the patch set included in the Solaris 10 05/08 Patch Bundle.
Access
The Solaris 10 05/08 Patch Bundle is available from the usual patch cluster location.
As with all patch clusters, you need a valid support contract (Solaris Subscription, Sun System Service Plan (Sun Spectrum), or Enterprise Support Plan) to download the cluster.
Log onto Sunsolve, click on Patches and Updates, then Recommended Patch Clusters and scroll down the box under "Recommended Solaris Patch Clusters, J2SE and Java Enterprise System Clusters" to the Solaris 10 SPARC 05/08 Patch Bundle and Solaris 10 x86 05/08 Patch Bundle entries.
The cluster is chunked to aid download. There are 2 chunks for x86 and 3 chunks for SPARC.
Follow the download instructions to the right of the scroll-down box or read the README file for any chunk.
Installation
Read the Patch Bundle README file for full installation instructions.
The patch bundle can be installed either on the active boot environment (i.e. the live system) or an inactive boot environment.
Patching an inactive boot environment is recommended as, depending on the starting patch level of the target system, it may involve less system downtime as only a single reboot is required at the end to activate the boot environment.
If you patch the active boot environment (i.e. the live system), then depending on the starting patch level of the target system, you may need to reboot an x86 system up to three times (twice at specific points during the installation process and once at the end) and a SPARC system up to two times (once after installing Kernel patch 118833-36 and once at the end). See the patch bundle README for details.
The Solaris 10 05/08 Patch Bundle includes a new install script, installbundle.sh, which guides users through the installation process.
The patches are ordered in such a way as to process any reboots required when patching an active boot environment as near the start of the installation process as possible. This is to facilitate System Administrators by allowing them to get over the interim reboots early in the process and kick off the final patching sequence and let the process complete.
The screen output and logfiles produced are also designed to be as clear and self-explanatory as possible, providing both overview and drill-down capabilities.
Approximate Installation Time
How long it will take to install the Solaris 10 05/08 Patch Bundle will depend upon a number of factors:
- The speed of the hardware and its I/O.
- Which Solaris 10 release is installed on the target system and what patch level the system is at. The higher the Solaris 10 Update release or patch level, the quicker the patch bundle will apply.
- Whether Zones are installed on the system and which type of Zone. Currently, the time to apply the cluster to each whole root non-global Zone will be approximately linear - i.e. multiple the install time by the number of whole root non-global Zones on the system. Sparse root non-global Zones will be a little faster. (BTW: Sparse root non-global Zones is the recommended option when creating non-global Zones.) As mentioned in a previous blog posting, there is a project in development to improve Zones patching performance.
For example, I installed the Solaris 10 x86 05/08 Patch Bundle on a v65x running the original Solaris 10 3/05 "FCS" (First Customer Shipment) release with no additional patch applied (worst case) and no non-global Zones. I applied the patch bundle to the active boot environment. Installation took a total of 3 hours and 58 minutes plus 3 reboots (see the Patch Bundle README for an explanation of the reboots when patching an active boot environment).
Conclusion
The Solaris 10 05/08 Patch Bundle will not suit everyone. It is a large collection of patches and hence is slow to download and install.
As described in a previous blog posting, the Sun Alert patch clusters (available from the same location on SunSolve - see above) provide the minimum amount of change to address the most critical Solaris issues. The Sun Alert cluster contains all available Solaris patch fixes for Security, Data Corruption, and System Availability issues. New versions of the Sun Alert cluster are posted whenever a new patch to fix a Sun Alert issue becomes available. Customers should try to keep as current as possible with the contents of the Sun Alert clusters.
For customers who want to bring all their systems to the Solaris 10 05/08 (Update 5) release patch level, installing or upgrading to the Solaris 10 05/08 (Update 5) Release image remains the recommended option where feasible. The Solaris 10 05/08 Patch Bundle was simply created in response to a demand from customers for an alternative option where upgrading was not feasible due to internal customer policies.
Since Solaris Update releases are intensely tested, the patch bundle provides a good quality patch "baseline" on which to standardize systems.
From customer feedback to date, the next Patch Bundle for the equivalent set of patches for Update 6 is likely to also be a complete set of patches from Solaris 10 3/05 "FCS" (First Customer Shipment - i.e. the original Solaris 10 release) and not an incremental bundle just containing the patch set delta between Updates 5 and 6 as I had previously suggested. Feel free to post a comment with your preference.
Enjoy!
Posted at 12:36PM Jun 09, 2008 by Gerry Haskins in Sun | Comments[7]
Tuesday June 03, 2008
Firmware patching
It's important to consider Firmware patches when considering an appropriate patching strategy for your systems.
In this regard, there's a new Firmware patching tab on BigAdmin which you may find useful, http://www.sun.com/bigadmin/patches/firmware/
Firmware issues are notoriously difficult to diagnose, so keeping up to date with firmware patches is a good preventative measure. Also, firmware patches also occasionally deliver support for new features, such as LDoms (on appropriate hardware).
Due to the nature of what they deliver, firmware 'patches' are not standard patches applicable using patchadd. Users must follow the install instructions in the patch README.
Posted at 12:46PM Jun 03, 2008 by Gerry Haskins in Sun | Comments[0]
Tuesday May 13, 2008
Solaris 8 and Solaris 9 Kernel PatchID Sequence
As mentioned in a previous posting, the practice of patch "rejuvenation" to break out large complex patches (typically Kernel patches) into smaller, simpler components going forward has a side effect of making it difficult to follow the sequence of PatchIDs. If you have the parent patch (e.g. an old Kernel patch), it's not obvious which child patches supercede the parent (e.g. what's the latest Kernel PatchID) as the parent isn't obsoleted by rejuvenation. Instead, the children of the rejuvenation each specify a Requirement on the parent patch from which they were rejuvenated.
I've listed the Solaris 10 Kernel PatchID Sequence in a previous posting. For the sake of completeness, here's the Solaris 8 and Solaris 9 Kernel PatchID Sequences (with the most current PatchID top of the list):
Solaris 8 Kernel PatchID Sequence
| SPARC | x86 |
| 117350-01 to -xx | 117351-01 to -xx |
| requires | requires |
| 117000-01 to -05 | 117001-01 to -05 |
| requires | requires |
| 108528-01 to -29 | 108529-01 to 29 |
Solaris 9 Kernel PatchID Sequence
| SPARC | x86 |
| 122300-02 to -xx | 122301-02 to -xx |
| requires | requires |
| 118558-01 to -39 | 118559-01 to -39 |
| requires | requires |
| 117171-01 to -17 | 117172-17 only |
| requires | obsoletes |
| 112233-01 to -12 | 112234-04 to -11 |
Posted at 10:49AM May 13, 2008 by Gerry Haskins in Sun | Comments[0]
Tuesday April 29, 2008
Patch Management Best Practices
Please see the Patch Management Best Practices guide which my colleague, Enda O'Connor, has published on the BigAdmin Patching Hub. I hope you'll find it useful.
Enda is a senior engineer in Patch System Test and he is far more technical than I am.
Enda has more practical experience of patching Solaris 10 Zones environments than anyone else in Sun.
Enjoy!
Posted at 07:17PM Apr 29, 2008 by Gerry Haskins in Sun | Comments[2]
Wednesday April 16, 2008
Solaris 10 Kernel PatchID Sequence
Patch Rejuvenation
When a patch becomes complex and unwieldy, it is "Rejuvenated". That is, no more revisions of the patch are created. Instead, further code changes to objects contained in the patch are delivered in a series of smaller, simpler, new child PatchIDs, each of which declares a dependency upon (i.e. requires) the parent Patch.
This process is known as Patch Rejuvenation and is typically performed on the Kernel patches associated with Solaris Update releases.
Customers still need to install the large parent Patch once, but subsequent bug fixes are delivered in smaller, simpler patches.
The parent patches effectively provide "stepping-stones" to reach certain key functionality levels, with rejuvenation enabling smaller incremental change in between Update releases.
If a child patch itself becomes complex and unwieldy over time, it too may be Rejuvenated, so we end up with a "family tree" of PatchIDs providing a lineage of bug fixes for particular code areas such as the Kernel.
See http://sunsolve.sun.com/search/document.do?assetkey=1-9-86481-1 for further information.
Effect of Solaris Update "SplitGate" process on PatchIDs
Starting in February 2007, a new, improved, source code "Gate" management process was introduced for core Solaris. This is known as the "SplitGate" process. This replaced the old "Feature Foldback" Gate management model. "SplitGate" provides much better separation during the development process between feature code destined for release as part of a Solaris Update release and sustaining (bug fix) patches. This addresses the problem with earlier Solaris 10 Update releases where issues with features destined for an Update release was adversely impacting the releasability of sustaining (bug fix) patches for customers in production on earlier Solaris 10 releases.
Note, as described in earlier blog postings, any change to pre-existing code, whether as the result of new feature code or bug fixes, is always delivered in a patch. Therefore, the Kernel patches released at the end of each Update do contain a mixture of feature and bug fix code. What "SplitGate" provides is better separation of the feature code from bug fix patches until the Update is ready for release.
The improvement in Solaris 10 Kernel patch releasability has been dramatic:
Releasable Solaris 10 Kernel Patches using “Feature Foldback” model:
SPARC: 21 out of 66 = 32% (from March 2005 to February 2007)
x86: 12 out of 66 = 18% (from March 2005 to February 2007)
Releasable Solaris 10 Kernel Patches using “SplitGate” model:
SPARC: 18 out of 22 = 82% (from February 2007 to current [April 16th 2008])
x86: 21 out of 22 = 95% (from February 2007 to current [April 16th 2008])
A side effect of the "SplitGate" process, is that each Solaris 10 Update release, starting with Solaris 10 8/07 (Update 4) introduces a new set of PatchIDs which accumulate and obsolete the preceding set of PatchIDs.
So, for example, a single new Kernel PatchID revision will appear at the end of each Solaris 10 Update release. For instance, 120011-14 (SPARC) and 120012-14 (x86) is the Kernel PatchID associated with Solaris 10 8/07 (Update 4). Revisions -01 to -13 of this patch are not released to customers as these are purely for the interim internal builds of the Update. The Kernel patch associated with the Update release is then Rejuvenated, so subsequent bug fixes will appear in a new set of PatchIDs, each of which will depend upon (i.e. require) the parent PatchID from which they were rejuvenated. Therefore, 120011-14 (SPARC) and 120012-14 (x86) are the only revisions of these PatchIDs to be released to customers.
Solaris 10 Kernel Patch Lineage
The impact of Patch Rejuvenation and the "SplitGate" process results in the following sequence of Solaris 10 Kernel PatchIDs, starting with the youngest (newest) child PatchID:
| Solaris 10 SPARC Kernel PatchIDs |
Description |
Solaris 10 x86 Kernel PatchIDs |
| 137137-xx only |
Solaris 10 Update 6 Kernel PatchID |
137138-xx only |
| 137111-01 to 137111-xx |
Sustaining Bug Fixes |
137112-01 to 137112-xx |
| 127127-11 only |
Solaris 10 5/08 (Update 5) Kernel PatchID |
127128-11 only |
| 127111-01 to 127111-11 |
Sustaining Bug Fixes |
127112-01 to 127112-11 |
| 120011-14 only |
Solaris 10 8/07 (Update 4) Kernel PatchID |
120012-14 only |
| 125100-04 to 125100-10 |
Sustaining Bug Fixes |
125101-01 to 125101-10 |
| 118833-02 to 118833-36 |
118833-33 (SPARC) / 118855-33 (x86) is the Kernel patch included in Solaris 10 11/06 (Update 3) but these patches were not releasable as "standalone" patches to SunSolve. 118833-17 (SPARC) / 118855-14 (x86) is the Kernel patch included in Solaris 10 6/06 (Update 2). 118855-14 was not releasable as a "standalone" patch to SunSolve. |
118855-01 to 118855-36 |
| 118822-01 to 118822-30 |
118822-25 (SPARC) / 118844-26 (x86) is the Kernel patch included in Solaris 10 1/06 (Update 1). 118844-26 was not releasable as a "standalone" patch to SunSolve. |
118844-01 to 118844-30 |
Posted at 12:40PM Apr 16, 2008 by Gerry Haskins in Sun | Comments[16]
Thursday March 06, 2008
Sun Alert Notifications
You can sign up to receive a weekly notification advising of new and updated SunAlerts .
Sun Alerts inform customers of the most critical issues affecting Sun's hardware and software.
They cover Security, Data Corruption, and System Availability issues.
Customers with a valid support contract will be able to access all Sun Alerts and patches which fix Sun Alert issues, including the Sun Alert patch clusters available on SunSolve which contain all Solaris OS patches which address Sun Alert issues.
Customer without a valid support contract will be able to access Sun Alerts and Patches only for Security related issues when they log onto SunSolve.
Posted at 04:13PM Mar 06, 2008 by Gerry Haskins in Sun | Comments[1]
Potentially Problematic Solaris 10 patches
As mentioned in previous blog postings, when applying patches to a live boot environment, the Solaris 'patchadd' utility may end up invoking objects which it has just patched during the installation of the remainder of the patch or patches. This can cause the system to get into an inconsistent state during patching, as the new objects may be incompatible with objects already loaded in memory. This is especially true in a Zones environment, where 'patchadd'
calls the Zones utilties in order to patch the non-global Zone(s) after
patching the Global Zone.
This wasn't a problem in Solaris 8 or Solaris 9, as the amount of code change delivered in patches was limited. But due to the large features included in Solaris 10 Updates, this became a problem for Solaris 10.
For example, a new library may be invoked which makes a system call and passes 5 arguments, but the old Kernel is still running, not the newly applied Kernel, and if the old Kernel only expects to receive 3 arguments for the system call, problems are likely to result.
Deferred Activation Patching addresses this issue for new
patches (see previous postings below), but customers must follow the Special Install Instructions
listed in the patch READMEs of some older Solaris 10 patches to avoid
such problems as it is not possible to retrofit Deferred Activation Patching into previously released patches.
118844 (x86) Library compatibility
When patching a Solaris 10 x86 live boot environment, Kernel patch 118844-19 or higher must be active to ensure compatibility with library changes provided in subsequent patches.
Therefore, if you are patching an old Solaris 10 x86 system which is below this Kernel patch level, you will need to reboot the system after applying 118844-19 or higher.
For example, 118844-20 is included in the Solaris 10 x86 Recommended and Sun Alert Clusters on SunSolve to fulfill this purpose. See the CLUSTER_README files for further information.
118844 (x86) NewBoot (GRUB)
Solaris 10 x86 Kernel patch 118844-21includes the GNU GRand Unified Bootloader (GRUB) architecture.
Please follow the appropriate system specific instructions specified in SunAlert http://sunsolve.sun.com/search/document.do?assetkey=1-26-102087-1 .
Failure to follow these instructions might result in the system failing to boot.
118833-36 (SPARC) / 118855-36 (x86)
118833-36 (SPARC) and 118855-36 (x86) are the Solaris 10 Kernel patches associated with the Solaris 10 11/06 (Update 3) release. They contain significant amounts of code change.
To avoid the system getting into an inconsistent state during patching, lofs loopback mounts are used in the patch's scripts to mount the original objects over the newly installed objects. This ensures that any objects invoked by the patch utilities during patch application will be the old versions and will be consistent with processes loaded in memory. Upon reboot, the lofs mounts get torn down, exposing the patched objects for use. As a safety precaution, code in the patch's scripts overlay mount the patch utilities with a no-op object to ensure the system is rebooted before further patches can be applied. This is to prevent subsequent patches from potentially patching the lofs mounted objects rather than the underlying patched objects.
This concept of utilizing lofs loopback file system mounts was later formalized as Deferred Activation Patching, which is now implemented in the Solaris patch utilities patch. Deferred Activation Patching is described in a previous posting below. In the Deferred Activation Patching implementation, the solution has been enhanced to enable further patches to be applied without having to first reboot. This is achieved by applying all subsequent patches affecting the same objects in implicit Deferred Activation Patching mode (i.e. they are also applied utilitizing loopback mounts).
Customers are strongly advised to read and follow the Special Install Instructions in the README files in patches 118833-36 (SPARC) and 118855-36 (x86). These are the most complex patches which Sun has ever released.
When is an obsoleted patch not fully obsolete ?
Answer: When removing it from a patch set would introduce a circular dependency between the remaining patches.
A circular dependency is where Patch A requires Patch B to be installed first and Patch B requires Patch A to be installed first. Catch 22. Neither patch can be installed.
The situation is pretty obvious where only 2 patches are involved, but more typically the situation can potentially arise where 3 or more patches are involved as newer patches accumulate and obsolete older patches.
For example, if Patch B requires Patch C, and Patch A requires Patch B and Patch A also obsoletes patch C, then if all three patches are present, the following is a valid install order:
Patch C
Patch B
Patch A
However, if Patch C is removed from the patch set because it has been obsoleted and therefore considered no longer needed (it has been accumulated by Patch A so all dependencies on Patch C would normally be resolved by installing Patch A), then there's no valid install order as Patch A requires Patch B to be installed first and Patch B requires Patch A to be installed first.
Unfortunately, the first thing almost any patch tool does when processing patches is to discard obsolete patches. This is true of 'patchadd -M' and most higher level patch automation tools.
Normally, this isn't a problem, as audits are in place to catch such issues during the patch creation and test processes. Typically, if such a patch relationship exists, Patches A and B would be accumulated together into a single patch.
However, in Solaris 10, this wasn't possible with Zones patch 122660-10 (SPARC) / 122661-08 (x86) as outlined below.
While this situation has only occurred once, and every effort will be made to prevent recurrences, there's no guarantee that similar complex patch relationships won't arise in the future. Therefore, patch automation tools and home-grown customer patching processes may need to cope with situations where "obsoleted" patches may still need to be installed.
Zones patch 122660-10 (SPARC) / 122661-08 (x86)
Zones patch 122660-10 (SPARC) and 122661-08 (x86) must be applied before Kernel patch 120011-14 (SPARC) and
120012-14 (x86) can be applied due to CR 6471974 "zoneadm mount mishandles shared file systems".
CR 6471974 is fixed in 122660-10 (SPARC) / 122661-08 (x86).
Customers with Zones environments
where non-global zones include a non-IPD entry that references a file
system shared between two boot environments are affected by CR 6471974 unless they install 122660-10 (SPARC) / 122661-08 (x86).
Such customers cannot alt mount their zones.
On SPARC, 122660-10 is also obsoleted by 120011-14 and on x86, 122661-08 is obsoleted by 120012-14.
The problem is that we need the fix for CR 6471974 in place before Kernel patch 120011-14 (SPARC) / 120012-14 (x86) can be applied to such systems.
So we need the fix which is contained in the Kernel patch which has accumulated and obsoleted the zones patch before we can add the Kernel patch. Catch 22.
We could not fix the problem by checking for the presence of the Zones patch in the prepatch script of Kernel patch 120011-14 (SPARC) / 120012-14 (x86) as patchadd would fail to alt mount the zones, and therefore never get as far as running the Kernel prepatch script.
Also, we could not have the Kernel patch require the Zones patch 122660-10 / 122661-08 directly as it already accumulated it.
To fix the problem, we had to take the unusual step of creating a "zones indirection"
patch, 125547-02 (SPARC) and 125548-02 (x86) , which includes a prepatch script to check that the Zones patch is installed on the target system and exit gracefully if it is not. 120011-14 (SPARC) / 120012-14 (x86) require the respective "zones indirection" patch, which in turn ensures Zones patch 122660-10 (SPARC) / 122661-08 (x86) is installed.
122660-10 (SPARC) and 122661-08 (x86) are included in the Solaris 10 SPARC and x86 Recommended and Sun Alert patch clusters available from SunSolve and are automatically installed by the cluster_install script.
However, if a customer is not using the patch clusters and instead provides a patch list including 122660-10 (SPARC) or 122661-08 (x86) to 'patchadd -M' to install, it will be discarded by patchadd during its patch install ordering process as it recognizes that these patches are obsolete and hence assumes they are no longer needed.
Many higher level patch automation tools are likely to make the same assumption. A number of them have been modified to correctly handle this situation.
If a customer encounters this issue, simply install 122660-10 (SPARC) or 122661-08 (x86) separately, and then apply the rest of the patch set as normal.
Posted at 03:57PM Mar 06, 2008 by Gerry Haskins in Sun | Comments[2]
Tuesday January 22, 2008
Patch Automation Tools
First of all, let me say that my personnel experience of Sun's patch automation tools is limited. I work upstream from the SysNet and Services groups who produce most of Sun's patch automation tools, so I and my team mostly patch from first principles using the basic Solaris patch utilities, patchadd and patchrm.
My
team does have some experience of working with some of the patch
automation tools. I've supplemented this with information from SysNet
and Services folk.
Sun Connection 1.1.1 Satellite and xVM Ops Center 1.0
The official Sun patching tool of choice is now Sun Connection 1.1.1 Satellite Edition. An enhanced version is incorporated in xVM Ops Center 1.0.
Sun acquired Aduva a couple of years ago. Aduva has a track record of providing patch and update automation tools for multiple Operating Systems.
The next-generation Aduva-based tools are starting to come on stream. Sun Connection 1.1.1 Satellite is based on Aduva. Note, "Satellite" has a completely different back end to the Sun Update Connection Hosted edition and Solaris Update Manager, which are based on PatchPro (see below).
I'm hearing good things about the Satellite. I understand that its initial target market is customers with 50+ systems.
Sun Connection 1.1.1 Satellite Edition
is based on Aduva Onstage and Update Connection Enterprise. A
central server (Satellite) at the customer site is used to analyze and
update all attached client systems in a fully automated manner. It
builds upon a central Knowledge Database fed by Sun. It covers the
provisioning of patches, packages, config files and scripts.
It is available to customers who pay for it.
Sun Connection 1.1.1 Satellite provides a
solution for customers primarily interested in patch and package provisioning.
Sun Connection Satellite will be a component of the recently announced xVM Ops Center 1.0.
xVM Ops Center 1.0 is a merge of Sun Connection and N1SM.
For further information, please see the Sun Connection hub's Product Tour page on BigAdmin.
EIS
EIS stands for Enterprise Installation Standards and originated from Sun field personnel wanting to develop best practice installation standards for systems installed on customer sites.
EIS has proved extremely popular with Sun field personnel and approved partners. It's widespread adoption was due to it successfully addressing a real need. I view it's widespread adoption among field personnel and OEMs as proof positive of its efficacy.
The EIS patch baseline goes through QA testing prior
to release.
The images installed by Sun's manufacturers on servers are also based
upon the
EIS patch baseline. Additional testing by Sun's manufacturers plus
feedback from the EIS
user community raises confidence in the EIS patch baseline content
further. Since many system installations
world-wide use the EIS
methodology, any
inherent problems will quickly appear and can be dealt with. In the
event of
there being issues with the EIS patch baseline recommendations are
communicated to the
EIS community.
This same EIS set of patches which are considered by Sun Field
Engineers as best practice to install on a new system, can also be used
to patch existing systems to the same patch level. The EIS set of
patches is based on the Recommended Patch Cluster for the Solaris OS with additional
patches included by the Field Engineers for additional products and to
address irritating issues which do not meet the criteria for inclusion
in the Recommended Patch Cluster.
The EIS patch baseline covers Solaris and other products such
as SunCluster, SunVTS,
SSP, SMS, QFS, SAM-FS, and includes patches which provide firmware
updates.
EIS has traditionally only been available via Sun Services personnel but is now available direct to customers via Sun Connection Satellite. This provides a good option to customers to patch to a defined and tested patch baseline. See the Sun Connection blog for further information.
pca
pca is a popular 3rd party tool developed by Martin Paul. I've only ever heard positive feedback about pca.
pca is available from http://www.par.univie.ac.at/solaris/pca/
To try out pca, just run this on any Solaris machine:
$ wget http://www.par.univie.ac.at/solaris/pca/pca
$ chmod +x pca
$ ./pca
pca is a good solution for customers interested in a simple, easy to use, patch automation tool.
smpatch, Update Manager, and Sun Connection Hosted Edition
smpatch is a command line tool and part of Solaris. It
allows customers to analyze and update Solaris with current patches.
For customers without a valid support contract, only security and
driver patches are available. For customers with a valid support
contract, all patches are available.
updatemanager is a GUI wrapper around smpatch and is also
part of Solaris. It can be used to see what patches/updates are
available and to easily select the patches, which the customer wants to
install. Again, for customers without a valid support contract, only
security and driver patches are available. For customers with a valid
support contract, all patches are available.
Sun Connection - Hosted Edition
is the internet portal version of updatemanager. The customer can register
all their servers and
can schedule and review the installation of patches from a central portal.
This is only available to customers who pay for it.
The above tools rely
on the "PatchPro" analysis engine to recommend patches to
customers.
The vast majority of Realizations simply associate a patch to packages installed on the target system, in the same way patchadd determines whether or not to apply a patch. That is, if the package name, package version, and platform architecture in the pkginfo file(s) in the patch match at least one package name, package version, and platform architecture on the target system, the patch can be applied, else not.
Errors in writing Realization Detectors cause patch
automation tools which utilize the PatchPro analysis engine to
occasionally recommend inappropriate patches. This has impacted the reliability of PatchPro based tools.
Work is underway to write a generic realization detector to match patches to packages. This will save patch creators from writing their own realization detectors for the common case, simplifying the process and reducing error opportunity. Patch creators will still be able to write specific Realization Detectors where necessary.
See Instructions for Getting Started with Sun Connection's Update Manager and Sun Hosted solutions and Patch Manager 2.0 FAQs for further information.
TLP
TLP stands for Traffic Light Patching, and is another tool which was developed by Sun Service folk for Sun Service folk to address the need for Patch Automation.
TLP is not directly available to customers. It's used by Sun Service personnel to determine the appropriate patches to be installed on a customer's system, including things like firmware patches.
TLP has a modular design. It utilizes the concept of a "baseline" of patches chosen by the user, from the Recommended Patch Set, to the EIS patch set, to a user defined set of patches. TLP allows a number of different patch analysis engines to be used to determine which patches from the "baseline" to apply to a particular target system.
TLP is popular with customers who use it, as it's reliable and works well.
TLP was End-Of-Lifed (EOL'd) in September of 2006 and reached End-Of-Service-Life (EOSL) in December 2007. However, a number of customers have been given extensions on TLP support for transition purposes.
Sun Services Patch Recommendations
Most European countries provide a service where customers can submit Explorer logs and the local Sun office provides back a patch bundle. These services may use SRAS and TLP in the background.
Please contact your local Sun Services office for further details.
I believe the plan will be to consolidate these services into a consistent official worldwide service.
Posted at 05:39PM Jan 22, 2008 by Gerry Haskins in Sun | Comments[51]
Monday January 21, 2008
Patch Access Entitlement
Sun changed its Software business strategy a few years ago.
Customer's used to have to buy most Sun Software while most support, such as patch access, was free.
Now, Sun's Software frequently uses a model similar to many Linux vendors, where the software is often given away for free, but a customer must pay for most support.
From the perspective of accessing patches, patches which address Security issues remain free. So do patches which provide new hardware drivers.
Customers must have a valid support contract to access most other Solaris patches, including the Solaris patch clusters such as the Recommended Patch Cluster or Sun Alert Patch Cluster.
See Solaris Subscriptions or Sun Spectrum Service Plans for information on how to buy a Solaris Subscription (Solaris only) or Sun Spectrum (entire system) support contract. These support contracts entitle customers to access all patches, plus a wide range of additional support services. See also http://www.sun.com/service/subscriptions/index.jsp
The infrastructure behind SunSolve is in transition at the moment. Customer's may notice changes to patch entitlement as these changes are rolled out.
Additional support services for customers with support contracts
will continue to be expanded and enhanced over the coming months.
The changes are largely being implemented by the SysNet and Services teams. I have relatively little insight into the specific changes and timelines. Please use your normal support channels to get further information.
The roll-out of patch entitlement changes has been somewhat
patchy (excuse the pun), but the general direction of stricter patch
access entitlement is likely to continue.
For example, to see which Solaris patches are currently free, go to the SunSolve patch page http://sunsolve.sun.com/show.do?target=patchpage , and where it says:-
Product Patches
Obtain latest product patch bundle
Software
» Solaris
...use the pull-down menu to select the relevant Solaris version.
The resultant list shows which patches are free or not using a key symbol. You will only see this key symbol if you are not logged in as a user with a valid support contract, since nothing is locked for you if you have a valid support contract.
Posted at 12:55PM Jan 21, 2008 by Gerry Haskins in Sun | Comments[2]
Thursday January 10, 2008
Patching Strategies
As mentioned in my initial posting, there isn't a "one size fits all" patching strategy for all customers to use in all circumstances.
Perhaps the most common question which customers ask is, "What patches should I apply to my system ?"
The answer, unfortunately, is "It depends."
Many factors determine what patching strategy is appropriate for a particular system. These may include:
- Risk profile of the customer. For example, Financial institutions tend to be very risk adverse. Their change control processes can be onerous.
- Criticality of the system. Is it Life Critical, Mission Critical, Business Critical, or relatively expendable ?
- Risk profile of the system. For example, is it behind a firewall, is it vulnerable to Denial Of Service attacks, etc. ?
- Cost of planned downtime (for proactive patching and maintenance) versus the cost of unplanned downtime (for reactive break-and-fix patching and maintenance).
- Available Maintenance Windows
- Upgrade strategy - Is the customer still on older versions such as Solaris 8 or Solaris 9 or is there a desire to leverage some of the cool new software features (e.g. Containers (Zones), ZFS, DTrace, etc.) or support for cool new hardware available in Solaris 10 ?
- Desire to keep a relatively homogeneous Operating Environment across similar servers
- etc.
While I can discuss some evolving thoughts on patching strategies here, please note that Sun Services offer comprehensive solutions tailored for the needs of specific customers. The thoughts expressed here are not a substitute for careful analysis of the specific needs of individual customers.
Minimizing Risk
Risk minimization is a key consideration for many customers when deciding on a patching strategy.
Change implies risk.
There are industry studies which show that for every x number of bug fixes or lines of code changed, a new bug or regression is introduced.
One might logically conclude therefore, that the more change that is applied to a system, the more the risk of introducing a regression. Hence, one might conclude that applying the minimum number of patches and hence the minimum amount of change would minimize risk.
But that's just one factor.
It's also important to consider the test coverage of the various change delivery options. This includes test coverage by Sun as well as test coverage by the customer, channel partner, or other vendor.
Always install the latest patch utilities patch first
Always install the latest version of the Solaris patch
utilities patch before installing any other patches. This is important to ensure that
you have all the latest fixes to the patch utilities. The patch utility patches are
always contained in the Solaris Recommended and SunAlert patch clusters and are
always installed first, along with any patches which they themselves require.
The patch utilities patches are currently:
Solaris 10 SPARC: 119254
Solaris 10 x86: 119255
Solaris 9 SPARC: 112951
Solaris 9 x86: 114194
Solaris 8 SPARC: 110380
Solaris 8 x86: 110403
Depending on the OS version, several other patches may be required to avoid issues which can impact correct patch application. Such patches are listed in the "Latest patch updates" section on the SunSolve home page.
Solaris Patch Management: Recommended Strategy
The Solaris Patch Management: Recommended Strategy available from http://docs.sun.com and linked off the SunSolve "Patches and Updates" page is a good starting point.
Perhaps surprisingly, it shows from an analysis of customer Explorer data that the more patches which are applied to a system, the more downtime that system will experience. This is largely because, as discussed in the preceding posting, a number of patches require downtime in order to be installed on a live boot environment.
However, in many cases the cost of unplanned downtime to fix issues is much, much higher than the cost of planned downtime to facilitate preventative patching to prevent issues from occurring in the first place.
The trick is to know which patches are likely to prevent issues on a particular system.
Recommended and Sun Alert Patch Clusters
When deciding what patches to apply to Solaris, the Recommended and the Sun Alert Patch Clusters, which are available from SunSolve to customers with a valid support contract, are a good starting point. They provide:- The latest revision of patch utilities patch
- Solaris patches which address Sun Alert issues - that is, patches which address Security, Data Corruption, or System Availability issues.
- Any patch which is required by either of the above.
The main difference between the Recommended Patch Cluster and the newer Sun Alert Patch Cluster is that the Sun Alert Patch Cluster contains the lowest revision of patches which address Sun Alert issues while the Recommended Patch Cluster contains the latest available revision of such patches. Both are good options.
Note, the Recommended and Sun Alert Clusters only contain patches for the Solaris OS. They do not contain patches for middleware or application layer products such as Java ES, SunStudio, etc.
Both the Recommended and the Sun Alert Patch Clusters come with an install_cluster script and a patch_order file listing the order in which the patches are to be installed. See the Cluster README files linked off the "Patches and Updates" page on SunSolve for further information. (On http://sunsolve.sun.com/show.do?target=patches/patch-access , "Solaris 10 x86" is the Solaris 10 x86 Recommended Cluster and "Solaris 10 x86 Sun Alert Patch Cluster" is self-explanatory.)
Applying the Solaris Recommended or Sun Alert patch cluster at each available maintenance window, plus any patches for fixes for bugs which you as a customer have filed yourself, is a good approach to proactive patching.
In between maintenance windows, monitor new Sun Alerts which are issued and determine whether your systems are vulnerable to the issue. If the risk of the issue occurring is low or the consequences of the potential problem manageable, you may decide that it's OK to wait until the next maintenance window before applying the patch or taking whatever other action is recommended in the Sun Alert. If the risk of the issue occurring is high and the potential problem severe, consider applying the patch or taking whatever other action is recommended in the Sun Alert as soon as possible.
Apart from these Solaris patch clusters, other patch clusters are available on http://sunsolve.sun.com/show.do?target=patches/patch-access for other products, such as J2SE and Java ES.Installing or Upgrading to the latest Solaris Update Release
Each bi-weekly build of the next Solaris Marketing Release ( "Nevada" ) and the next Solaris 10 Update Release is intensely tested by a large number of test teams throughout Sun. Each team has a particular focus, from functional testing of new features, regression testing of pre-existing features, performance improvement testing (each release should be faster than the last), new hardware testing, hardware regression testing, Desktop, Globalization, Accessibility, SunCluster, Java Enterprise System, patch testing, etc.
Due to the intensive testing of Update releases, installing or upgrading a system to the latest available Update Release should be seriously considered by customers wishing to minimize risk.
While each Update release contains significant amount of code change, it has been intensely tested as a unit. As previously mentioned, each Update contains all the available bug fixes at the time it was built. Therefore, pre-existing functionality should be more stable and more performant in each successive Update. The latest available Update should therefore provide a good stable baseline for customers.
For example, Solaris 10 8/07 (Update 4) not only introduces cool new software features and support for cool new hardware, it also contains many fixes and enhancements to pre-existing functionality such as Containers (Zones), such as the ability to Upgrade Zones, significantly improving the maintainability of Zones.
The complexities of patching the live boot environment of a pre-Update 4 Zones systems can be avoided by upgrading to Update 4 instead.
"Dim Sum" Patching
As previously mentioned, new bug fixes as well as
features "soak" in the next Marketing Release of Solaris under
development (currently "Nevada"
to shake out any bugs in the code, before the code changes
are allowed to be put back into an older release of Solaris for
inclusion in a patch and, if the patch is for Solaris 10, included in the next Update Release.
In this way, patches leverage the intensive testing done on the Marketing and Update releases. Indeed, Solaris 10 patches leverage this intensive testing twice: once in the Marketing Release "soak" test, and again when the bug fix is included in builds of the next Solaris 10 Update.
The bug fix in each patch is verified by the responsible engineers in-house using a test case and/or by providing the T-Patch (Test Patch) to escalating customers to verify that it fixes the issue. In addition, the patch will be tested by the Patch System Test group and potentially by other test teams such as the Desktop QA or Hardware QA teams. Only when all verification and testing has been successfully completed will the patch be released to SunSolve. See an Overview of Patch Testing on SunSolve for further details.
Patch System Test test the patch both individually with any required patches, and cumulatively along with all other available patches. Testing the patch on its own helps ensure that all patch requirements have been correctly specified. Testing the patch in combination with all other available patches helps ensure that there are no bad interactions between patches. Testing these boundary conditions gives confidence that other patch combinations should work.
Extreme lengths are also taken in the code development and putback approval processes to ensure that patch requirements are correctly specified and that the change is compatible, well designed, and will not introduce regressions.
Nevertheless, if a customer takes individual patches which they feel are appropriate to their system, outside of a defined patch cluster such as the Recommended or Sun Alert Patch Cluster, they may end up running a code combination which has never been tested as a unit.
The various checks and balances in the patch process should be fully sufficient to ensure this code combination is stable and functional. But from a risk management perspective, running code which may not have been tested as a unit remains a finite risk.
This is what Bart Smaalders refers to as "Dim Sum" patching.
Most customers have practiced "Dim Sum" patching for years and, in general, it works very well. Even with the massive amount of code changes included in Solaris 10 Update Releases compared to Solaris 8 or Solaris 9 Update Releases, there have been very few issues as a result of "Dim Sum" patching.
But using the latest available Solaris Update release or Recommended or SunAlert Cluster or EIS CD (via Sun Connection 1.1.1 Satellite or xVM Ops Center 1.0) or other set of patches as a baseline has the advantage that that baseline has been tested to varying degrees as a unit, with Solaris Update releases the most intensely tested of those options.
This is a case where taking more change by installing or upgrading to the latest Solaris Update may actually imply less risk.
Customer Testing
Testing by Sun is just one factor. Testing by the customer, channel partner, or other vendor also plays a significant part in managing risk.
If the customer has a test set up which exactly mirrors their live production environment, with tests which mimic normal and peak loads, then their confidence level in any patching strategy they choose can be very high.
The less sophisticated the customer test environment, the more the customer is relying on Sun's Development and QA processes to catch all the issues.
Patch Quality
The good news is that the Sun's Development processes are meticulous and mature and the QA processes sophisticated and effective.
That doesn't stop all issues escaping to customers but, in general, the quality of patches is very high.
For example, out of approximately 4,500 patches released by Sun in 2007, only 70 have been subsequently withdrawn due to serious issues with them.
Patch Maturity
A number of customers like to wait a set period of time after a patch has been released before considering installing it to see if Sun or other customers find issues with it.
This is a reasonable strategy.
Some customers wait until 6 weeks after a patch has been released before applying it. Data analysis shows that there is no particular significance in this time period.
Data analysis shows that on the rare occasion where patches which are withdrawn from SunSolve after their release due to serious issues with them, the length of time between when a patch was released and when it was withdrawn shows no clear period of time before which a patch can be considered unsafe or after which a patch can be considered safe. Some patches are withdrawn within the first 3 weeks of release. Others not until 18 months or 2 years later.
However, it is reasonable to assume that patches which aren't withdrawn for a significant period of time only have a serious issue is a rare configuration which most customers won't encounter.
Posted at 12:53PM Jan 10, 2008 by Gerry Haskins in Sun | Comments[5]
Wednesday January 09, 2008
Patch Install Downtime Requirements
As mentioned previously, Solaris Live Upgrade can help minimize the downtime associated with patching, by enabling users to patch an inactive boot environment. When all modifications have been made to the inactive boot environment, a single reboot is required to activate it. Also, most Special Install Instructions specified in patch READMEs can be ignored when patching an inactive boot environment.
When patching a live boot environment, certain patches require system downtime in order to complete their installation.
Such requirements will be specified in the patch README file and is also specified by the SUNW_PATCH_PROPERTIES field in the patch's pkginfo file(s).
As mentioned previously, the problem with patching a live boot environment (without Deferred Activation Patching) is that some objects delivered in a patch, such as shared objects, may be invoked immediately while other objects, such as genunix, will only be activated when the system is rebooted. Also, processes already running in memory may be using the old versions of objects while processes started after the patch(es) were applied will be using the new versions of the same objects (when Deferred Activation Patching isn't specified). In some cases this is harmless. In other cases, the system may be in a potentially inconsistent state until it is rebooted.
Some patches require that they be installed in Single User Mode (run level S) when applied to a live boot environment. This is to ensure that the system is in a quiesced state to avoid the above potential problems. This will be specified in the patch README file. Also, the SUNW_PATCH_PROPERTIES field in the patch's pkginfo file(s) will contain the entry singleuser_required.
Some patches require that the system be rebooted at some convenient point after the patch is applied in order to activate its contents. Either a normal reboot or a reconfigure reboot may be required. This will be specified in the patch README file. Also, the SUNW_PATCH_PROPERTIES field in the patch's pkginfo file(s) will contain the entry rebootafter or reconfigafter. For such patches, the system remains in a consistent state until the reboot takes place. The reboot is simply to allow the changes supplied by the patch to be activated. If a reconfigure reboot is required, application of the patch will cause the creation of a /.reconfigure file which will result in a reconfigure reboot when the system is rebooted.
Some patches require that the system be rebooted immediately after the patch is applied to a live boot environment. For such patches, the system is in a potentially inconsistent state until the reboot takes place. Either a normal reboot or a reconfigure reboot may be required. This will be specified in the patch README file. Also, the SUNW_PATCH_PROPERTIES field in the patch's pkginfo file(s) will contain the entry rebootimmediate or reconfigimmediate. If a reconfigure reboot is required, application of the patch will cause the creation of a /.reconfigure file which will result in a reconfigure reboot when the system is rebooted. Normally, it is OK to apply further patches to the live boot environment before initiating the reboot. However, normal operations must not be resumed until after the reboot is performed. On the rare occasion where the reboot must be instigated before any further patches are applied, such as is the case with Solaris 10 Kernel patch 118833-36 (SPARC) / 118855-36 (x86), such patches will typically contain code to prevent further patches from being applied as an added safety mechanism.
Posted at 06:01PM Jan 09, 2008 by Gerry Haskins in Sun | Comments[7]
Friday January 04, 2008
Using Solaris Live Upgrade for patching
Solaris Live Upgrade can be used to patch a system as well as to upgrade from an earlier Marketing or Update release of Solaris.
Live Upgrade avoids many of the problems encountered when patching a live Solaris 10 boot environment as Live Upgrade modifies an inactive boot environment, rather than the live boot environment. This means that the live boot environment remains in a consistent state throughout the modifications. Once the inactive boot environment has been patched to the appropriate level, the system is rebooted to activate the modified boot environment. Live Upgrade has the additional advantage that it provides a ready-made fallback option - simply reboot back into the old boot environment if you encounter problems with the modified boot environment.
The downside of Live Upgrade is that it requires a number of prerequisite patches to be installed on the live boot environment before it can be used. See Infodoc 72099 available from SunSolve for a list of prerequisite patches. This is OK for pre-Solaris 10 systems and Solaris 10 systems without Containers (Zones).
However, for Solaris 10 systems with non-global Zones, this list of pre-requisite patches includes complex Kernel patches such as 118833-36 (SPARC) and 118855-36 (x86). This somewhat limits the usefulness of Live Upgrade as a Solaris 10 patching aid for systems with non-global Zones running a version of Solaris older than Solaris 10 11/06 (Update 3) or otherwise at a Kernel patch level of less than 118833-36 (SPARC) / 118855-36 (x86).
But for other environments, Solaris Live Upgrade provides a good method to patch systems.
For further information, please see the following articles on the How To page of the Big Admin Patching Hub:
How to Use Solaris Live Upgrade to Install Patches On Your System
How to Patch a System With RAID-1 Volumes by Using Solaris Live Upgrade
How to Patch and Upgrade a System by Using Solaris Live Upgrade When Non-global Zones are installed
Posted at 06:43PM Jan 04, 2008 by Gerry Haskins in Sun | Comments[1]
Deferred Activation Patching
The generic issue to be solved is the need to be able to patch safely across arbitrary change and is logged as CR 6486471.
The problems of patching a live Solaris 10 boot environment became most acute with the Kernel patch associated with the Solaris 10 11/06 (Update 3) release.
The problem when patching a live boot environment, is that some of the changes delivered in a patch, such as shared objects, may be invoked by processes as soon as they are applied to the live boot environment. Other objects, such as genunix, will only be activated when the system is rebooted. Problems can occur where the scope of the change applied is very large compared to that which is running on the live boot environment. In such cases, the new objects which are invoked, e.g. zoneadmd, may be incompatible with the old objects running in memory, e.g. genunix. This can cause the live boot environment to get into an inconsistent state during patching. The problem is most acute on a system running Zones, since the patch utilities need to invoke the zones utilities during patching to patch the non-global Zones.
A solution using loopback file system mounts (lofs) was incorporated into Kernel patch 118833-36 (SPARC) and 118855-36 (x86). This overlays the patched objects with the original versions which were present on the system, ensuring that the system remains in a consistent state during the application of the patch. When applied to a live boot environment, these patches require a reboot before any further operation, including the application of any further patches can be performed. This forced reboot requirement is not good from a system availability standpoint.
The solution was subsequently refined and formalized as Deferred Activation Patching.
Loopback mounts are used to overlay the original objects on top of the patched objects. This keeps the live boot environment in a consistent state during patching, irrespective of how much change is delivered in the patches. Once all patches have been applied, the system is rebooted to activate the changes delivered in the patches. At all points in time, the live boot environment remains in a consistent state. Patches specifying application in Deferred Activation Patching mode
set the SAFEMODE flag in the pkginfo file(s) in their packages. The Solaris patch utilities will automatically recognize if any subsequent patches applied prior to the reboot touch the same objects as the patch applied in Deferred Activation Patching mode and will automatically install such patches in Deferred Activation Patching mode too. Patches which don't intersect with a patch specifying Deferred Activation Patching mode will continue to be installed as normal.
Deferred Activation Patching was initially delivered in the Solaris 10 patch utility patch 119254-42 (SPARC) and 119255-42 (x86). It is recommended that customers use patch utility patch 119254-50 (SPARC) and 119255-50 (x86) or later revision as these address some bugs in Deferred Activation Patching.
The Kernel patch associated with the Solaris 10 8/07 (Update 4) release, 120011-14 (SPARC) and 120012-14 (x86), are the first patches to utilitize Deferred Activation Patching.
Deferred Activation Patching will only be specified in patches which require it. This is likely to be the Kernel patches associated with future Solaris 10 Update releases.
See the Deferred Activation Patching article on the BigAdmin Patching Hub for further information.
Posted at 06:42PM Jan 04, 2008 by Gerry Haskins in Sun | Comments[0]