Patch Corner: Patch news and best practice

I'd just like to make the following clear to customers:

Customers who install packages from one Update release (e.g. S10U6) on a system installed with another release (e.g. S10U3) risk corrupting their system.

You must not install packages from one Update release onto a system installed with any other Update release.

The reason for this is that all available patches are pre-applied into all packages for each Update release.   Patches and packages have a many-to-many relationship.  That is, one patch can patch many packages.   One package can be patched by many patches.

If you install a package from an Update release onto a system installed with a different Update release, you completely compromise future patch dependency checking as you've introduced patch metadata from a later Update release.   This is likely to lead to system corruption as further patches are applied.

'patchadd' checks that dependencies are satisfied when installing a patch.  If 'patchadd' finds any installed package patched with a patch which satisfies the dependency, it assumes the patch is applied to all packages.  This is done for performance reasons.  Hence, if a package from a later release is installed on the system, it's pre-applied patches may fool subsequent 'patchadd' invocations into thinking that a hard code dependency has been satisfied for all packages on the system when this is not the case.   The patch application will be allowed to continue, potentially corrupting the system.

The converse is also true.   If a package from an earlier Update is applied to a system, the patch delta from that Update to the Update installed on the system plus any additional patches installed on the system would need to be applied to the package to avoid a mismatch in software levels between the packages on this system, which could lead to incorrect patch dependency resolution and hence to system corruption.  Since this is difficult to get right, adding packages from a different Update release onto an installed system is, in general, unsupported. 

There are two exceptions to the above rules, for Live Upgrade and encryption packages.

• Live Upgrade: If you are upgrading from Solaris 8 or Solaris 9 to Solaris 10, you need to apply the Live Upgrade packages from the Solaris 10 system to your Solaris 8 or Solaris 9 system.  See http://sunsolve.sun.com/search/document.do?assetkey=1-9-206844-1 for details.

• SUNWcry* encryption packages: These weren't included in the Solaris distribution prior to Solaris 10 8/07 (Update 4).  For pre-Update4 systems, the recommended way to get these packages is to upgrade to a later Update release.  For the reasons outlined above, while not recommended, a possible alternative is to download the packages from the Sun Download Center (SDLC), install them, and then re-apply any patches that patch the SUNWcry* packages which you have already applied to the system (**). Also, for the reasons outlined above, it is not recommended to apply the packages from the media of a later Update release, as you would need to ensure that all the other packages installed on the system are patched to the same or higher patch level  (e.g. by installing the appropriate Solaris Update Patch Bundle available from http://sunsolve.sun.com/show.do?target=patch-access) to avoid completely compromising future patch dependency checking on the system.

** A way to check which patches need to be re-applied in this scenario is as follows:

cd /var/sadm/patch
egrep -i "PKG=SUNWcryr|PKG=SUNWcry" */log|cut -f1 -d /|sort -u

as in:

# cd /var/sadm/patch
# egrep -i "PKG=SUNWcryr|PKG=SUNWcry" */log|cut -f1 -d /|sort -u
127127-11
137137-09
139555-08
141444-09
#

The above example is on a system installed with Soalris 10 11/06 (Update 3).  The user would need to re-apply the patches listed above and be extremely careful to ensure they are applied in the correct dependency order as 'patchadd' will not be able to ensure the correct dependency order as it's dependency checking remains compromised until the added packages are brought up to the same patch level.

Please note that if a package from the same Update is applied to a system, then any additional patches already installed on the system that patch the added package must be re-applied to bring that package up to the same software level as the rest of the system.  This is called "incremental patching".   This is supported, but care must be taken.  The easiest way to do this is to reapply all patches installed on the system (as listed by 'patchadd -p').  This will bring the added package(s) up to the same software level as the rest of the system.  Again, you need to be extremely careful to ensure they are applied in the correct dependency order as 'patchadd' will not be able to ensure the correct dependency order as it's dependency checking remains compromised until the added packages are brought up to the same patch level as the rest of the system.

Best Wishes,

Gerry Haskins,
Director, Software Patch Services

I'm delighted to announce the availability of 10 new free online patch training modules.

This is the result of a lot of work from those nice people in Sun Learning Services, the Install Revenue Product Engineering (RPE, a.k.a. Sustaining) team, and my own folk.

The modules concentrate on using Live Upgrade for patching, as well as providing background on Deferred Activation Patching, Kernel patches, and other useful information.

You can access the modules as follows:

• Direct link to the course: Book 3-10: Solaris Patching Best Practices: https://learning.sun.com/solc/tblocontainer/1179270477

• MyLearning: https://learning.sun.com/display/Courses/WS-2700-S10+Solaris+10+Patching+Best+Practices
  

• Link to the Sun Online Learning Center (SOLC): https://learning.sun.com/solc/smartstart

I think even experienced Sys Admins will find the modules useful in clarifying patching best practices and providing context and background information on the evolution of patching technology and best practices in Solaris 10.

If you don't like the online course format, or if you want a reference document to refer back to after taking the course, please see the attached .pdf.

Enjoy!

Best Wishes,

Gerry Haskins,

Director, Software Patch Services

Here's some interesting tricks-of-the-trade and security related resources which I saw in a couple of email threads last week, which you may find useful:

What patches patch a specific object ?

We'll soon be enhancing the PatchFinder tool further to enable you to search for patches which patch a specified object.  So, if you're experiencing a problem with an object, you'll be able to see what patches exist for that object and look at the Bug fix synopses to see if any look like the issue you are experiencing.

But what patches on an installed system patch a specific object ?

The question which sparked the thread was: "What's the easiest way to determine what patch a binary (e.g. mpt(7D) driver) is tied to on a system?"

Option 1:  What patches installed on the system patch a specific object (e.g. /kernel/drv/mpt) ?

# cd /var/sadm/patch

# for x in `ls -rt` ; do grep "^/kernel/drv/mpt *$" $x/README.$x > /dev/null && echo $x; done

118855-36

127128-11

137138-09

139556-08

141445-09

Option 2: What patches installed on the system patch a specific object (e.g. /kernel/drv/sparcv9/mpt) ?  (This output is from a different system at a different patch level to the previous example.)

# /usr/ccs/bin/mcs  -p /kernel/drv/sparcv9/mpt
/kernel/drv/sparcv9/mpt:

@(#)SunOS 5.10 Generic 143128-01 Nov 2009

Option 3: What patches installed on the system patch a specific object (e.g. /usr/bin/ls) ?  (See Sun Blueprint on the SunSolve fingerprint DB: http://www.sun.com/blueprints/0306/816-1148.pdf )

# digest -a md5 /usr/bin/ls
6f20408d15ddfce2261436a27e33c0bd
#
and from http://sunsolve.sun.com/fileFingerprints.do
{
Results of Last Search

6f20408d15ddfce2261436a27e33c0bd - - 1 match(es)

        * canonical-path: /usr/bin/ls
        * package: SUNWcsu
        * version: 11.10.0,REV=2005.01.21.15.53
        * architecture: sparc
        * source: Solaris 10/SPARC
        * patch: 138377-01
}

Security Resources

Here are some excellent resources from Sun Distinguished Engineer, Glenn Brunette:

Everything you ever wanted to know about Solaris security...
http://mediacast.sun.com/users/gbrunette/media/s10-security-dive-20091021.pdf/details

The Solaris Package Companion is a small Korn shell script that allows you to ask quite a number of interesting questions about the relationships between Solaris metaclusters, clusters and packages as well as their respective dependencies.  Useful for system hardening, etc.: http://hub.opensolaris.org/bin/view/Project+svr4_packaging/package_companion

A Sun Blueprint on the SunSolve fingerprint DB: http://www.sun.com/blueprints/0306/816-1148.pdf

Enjoy!

I'm delighted to announce that the Solaris 10 10/09 (Update 8) Patch Bundle is now available for download by customers with a Solaris support contract.

Each Solaris Update Patch Bundle contains the equivalent set of patches which are pre-applied into the corresponding Solaris Update release image.

It is provided to enable customers who cannot upgrade for whatever reason to be able to patch systems up to the same patch level as the Update release.

Each Solaris Update is intensely tested as a unit by myriad QA teams across Sun.  Therefore, Solaris Updates and their corresponding Solaris Update Patch Bundles provide good quality "baselines" on which customers can standardize their deployments.

Standardizing deployments on such "baselines" also provides customers with a "safety in numbers" effect, as any pervasive issues are likely to be found and fixed quickly, so each customer benefits from the experience of others.

The Solaris Update Patch Bundle brings all existing packages up to the same software level as the Update release.   Any features which are entirely contained in pre-existing packages, such as Zones and ZFS functionality, are entirely available in patches and hence applying the Solaris Update Patch Bundle brings them up to the same functional level as the Update release.

However, installing the Patch Bundle is not completely equivalent to upgrading to the corresponding Solaris Update as the Patch Bundles do not include any new packages introduced in the Solaris Update release image.  Therefore, any new features which are dependent upon new packages will not be available by applying the Solaris Update Patch Bundle.

Here's a summary of the new packages in Solaris 10 10/09 (Update 8) which are not available in the Solaris 10 10/09 Patch Bundle:

SUNWhxge: SUN 10Gb hxge Ethernet Network Adapter Driver
SUNWio-tools: Administrative tools to modify the pci/pcie fabric
SUNWmrsas: LSI MegaRAID SAS2.0 HBA driver
SUNWpixman: Pixman Library
SUNWntp4r: NTPv4 (root)
SUNWntp4u: NTPv4 (usr)
SUNWntp4S: NTPv4 (source)
SUNWmptsas: LSI MPT SAS2.0 HBA driver

Please remember to apply the latest Sun Alert Cluster on top of the Solaris Update Patch Bundle in order to get all Solaris OS security, data corruption, and system availability fixes released since the final build of the Update release.

Please see previous blog entries for further details on Solaris Update Patch Bundles.

Top Tip: If you are installing in a zones environment, make sure you have the latest patch utility patches installed and Zones Parallel Patching configured before you apply a Solaris Update Patch Bundle as Zones Parallel Patching will improve non-global zone patching performance by ~300%.   See this blog entry for details.

BTW: There is no need to take any action to enable "Turbo-Charging SVR4 Package Installation" as the necessary patches are installed early on when installing the Solaris 10 10/09 Patch Bundle and will be automatically enabled for subsequent patch application when the bundle is applied to the live boot environment.  While "Turbo-Charging" has little affect when installing most patches, it does significantly speed up the application of a small number of older patches with non-optimized deletes file processing install scripts and so does speed up the Solaris 10 10/09 Patch Bundle installation somewhat.

Best Wishes,

Gerry.

I'm delighted to announce the release of the 2nd phase of our PatchFinder tool enhancements, which include:

• The ability to see the "Entitlement Classes" of patches and get information on the support contracts necessary to access and use them.  

• A "Patch Basket", into which you can add selected patches from multiple search results.

• When you click on the "Go To Patch Basket" link, the patch dependencies for all the patches you have in your Patch Basket will be dynamically resolved, including filtering out redundant dependencies.   This saves you having to manually transfer patch dependency trees!   If you already have some of these installed, you can de-select them.
  

• You can then click the "Download Selected" button to download a 'wget' script and instructions which you can use to download all of the selected patches from SunSolve.   Once you make sure you install the latest version of the patch utilities patch first, you can then use "patchadd -M" to install all the patches in the correct order on your target system.
  

Sample Searches

Let's assume you applied the Solaris 10 SPARC Recommended Patch Cluster on August 15th 2009.  So what Solaris 10 SPARC Recommended Cluster patches have been released since then ?   To find out, for "OS Release" select "Solaris 10", for "Architecture" select SPARC", select "Recommended Only", and select August 15th 2009 from the calendar beside the "Released After" box.   (Select view 50, 100, or 200 to see the entire list in one page.)   You can then decide if you want to download some of all of these patches to add to your system.  Coupled with the dynamic dependency resolution and 'wget' download capability, this effectively enables you to create customized patch clusters for yourself with just the patches you need, rather than having to download the entire Recommended Cluster each time.

Or you could bookmark a search to show you all the patches released in the last day: Simply enter the number "1" into the "Released After" box and select any other selection criteria you are interested in and click "Search".  Depending on timezone differences with respect to California and your local time of day, you may need to enter the number "2" in the "Released After" box.

You can also use PatchFinder to see what Solaris 8 Vintage patches Sun has released since Solaris 8 entered End-Of-Service-Life (EOSL) Phase 2 on April 1, 2009.   Simply select "Solaris 8" for "OS Release", select "OS Patches Only" and click "Search".  Since the patches are listed in date order, most of the patches with a release date after April 1, 2009, including patches delivering security fixes, will have the "Solaris8VintageSoftwareUpdate" Entitlement Class associated with them if you mouse-over the red padlock symbol shown for them (assuming you don't have a Solaris 8 Vintage Patch Service Plan associated with your Sun Online Account).   You will see a couple of non-Vintage patches released after April 1, 2009.  This is a transition phase and these patches address issues escalated by customers prior to April 1, 2009.

Some other sample searches to satisfy your curiosity:

Ever wondered how many patches Sun has ever released ?   To find out, simply select "Show Obsolete" and then click "Search".

How many current "active" patches does Sun have ?   De-select "Show Obsolete" and then click "Search".

How many patches can be installed on Solaris 10, including application product patches ?   For "OS Release" select "Solaris 10" (and optionally "Show Obsolete" ) and then click "Search".

How many current "active" Solaris 10 OS patches there are for SPARC ?  For "OS Release" select "Solaris 10", for "Architecture" select "SPARC" and then select "OS Patches Only" and then click "Search".

Patch Access Entitlement Classes

When you look at the list of patches returned from a search, the letter-P-in-a-circle symbol shows which patches are "Public" and can be accessed and used without a support contract.  A green open padlock symbol shows the patches you have access to thanks to the support contracts which you currently have associated with your Sun Online Account (SOA).  A red closed padlock shows the patches which you are not currently entitled to access or use with the support contracts you currently have associated with your Sun Online Aaccout.  

You can mouse-over these symbols for any patch and it will show you the "Entitlement Classes" associated with the patch. 

Read the "What is it?" help link and the SunSolve "How Entitlement Works" wiki to find out about the support contracts which you need to buy in order to access and use these patches.

Feedback

I hope you'll find the new PatchFinder enhancements useful.

We are really interested in your feedback as to what further enhancements you would like to see, so feel free to post your comments here or else use the feedback link on the PatchFinder page.

Many thanks to Brian Kidney and Julien Colomb for all their work on this - nice work guys!

I've updated the Solaris 10 Kernel PatchID sequence table in http://blogs.sun.com/patch/entry/solaris_10_kernel_patchid_progression with the latest Kernel PatchIDs for Solaris 10 10/09 (Update 8) and Solaris 10 Update 9.

The new Patching Pre-flight Checks ('ppc') tool is now available to all customers who have a support contract.

The idea for this tool comes directly from customer feedback.  

The customer wanted to reduce the cost of patching Solaris systems by enabling more junior Sys Admins to successfully patch Solaris 10 zones systems.   Their concern was that potential zones patching issues in versions of Solaris prior to Solaris 10 8/07 (Update 4) meant that they needed to assign senior System Administrators to patch such systems to identify and resolve potential issues.  

Furthermore, the customer was concerned that such issues had the potential to derail planned maintenance windows - for example, if during the patching session an unexpected issue was encountered and the patching session couldn't be completed as planned.

To address these concerns, my colleague, Ronan O'Connor, has written the Patching Pre-flight Checks tool, 'ppc'.  It can be run prior to a planned patching session to check that the target system is in a clean state ready for patching.

It's important to understand the scope of the tool.  It checks a target system (and a patch set, if supplied) for a variety of inconsistencies which could cause problems.

It looks for left over lock files from previously aborted patching or packaging operations, inconsistencies in the contents database, IDRs installed on the target system, zones "mountability", space issues, etc.  Some of these issues can occur on early versions of Solaris 10, particularly in a Zones environment.  Many of the underlying causes of such issues are fixed in the latest versions of the patch utility patches (119254 SPARC / 119255 x86), which is why we always recommend you apply the latest patch utility patches before applying other patches.

If you have a directory of patches to be applied, 'ppc' checks the integrity of those patches, and cross-checks whether any of the patches patch pkgs which have been locked down by any IDRs on the system and warns if there is a conflict.

The 'ppc' Release Notes provide information to help interpret the messages produced.

The idea is that 'ppc' can be run by a junior Sys Admin prior to a planned patching session, and any potential issues uncovered can then be analyzed by a more experienced Sys Admin.  This helps avoid nasty surprises during patches sessions and also helps to reduce the level of expertise required to patch Solaris systems, leading to cost savings for customers.

It is outside the scope of the 'ppc' tool to do root cause analysis of why the inconsistency arose or what actions may be needed, if any, to correct the situation.

If 'ppc' returns without noting any problems, you can be pretty confident that the patching session will succeed.  If 'ppc' notes potential issues, they can be investigated prior to the planned maintenance window.

The next version of 'ppc' will include a Zones consistency check to check that all zones are at a consistent patch level.   It will also contain a more sophisticated space checking algorithm.  There's no planned release date yet for Version "2.0" yet as we're awaiting feedback on Version 1.0.x first.

Some of the ideas in 'ppc' may find their way back into 'patchadd', although it's probably appropriate to keep 'ppc' as a separate tool.

To download the Patching Pre-flight Checks tool, 'ppc', go to the 'ppc' thread on the Customer Patch Forum.  If you have not accessed the Customer Patch Forum before, please see my blog entry on the initial "secret-handshake" login. 

The Patching Pre-flight Checks tool, 'ppc', and the Customer Patch Forum are only available to customers with a support contract.

We're very interested in your feedback as to the usefulness of this tool and how you'd like to see 'ppc' develop going forward.

Many thanks to Ronan O'Connor for all his work on the tool!

Best Wishes,

Gerry Haskins
Director, Software Patch Services

I am delighted to announce that Casper Dik's "Turbo-Charging SVR4 Package Install" feature enhancement is now available by downloading and installing the following patches:

119254-70 (SPARC) / 119255-70 (x86): Patch utilites patch
121428-13 (SPARC) / 121429-13 (x86): Live Upgrade Zones Support Patch
121430-40 (SPARC) / 121431-41 (x86): Live Upgrade Patch
124630-28 (SPARC) / 124631-29 (x86): System Administration Applications, Network, and Core Libraries Patch

It is important to apply 119254-70 (SPARC) / 119255-70 (x86) and 121428-13 (SPARC) / 121428-13 (x86) if the system is running non-global zones.  Otherwise booting newly installed zones will fail until the pkgserv daemon exits, about 5 minutes after zoneadm install finished.  Zones which were already installed can be booted as expected.

"Turbo Packaging" is designed to dramatically improve the performance of install, upgrade, Live Upgrade, and zone creation on Solaris 10.  It delivers only a small improvement for patching performance.   (See my Zones Parallel Patching blog entry for information on dramatic patching performance improvements.)

For background reading on the "Turbo Packaging" feature, please see
http://www.opensolaris.org/jive/thread.jspa?messageID=358081 and http://arc.opensolaris.org/caselog/PSARC/2009/173/. 

The "Turbo-charging SVR4 Package Install" feature will also be included in the forthcoming Solaris 10 Update 8 release and will be documented in the Release Notes for that release.

Great work, Casper - well done!

Internetnews.com has an interesting article on IBM's X-Force Report which praises Sun for fast fixes and being best for patching the highest percentage of reported security vulnerabilities:  http://www.internetnews.com/security/article.php/3836436/IBMs+XForce+Report+Praises+Sun+for+Fast+Fixes.htm

Ever wanted to run something past a patching expert ? 

Want to pick the brains of your peers in other companies ?

Lobby to get a patching enhancement implemented ?

Debate what is an appropriate patching strategy and associated best practices ?

Ask other customers if they are seeing the same issues you are ?

Your wait is over.  There are new Patch Forums available to customers with support contracts on http://forums.sun.com.

If you haven't accessed the private contract customer forums on forums.sun.com before, I'm told there's a bit of a "secret-handshake" procedure you must follow for your initial login:

• Go to https://identity.sun.com/amserver/UI/Login?org=self_registered_users&goto=http://sunsolve.sun.com/cwpGoto.do?target=home
• Log in with your Sun Online Account (SOA) username/password as normal.
• Click on "Edit Personal Information" in the menu on the top right of the screen and ensure that you have a "Screen Name" set.  If not, set one here.  This will be the name displayed when you post on forums.sun.com.
• If you are not an Member Support Center (MSC) registered user, click on "Change Contract" in the menu on the top right of the screen and ensure you have one or more support contracts associated with your Sun Online Account (SOA).  If not, associate your support contract number(s) to your SOA.  For MSC registered users, contracts are automatically associated to your SOA.
  
• Go back to http://sunsolve.sun.com/show.do?target=home and click on the "[ Access ]" link under the "Sun Community Forums" section.  This link is only visible if you have a support contract associated to your Sun Online Account.  Your initial access to the Patch Forum must be via this link.
  
• Scroll down the page you are redirected to, you will now see a Forum called "Patches" in the "Private" section.
• Click on "Patches" will bring you to the Patch Forum, http://forums.sun.com/category.jspa?categoryID=162
• If you do not see the "Patches" link, please try and logout of forums.sun.com and then log back in, as sometimes there are issues with single sign on preventing you from seeing the correct links in the "Private" section.  (Alternatively, clear your cookies and follow the above steps.)
  

Please note that the above process need only be followed once, in order to register your "Screen Name" for access to the private contract customer forums on forums.sun.com

For all future access, you may go directly to the Patch Forums and login via the "Login" link on the top right of the screen to see the Patch Forum content.

Patches Forum

http://forums.sun.com/category.jspa?categoryID=162

You can use the "Let's talk patching!" Community Forum to discuss Patching Strategy and Best Practices with peers and Sun folk, including potential enhancements to improve your patching experience.  It is important to note that this is not an official Support channel.  To ensure optimal support, you must continue to raise Service Requests for specific patching issues through the normal support channels, although you may use the Patch Issues sub-forum to see if others have experienced similar issues and know of workarounds or available fixes.  As with other forums of this nature, Sun does not guarantee to answer all questions posted. We'd like your feedback on how you would like to see this forum evolve and keep the questions coming so we can better serve your patching needs!

There are two sub-forums available:

Patching Strategy, Best Practices, and Enhancements

http://forums.sun.com/forum.jspa?forumID=1029

Discussions on patching strategies, best practices, news, and potential enhancements to improve your patching experience.

Patch Issues

http://forums.sun.com/forum.jspa?forumID=1028

Please note that this is not an official Support channel. To ensure optimal support, you must continue to raise Service Requests for specific patching issues through the normal support channels, although you may use the Patch Issues sub-forum to see if others have experienced similar issues and know of workarounds or available fixes.

I hope you find these forums useful!

They are designed to facilitate dialogue between you and other customers and with Sun subject matter experts to help improve your patching experience.

Best Wishes,

Gerry Haskins
Director, Software Patch Services

My colleague, Don O'Malley, asked me to post the following on resolving issues using 'wget' to automate patch downloads.  'wget' is a popular download method, and is used by patch automation tools such as 'pca'.

Summary: You can use versions 1.10.x and 1.11.x of 'wget' but not version 1.11.  Details of options to use are set out below.  See also Patch Download Automation using wget.

SunSolve recently migrated to using Akamai for patch and patch cluster downloads, to provide customers with a faster and more reliable experience.

Some customers have experienced issues accessing patches using 'wget'.  Here's information on the issues and how to resolve them:

1) You must use a version of 'wget' which supports 'https'.

Why?

SunSolve's new patch download service is accessed by redirecting requests to https://getupdates2.sun.com, which subsequently redirects to https://a248.e.akamai.net (Akamai).

Which versions of 'wget' support 'https'?

'wget' version 1.10.x or later has 'https' support.

How can I check which version of 'wget' I am using?

Run the command 'wget --version'

2) You must use the '-O' or '--output-document' switch in 'wget' to provide an output filename.

Why?

The Akamai URI identifying a patch is very long.  By default 'wget' will name the downloaded file the same as the URI.  As the filename is too long an error is thrown and the download will fail.

Example of the correct syntax:

# /usr/sfw/bin/wget ---http-user="xxxxxxxx" --http-passwd="xxxxxxx" --no-check-certificate "http://sunsolve.sun.com/pdownload.do?target=119255-01&method=h" -O /tmp/119255-01.zip

Example of some the output for a failing 'wget' request:

140778-01.zip?AuthParam=1251205908_479a27379ab5595128ae9170de4228c9&TUrl=L0QdUQV8Z4i0fdED3QTP3SJDWA8FMyaJsHfIWf4X29kTWQpKEzIbwqFuyRPZ&TicketId=3q3wk1CPNxhU&GroupName=SWUP&BHost=sdlc2h.sun.com&FilePath=%2Fpatches%2Fpatchroot%2Fall_unsigned%2F140778-01.zip&File=140778-01.zip: File name too long

Cannot write to `140778-01.zip? AuthParam=1251205908_479a27379ab5595128ae9170de4228c9&TUrl=L0QdUQV8Z4i0fdED3QTP3SJDWA8FMyaJsHfIWf4X29kTWQpKEzIbwqFuyRPZ&TicketId=3q3wk1CPNxhU&GroupName=SWUP&BHost=sdlc2h.sun.com&FilePath=%2Fpatches%2Fpatchroot%2Fall_unsigned%2F140778-01.zip&File=140778-01.zip' (Error 0).

3) If you are using 'wget' version 1.11.x you must use the '--auth-no-challenge' switch.

Why?

This is related to the manner in which 'wget' 1.11.x sends SunSolve a users Sun Online Account (SOA) information in this version of 'wget' (i.e. via '--http-user' & '--http-passwd'.)

Failure to include the '--auth-no-challenge' with 'wget' 1.11.x requests will result in the SunSolve Software License Agreement (SLA) being downloaded rather than the patch.

Example of the syntax for 'wget' 1.11.x users:

# /usr/sfw/bin/wget --auth-no-challenge --http-user="xxxxxxxx" --http-passwd="xxxxxxx" --no-check-certificate "http://sunsolve.sun.com/pdownload.do?target=119255-01&method=h" -O /tmp/119255-01.zip

Note, 'wget' version 1.11 does not have the '--auth-no-challenge' switch and so is not compatible with patch downloads from SunSolve.

4) You must provide 'wget' with direction on how to handle security certificate information.  Otherwise, patch downloads via 'wget' will fail.

Why?

Domains, getupdates2.sun.com & a248.e.akamai.net, are signed by trusted Certificate Authorities. (Verisign for Sun's and GTE Cybertrust for the case of Akamai.) Without a pointer to these certificates being provided to 'wget', download attempts will fail.

Which certs are required?

CN=GTE CyberTrust Global Root
CN=VeriSign Class 3 Secure Server CA - G2

What kind of error message can you expect to see from a failing 'wget' request?

ERROR: Certificate verification error for getupdates2.sun.com: self signed certificate in certificate chain
To connect to getupdates2.sun.com insecurely, use `--no-check-certificate'.
Unable to establish SSL connection.

Issue resolution:

If you wish to ignore this failure you can use the '--no-check-certificate' switch in 'wget'.  Example of the syntax:

# /usr/sfw/bin/wget --http-user="xxxxxxxx" --http-passwd="xxxxxxx" --no-check-certificate "http://sunsolve.sun.com/pdownload.do?target=119255-01&method=h" -O /tmp/119255-01.zip

If you wish to check against the certificates, you can use the '--ca-certificate' switch to point to a file containing the certificates.

http://sunsolve.sun.com/search/document.do?assetkey=1-9-240066-1 has an attachment called cacerts.pem, which is a concatenation of the two certificates.

If you save this file locally (eg to /tmp/cacerts.pem), you can use a syntax similar to:

# /usr/sfw/bin/wget --ca-certificate=/tmp/cacerts.pem --http-user="xxxxxxxx" --http-passwd="xxxxxxx" "http://sunsolve.sun.com/pdownload.pl?target=142284&method=h" -O /tmp/140778-01.zip

5) You may need to add firewall rules to enable 'wget' to work with SunSolve's new download service.

Why?

As the new download service is accessed by redirecting from http//:sunsolve.sun.com to https://getupdates2.sun.com initially and subsequently to https://a248.e.akamai.net, some customers may need to update their firewall rules to pass traffic from getupdates2.sun.com & a248.e.akamai.net in addition to sunsolve.sun.com.

How can I verify this?

Contact your System Administrator.

6) After associating a new contract to a SunSolve account there is a delay of up to 48 hours before 'wget' downloads will work for patches that the new contract should provide access to.

Additionally, customers registered in the Members Support Center must make an initial 'wget' call (which will fail) in order to trigger the synchronization process after associating a new contract to their party.

Why?

The delay is due to synchronization issues between SunSolve and the back-end access entitlement system.  Work is ongoing to reduce this delay.

What error message can you expect to see until this synchronization is complete ?

HTTP request sent, awaiting response... 403 You are not entitled to retrieve this content.

7) Attempts to download a patch README file by providing "method=r" in the URI is now failing.

Why?

Prior to the latest SunSolve release it was possible to download a patch's README file only via 'wget', using a syntax similar to :

# /usr/sfw/bin/wget --no-check-certificate --http-user="xxxxxxxx" --http-passwd="xxxxxxxx" "http://sunsolve.sun.com/pdownload.do?target=142284-01&method=r" -O /tmp/142284-01.README

There's a bug in the current SunSolve release this no longer works and attempts to download a patch README using this URI will result in a file of 0 Bytes being created.  This will be fixed at a later date.

Workaround:

Use "method=tr" to download a patch README file.  Example command syntax:

# /usr/sfw/bin/wget --no-check-certificate --http-user="xxxxxxxx" --http-passwd="xxxxxxxx" "http://sunsolve.sun.com/pdownload.do?target=142284-01&method=tr" -O /tmp/142284-01.README

My colleague, Ed Clark, has made significant improvements to the Solaris 10 Recommended and Sun Alert patch clusters.  These improvements have just been released and are in the current clusters available to contract customers from the Patch Cluster & Patch Bundle Downloads on SunSolve.

Ed's improvements include:

• Filtering out "false negatives" from the patch utility return codes, so that if the cluster install script returns "1", you know you've got a real problem which needs investigating.   As you may know, the Solaris patch utility, 'patchadd', can return errors for some acceptable situations - for example, if the patch is already applied to the system, or a later revision of the patch or a patch which obsoletes it is already applied to the system, or none of the packages in the patch are on the target system (e.g. because a reduced Install Metacluster was used to install it or the system has been security hardened by package removal), etc.   Such conditions are acceptable "errors" which do not usually require further investigation by the user.  By filtering these conditions out, if the 'installcluster' script returns "1", you know it isn't because of one of these acceptable "errors", and therefore you need to look at the logfiles to find out what's gone wrong.  For further information, please see the cluster README and Analyzing a patchadd or patchrm Failure in the Solaris OS.

• The new 'installcluster' script will exit as soon as it encounters an unexpected failure - i.e. not one of the acceptable "errors" mentioned above.  This prevents potentially compounding issues by attempting to apply further patches.
  

• The new 'installcluster' script includes context intelligence for patching operations.   It informs the user when zones need to be halted, and it provides phased installation to handle patches which absolutely require an immediate reboot before further patches can be applied.  Such interim reboots are only needed when patching a live boot environment on a system below Kernel patch 118833-36 (SPARC) / 118855-36 (x86) and well as the earlier interim reboot required on x86 related to 'libc.so' patches and Kernel patch 118844-14.  On systems below these patch levels, the 'installcluster' will stop at the appropriate point when patching the live boot environment, and inform the user to reboot and re-invoke the 'installcluster' script.  (In the old cluster install script, it simply tried to carry on blindly past such interim reboots, spewing out error messages, although code in the relevant patches prevented any harm from being done).  These interim reboots, when required, are dealt with relatively early in the cluster install sequence so that once completed, the Sys Admin can leave the rest of the installation to finish unattended and move onto other systems.

• The new 'installcluster' script provides better integration with Solaris Live Upgrade as the user can now specify the Live Upgrade alternate boot environment to patch by name.

• The new 'installcluster' script performs space checking prior to installing each patch, and will halt if it believes there is insufficient space to complete the installation successfully.  For example, this helps avoid non-global zones getting out of sync regarding patch levels with respect to the global zone.  This is an important enhancement as running out of space during patching can potentially leave the system in an inconsistent state and is to be avoided.  Even removing a patch requires space, so immediate removal of a patch which has failed to apply correctly due to space issues should be avoided until sufficient space is freed up and potential issues caused by its partial installation investigated - for example, was the undo.Z file successfully created to enable backout ? (Tip: It may be better to retry the patch installation once space has been freed up rather than patch removal in such circumstances.  Contact Sun Support for instructions if you encounter such issues.).   The space checking enhancements in the 'installcluster' script are designed to prevent such problems occurring.
  

• The messages and log files produced by the 'installcluster' script are clear and well structured.  For example, a "failed" log is created if a patch fails to apply.  See the Cluster README for further information.
  

• The 'patch_order' places patches in an optimal order for installation to avoid known issues - for example, the patch utilities patches are installed as early in the sequence as possible to avoid hitting patch installation bugs which are fixed in the patch utility patches, and the Kernel patch procedural script override patch, 125555 (SPARC) / 125556 (x86), is ordered prior to 137137-09 (SPARC) / 137138-09 (x86) to resolve some known issues.  When patching an alternate boot environment (which is recommended), a small sub-set of pre-requisite patches, primarily the patch utility patches, need to be applied to the live boot environment to ensure correct patching operation.  The 'installcluster' script will check for these pre-requisite patches are halt installation if they are not present, advising the user of the 'installcluster' script option to use to install these pre-requisite patches.   Further patches may need to be installed on the live boot environment to support Live Upgrade.  See the cluster README for further information.
  

• The patches have been moved to a 'patches' sub-directory, to de-clutter the top level directory of the unzipped cluster.
  

• Please see the cluster README file for further information.  Customers should read the cluster README file and look at the Special Install Instructions in the patches within the cluster prior to installation.
  

I really want to thank Ed Clark for the enormous amount of thought and effort he has put into improving the cluster installation experience.   The work he's done on the Solaris 10 Recommended and Sun Alert patch cluster is a continuation of his previous work on the Solaris Update Patch Bundles and the Solaris 10 Live Upgrade Zones Starter Patch Bundle.  Nice work, Ed!

While the 'installcluster' script is copyrighted, I am happy for customers to use it, and the 'patch_order' file, as a starting point for their own customized patch bundles, so long as it is for their own use and is not to be given to a 3rd party or used for commercial gain (e.g. by a 3rd party maintainer or 3rd party commercial automation tool).

We have also made significant improvements to the back end processes to ensure higher and more consistent cluster quality. 

Originally, the clusters were created by the Patch Operations and Distribution (POD) team after patch release.  The POD Cluster QA process left a lot to be desired, resulting in inconsistent cluster quality.   To plug this gap, my Patch System Test team have been testing the clusters for several years, but the old process only allowed us to test them in parallel with their release, which meant that we found issues at the same time that early downloaders of the cluster encountered them.  Although we ensured such issues were fixed as quickly as possible, it still obviously compromised our customers' experience.

In the new process, the clusters are routed to Patch System Test (PST) prior to release.  PST run a transformation script on them to optimize the patch installation order, etc.  The clusters will only be released once they have passed PST testing.  This should ensure higher and more consistent quality for customers.  Work is continuing to move the entire patch cluster generation process to PST, although these future backend enhancements in this regard should be invisible to customers.

SunSolve 7.3.0 Release, Akamai, and Vintage Solaris 8 patch access entitlement

The SunSolve 7.3.0 release was deployed to production August 11th. 

It includes major changes to back-end processes designed to provide a more robust, reliable, and consistent customer experience.  All patch downloads are now serviced by Akamai, which is the same process used by Sun's patch automation tools smpatch, Update Manager, UCE, and xVM Ops Center.

Firewall rules may need to be changed to permit the access to the following systems:

• sunsolve.sun.com
• getupdates2.sun.com
• a248.e.akamai.net

The move to using Akamai to service download requests should resolve the transient "500" error issues in Squid which was impacting the reliability of patch downloads in the old SunSolve download infrastructure.

This release also removes Member Support Center (MSC) from the critical path for Solaris 8 Vintage Patch access entitlement.   Prior to this release, Vintage Solaris 8 customers needed to register in MSC in order to access Vintage Solaris 8 patches (created after April 1, 2009).  This was difficult for some customers who needed to undergo a contract clean-up process prior to full registration in MSC.  Now, such customers can simply associate their Vintage Solaris 8 Patch Plan contract number with their Sun Online Account (SOA) using the "Change Contract" link at the top right hand corner of SunSolve pages once they have logged on.  This is now sufficient to grant patch download entitlement to patches covered by any support contract, including Solaris 8 Vintage patches.

Note, customers who are registered in Member Support Center (MSC) will not see the "Change Contact" link as their contract associations are automatically handled by MSC.

For non-MSC users, to ensure access to all patches to which you are entitled, please ensure your associate your Support Contracts with your Sun On-line Account.

Recognition of Support Contract Changes

Support contracts naturally get renewed, upgraded, extended, or expire.  

When a support contract changes - for example a new line item is added to provide support for additional products - then, for non-MSC registered users, to get this additional entitlement "recognized" quickly to enable manual download of access-entitled patches covered by this additional line item, either remove and re-add the Contract number to your Sun Online Account (SOA) using the "Change Contract" link on SunSolve while logged on or else simply log out and log back in again.  Both methods will grant the additional access entitlement as long as the back-end IBIS Contract database has been updated with the modified contract information.

For Member Support Center (MSC) registered users, the contract association will be handled automatically by MSC.    (BTW: A bug in the refresh of IBIS Materialized Views has now been fixed, so delays in automate updates of contract associations by MSC should no longer occur, once the contract amendments have been inputed to the backend database.)

Patch access entitlement information

We will be improving the ability for customers to clearly determine what they are / are not entitled to access in the next release of SunSolve and the new PatchFinder tool (due in October).

In the meantime, when logged into SunSolve, go to the "Change Contract" link at the top right hand corner of SunSolve pages.

This will display the "Entitlement Classes" provided by the support contracts which you have currently associated with your Sun Online Account (SOA).  Displaying the internal "Entitlement Class" names is not ideal and will be improved in the next release, but here's how to interpret them:

• "Public": You are entitled to access Public patches - i.e. patches which don't require a support contract to access them.
• "Solaris8VintageSoftwareUpdates": You have a Solaris 8 Vintage Patch Service plan and are entitled to access Solaris 8 Vintage patches produced after April 1, 2009.  (See previous blog posting on the Solaris 8 Vintage Patch Service plan.)
• "Solaris8SoftwareUpdates": You are entitled to access non-Vintage Solaris 8 patches.
  
• "Solaris9SoftwareUpdates": You are entitled to access Solaris 9 patches.
  
• "Solaris10SoftwareUpdates": You are entitled to access Solaris 10 patches.

There are a couple of additional entitlement classes, some of which are historical artifacts which overlap with the above.  These will be cleaned up in due course.

Did you know:

• You must have a Solaris 8 Vintage Patch support plan in order to access Vintage Solaris 8 patches created after April 1, 2009
• A SunSpectrum support plan or a Solaris 8 Software Subscription entitles you to access non-Vintage Solaris 8, 9, and 10 patches
• A Solaris 9 Software Subscription entitles you to access Solaris 9 and 10 patches
• A Solaris 10 Software Subscription entitles you to access Solaris 10 patches

Another "did you know":

Many documents on SunSolve have a "Document Audience:" of "PUBLIC".  However, in the case of patch README files, this does not necessarily mean that the patches they refer to have "public" access entitlement - i.e. that anyone can download the patch without a support contract.  The README is designed to make folk aware of the existence of a patch they may need.  However, they may still need to purchase a support contract in order to access the patch itself.

Using 'wget' to automate patch downloads

'wget' is a popular and efficient way to automate patch downloads.   Popular patch automation tools such as 'pca' and TLP utilize 'wget' for patch downloads.  Authentication is via the user's Sun Online Account (SOA), so customers should associate their support contracts to their SOA using the "Change Contract" link at the top right hand corner of SunSolve pages once they have logged on.

A version of 'wget' which support https transfers is now required in order to download patches.  For example, the 'wget' version in Solaris 10 supports https transfers.  To check whether the version of wget you are using is linked to SSL (to provide https support), you can use the following command:

# wget --help.

For example, the current development releases of wget (1.12-devel) shows:

   Options: +digest +ipv6 +nls +ntlm +opie +md5/openssl -gnutls
           +openssl +gettext

You also have to have your proxy configured to allow https connections through the proxy with the 'connect' command.

When contracts are added, renewed, or changed, MSC registered 'wget' users now need to attempt a download of a access-entitled patch (which will fail) in order to trigger a resynchronization of their contract data between the backend servers servicing the patch download request.  The modified contract entitlement will then be activated within 8 hours of this initial download attempt.

See Information on using wget for http download including example download script for further information.

Solaris 2.5.1 patch access entitlement removed

Solaris 2.5.1 is past its End Of Service Life (EOSL).   Access to Solaris 2.5.1 patches has therefore been removed.

Vintage Phone support, which includes access to existing patches (but no new patches will be created) is still available for Solaris 2.6 and Solaris 7 until the end of 2009, after which all access to Solaris 2.6. and Solaris 7 patches will also be removed.

I've changed the background "theme" used on this blog as the new theme provides Indexing and RSS feeds which was a requested enhancement.

Thanks to my colleague, Brian Kidney, for finding the theme. 

Otherwise, as Led Zep would say, the song remains the same.

Sun Learning Services are in the process of creating a number of patch related training lessons.

They've launched a blog, which contains the initial introductory videos.

Future lessons will be much more detailed, concentrating for example on Live Upgrade.   These lessons will be available on the Sun Open Learning Center (SOLC) website: https://learning.sun.com/solc/smartstart.

I'd like to give you a heads-up on a couple of Kernel patch installation issues:

1. There was a bug (since fixed) in the Deferred Activation Patching functionality in a ZFS Root environment on x86 only.  See Sun Alert 263928.  An error message to the effect that a Class Action Script has failed to complete and failure to set up environment for Deferred Activation Patching may be seen.   The relevant CR is 6850329: "KU 139556-08 fails to apply on x86 systems that have ZFS root filesystems and corrupts the OS".    SPARC systems are similarly affected.  The following error message is returned:

mv: cannot rename /var/run/.patchSafeMode/root/lib/libc.so.1.20102 to /lib/libc.so.1: Device busy
ERROR: Move of /var/run/.patchSafeMode/root/lib/libc.so.1.20102 to dstActual failed
usage: puttext [-r rmarg] [-l lmarg] string
pkgadd: ERROR: class action script did not complete successfully

Installation of <SUNWcslr> failed.

This issue is fixed in patch in the Patch Utilities patch 119255-70 or later revision.

BTW: The principal reason ZFS Root support was implemented in Live Upgrade is so that patch application like this to the live boot environment would not be necessary.   With ZFS Root, creating a clone Boot Environment is so easy that there's no good reason not to.   This avoids the need to use technologies such as Deferred Activation Patching which attempt to make it safer to apply arbitrary change to a live boot environment, which is an inherently risky process.

2. There are reproducible issues using jumpstart finish scripts and other scenarios to install Kernel patch 137137-09 followed by Kernel patch 139555-08.   Here's the gist of the issue which I've pulled from an engineering email thread on the subject:

Issue 1: I have a customer whose system is not booting after applying the patch cluster with Live Upgrade (LU).

Solution 1: If using 'luupgrade -t', then you must ensure that latest version of LU patch is installed first, currently 121430-36 is currently the latest revision on SPARC, 121431-37 on x86. Once these patches are installed, LU will automatically handle the build of the boot archive when 'luactivate' is called, thus avoiding the problem.

Issue 2: There are other ways to get oneself into situations where a boot archive is out of sync: e.g. If using jumpstart finish scripts to apply patches that include 137137-09.  Basically any operation that involves patching to an ABE outside of 'luupgrade' will involve a manual build of boot-archive.

Solution 2: One must manually rebuild the boot-archive on the /a partition after applying the patches.  Otherwise once the system boots, the boot-archive will be out of sync.

Here's some more detail on the jumpstart finish script version of this: 

We've seen the same panic a few times when the latest patch cluster is applied via a finish script to a boot environment prior to  s10u6 via a jumpstart installation. It appears that the boot archive is out of sync with the kernel on the system. The boot archive was created from the 137137-09 patch and not updated after the 139555-08 kernel was applied, therefore the mismatch between the kernel and the boot archive.

In these instances updating the boot archive allows the system to boot successfully. Boot failsafe (ok boot -F failsafe) will detect an out of sync boot archive.  Execute the automated update then reboot.  This will now boot from the later kernel (139555-08) which successfully installed from the finish script.

I reproduced the problem in a jumpstart installation environment applying the latest 10_Recommended patch cluster from a finish script. The initial installation was S10U5 which is deployed from a miniroot that has no knowledge of a boot archive (my theory anyway).  This is similar to a live upgrade environment if the boot environment doing the patching is also boot archive unaware (meaning the kernel is pre 137137-09).

In the jumpstart scenario the immediate problem was solved by updating the boot archive by booting failsafe as previously described.  The Solution was to update the boot archive from the finish script after the patch cluster installation completed.  BTW, all patches in the patch cluster installed successfully per the /var/sadm/system/logs.finish.log.

In a standard jumpstart the boot device (install target) is mounted to /a, therefore adding the following entry to the finish script solved the problem:

/a/boot/solaris/bin/create_ramdisk -R /a

Depending on the finish script configuration, and variables the following would also work:

$ROOTDIR/boot/solaris/bin/create_ramdisk -R $ROOTDIR

Issue 3: This above issues are sometimes mis-diagnosed as CR 6850202: "bootadm fails to build bootarchive in certain configurations leading to unbootable system".

But CR 6850202 will only be encountered in very specific circumstances, all of which must occur in order to hit this specific bug, namely:

1. Install u6 SUNWCreq - there's  no mkisofs so we build ufs boot archive

2. Limit /tmp to 512M - thus forcing the ufs build to happen in /var/run

3. Have a separate /var - bootadm.c only lofs nosub mounts "/" when creating the alt root for DAP patching build of boot archive

4. Install 139555-08

You must have all 4 of above in order to hit this, i.e. step 4 must be installing a DAP patch such as a Kernel patch associated with a Solaris 10 Update such as 139555-08. 

Solution 3: Removing the 512MB limit (or whatever limit has been imposed) to /tmp in /etc/vfstab and/or adding SUNWmkcd (and probably SUNWmkcdS) so that mkisofs is available on the system is sufficient to avoid the code path that fails this way.

Booting failsafe and recreating the boot archive will successfully recreate the boot archive.

Here's further input from one of my senior engineers, Enda O'Connor:

If using Live Upgrade (LU), and LU on the live partition is up to date in terms of latest revision of the LU patch, 121430 (SPARC) and 121431 (x86), the boot-archive will be built automatically once users runs shutdown ( after luactivate to activate the new BE ).  This is done from a kill script in rcd.0.

If using a jumpstart finish script, or jumpstart profile to patch a pre-U6 image with latest kernel patches, then you need to run create_ramdisk from the finish script after all patching/packaging operations have been finished.  Alternatively, you can patch your pre-U6 miniroot to the U6 SPARC NewBoot level (137137-09), at which point the modified miniroot will handle the build of the boot_archive after the finish script has run.

If patching U6 and upwards from jumpstart, the boot_archive will get built automatically after finish script has run, so there's no issue in this scenario.

If using any home grown technology to patch or install/modify software on an Alternate Boot Environment ( ABE ), such as ufsrestore/cpio/tar for example, you must always run create_ramdisk manually before booting to said ABE.

Best Wishes,

Gerry.

The Zones Parallel Patching enhancement for the Solaris 10 patch utilities was released this week giving customers a choice of how to improve zones patching performance.

In the Zones "Update On Attach" section of a previous blog posting, I mentioned that the Zones "Update On Attach" feature could also be used to improve Zones patching perfomance.

Zones Parallel Patching is a true patching solution utilizing the 'patchadd' utility.  

Whereas Zones "Update On Attach" uses zones functionality similar to that used during zones creation to provide a pseudo-patching solution that does not utilize 'patchadd'. 

So which one to choose ?

Let's look at the two options in more detail:

Zones Parallel Patching

Zones Parallel Patching is an enhancement to the standard Solaris 10 patch utilities and is delivered in the patch utilities patch, 119254-66 (SPARC) and 119255-66 (x86).

Simply install this patch, set the maximum number of non-global zones to be patched in parallel in the config file /etc/patch/pdo.conf, and away you go.

It works for all Solaris 10 systems. 

It also works well in conjunction with higher level patch automation tools such as xVM Ops Center. 

It can dramatically improve zones patching performance by patching non-global zones in parallel.  The global zone is still patched first.

While the performance gain is dependent on a number of factors, including the number of non-global zones, the number of on-line CPUs, the speed of the system, the I/O configuration of the system, etc., a performance gain of ca. 300% can typically be expected for patching the non-global zones - e.g. On a T2000 with 5 sparse root non-global zones.

See my previous Zones Parallel Patching blog entry for further information.

Since it's a pure enhancement to 'patchadd', it's normal 'patchadd' functionality.  You can subsequently remove patches using 'patchrm', etc.  Nothing has changed except that it's now much faster to patch non global Zones with Zones Parallel Patching invoked.

Zones "Update On Attach"

The primary purpose of Zones "Update on Attach" is Zones migration from one server to another.  

For example, a database instance in a non-global zone hosted on a server has grown to the extent that the Sys Admin wants to transfer it to a better spec'd server which can better handle the workload.   The Sys Admin can detach it from the old server (e.g. a Sun4u) and reattach it to the new server (e.g. a Sun4v) using Zones "Update On Attach".   This will bring the OS Software level on the non-global zone up to the same level as the new server's global zone.

Zones "Update On Attach" can certainly be used for patching but there are limitations you need to be aware of as outlined below.

For example, detach the non-global zones from a system, apply a bunch of patches to the global zone, reattach the non-global zones using "Update On Attach" and viola, the non-global zones will be brought up to the same software level as the global zone (for OS type packages), effectively patching the non-global zones without using 'patchadd' at all.   This is typically even faster than using Zones Parallel Patching.  But there are limitations to this approach which users must be aware of (see below).

My senior engineer, Enda O'Connor, has just published an interesting article on The Zones Update on Attach Feature and Patching in the Solaris 10 OS. 

Zones "Update On Attach" limitations as a patching aid

Zones "Update On Attach" only works for packages which are SUNW_PKG_ALLZONES=true - i.e. typically OS level packages, and not application packages.

So when to use Zones Parallel Patching in 'patchadd' and when to use Zones "Update On Attach" ?

Here's what my senior engineer, Enda O'Connor, says:

"The Zones Update on Attach Feature and Patching in the Solaris 10 OS document may help customers understand how the technology works, applying a cluster via patching and via zones Update On Attach is not quite the same really.

It really depends on the patches being applied, i.e. applying a firefox patch via Update On Attach would not work if you wanted it to apply to the global zone and all non-global zones as well.

One has to understand how Update On Attach works and then apply that to the list of patches to see if it gets them to a desirable state.

There is no black or white answer here.

I'd recommend Zones Parallel Patching using 'patchadd' as it has a known outcome all the time, whereas Update On Attach makes it's own internal determination based on a number of things, that can vary from system to system ( e.g. inherited directories ).

But if time to patch is critical then if the customer does proper testing to validate things, and are happy with the results, then by all means use Update On Attach.

But using Update On Attach without:

1. Understanding how it determines what packages to update

2. Not inspecting the patches being applied.

...will most likely lead to grief at some point."

And my other senior engineer, Ed Clark, says:

"In terms of giving guidance on which technology to use, there are a number of considerations -- two of these considerations are:

1. Using Update On Attach to update sparse zones can require significantly more disk storage space than would be needed by applying patches with 'patchadd' (3-4 times as much space would not be uncommon i think), due to Update On Attach copying fully populated global zone 'undo' files into the non-global zones, as opposed to having patchadd build sparsely populated 'undo' files in the non-global zones.

2. If a customer is really concerned about the ability to back out patches reliably, then 'patchadd' is a lower risk option than Update On Attach -- 'patchrm' of a patch from a non-global zone that has a copy of the global zones 'undo' pkg data (as is the case after Update On Attach) may potentially have unexpected side effects." [although we have yet to see any actual cases of negative results from this.]

Conclusion

In general, we recommend using the Zones Parallel Patching enhancement in the patch utilities rather than the Zones "Update On Attach" feature as Zones Parallel Patching is standard patching functionality, only faster, whereas Zones "Update On Attach" is really designed for migrating zones from one server as another and was not primarily designed to speed up patching.  

Because Zones "Update On Attach" uses Zones functionality similar to the zone creation functionality, rather than 'patchadd' functionality, limitations exist on what will be patched (typically the OS but not applications) and there's the potential for anomalies around things like the "undo" files which would be used by 'patchrm' if patches applied using Zones "Update On Attach" were subsequently removed from the non-global zones using 'patchrm' (although we have yet to see any actual cases of serious issues resulting from this).

So in patching situations where time is absolutely critical, Zones "Update On Attach" may provide a good option, as long as it's well tested in the customer environment prior to deployment on production systems.

Remember too, Live Upgrade is also your friend in such situations, enabling you to patch an inactive boot environment while the system is still in production.   So a combination of Live Upgrade and Zones Parallel Patching would be ideal.

I hope you find this helpful!

Best Wishes,

Gerry.

The Solaris 10 5/09 (Update 7) patch bundle is now available for download from the SunSolve Patch Cluster & Patch Bundle Download Page.  Click on the "Solaris Update Patch Bundles" link.

As with previous patch bundles, it contains the patches which are included in the corresponding Solaris Update, in this case Solaris 10 5/09 (Update 7).

This is useful for Sys Admins who wish to bring all their systems up to the same patch level as the Solaris Update without wanting to upgrade to the release - for example, due to change control policy restrictions in their organizations.

See previous blog entries for previous Solaris Update patch bundles for further information.

The Zones Parallel Patching feature is now available in the latest Solaris 10 patch utilities patch, 119254-66 (SPARC) and 119255-66 (x86).

This is available for use on all Solaris 10 systems. 

Simply install this patch, set the maximum number of non-global zones to be patched in parallel in the config file /etc/patch/pdo.conf, and away you go.

Prior to this feature, each non-global zone was patched sequentially, leading to unnecessarily long patching times for zones systems.  (Sequential patching remains the default behavior unless the config file is edited to enable Zones Parallel Patching.)

With this feature invoked, the global zone continues to be patched first, but then the non-global zones can be patched in parallel, leading to significant performance gains in patching operations on Zones systems.

While the performance gain is dependent on a number of factors, including the number of non-global zones, the number of on-line CPUs, the speed of the system, the I/O configuration of the system, etc., a performance gain of ca. 300% can typically be expected for patching the non-global zones - e.g. On a T2000 with 5 sparse root non-global zones.

Here's the relevant note from the patch README file:

NOTE 10: 119255-66 is the first revision of the patch utilities to deliver "zones parallel patching".
          This new functionality allows multiple non-global zones to be patched in parallel by patchadd.
          Prior to revision 66, patchadd would patch all applicable non-global zones sequentially,
          that is one after another. With zones parallel patching, a sysadmin can now set the number
          of zones to patch in parallel in a new configuration file for patchadd called /etc/patch/pdo.conf.

         The two factors that affect the number of non-global zones that can be patched in parallel are
         1. Number of on-line CPUs
         2. The value of num_proc in /etc/patch/pdo.conf

          If the value of num_proc is less than or equal to 1.5 times the number of on line CPUs,
          then patchadd limits the maximum number of non-global zones that will be patched in
          parallel to num_proc. If the value of num_proc is greater than 1.5 times the number of on line CPUs,
          then patchadd limits the maximum number of non-global zones that will be patched in parallel
          to 1.5 times the number of on line CPUs. Note that patchadd will patch all applicable non-global
          zones on a system, the above description outlines only how patchaadd determines the
          maximum number of job slots to be used during parallel patching of non-global zones.

          An example of this in operation would be where:
          num_proc=8
          and number of on line CPU's is 4

          In this case the maximum setting for num_proc would be 6, that is the maximum number
          of zones that could be patched in parallel is 6.  If there are more than this number of non-global zones on the
          system, the first 6 will be patched in parallel, then the remaining non-global zones will be patched
          as processes finish patching the first 6 non-global zones.   Only one patch process will be used for each
          non-global zone, so if there are less than 6 non-global zones on the system, then only the number of processes
          equal to the number of non-global zones will be initiated.

          Please see comments in /etc/patch/pdo.conf for more details on setting num_proc.

I would like to thank Ed Clark and Enda O'Connor from my own team for all their work in developing and testing Zones Parallel Patching.

I would also like to thank Jon Bowman, Arindam Sarkar, and the rest of the RPE (Sustaining) Install team for all their work in getting this feature integrated into the patch utilities and delivered to production.

I would also like to thank our selected key customers who kindly Beta tested the feature for us.

I believe this feature is an important milestone in improving our customers' patching experience in a Zones environment as it addresses a long standing customer complaint on Zones patching performance.

Enjoy!

The new PatchFinder tool is now available on http://sunsolve.sun.com/patchfinder/. 

It's linked off the main SunSolve Patch page, http://sunsolve.sun.com/show.do?target=patchpage.  Look for the following link immediately under the old PatchFinder search box:

The PatchFinder

Why a new PatchFinder tool ?

The old PatchFinder tool was a pet peeve of mine.  You needed to know at least the 6 digit base PatchID of the patch you were trying to find in order to find it.   Rather self defeating IMHO.

The new PatchFinder tool directly leverages Sun's internal Patch Metadata Web Services to provide a much richer search experience.

Features of the new PatchFinder tool

You can still search by PatchID if you want.  This will override all other search options.

But you can also search for all Recommended or Security patches, and restrict that search, for example, to Solaris 10 SPARC.

By the way, "Recommended" means it's part of the Solaris Recommended Patch Cluster, which contains the latest revision of all Solaris OS patches which fix Security, Data Corruption, or System Availability issues.  See the cluster inclusion criteria definitions by clicking the appropriate heading on the Patch Clusters & Patch Bundles download page, http://sunsolve.sun.com/show.do?target=patch-access.

"Security" includes all patches which address Security issues, including Solaris OS patches and application and middleware patches for other products.

If you click the "OS Patches Only" box, the search results can be restricted to patches for the Solaris OS only, which will exclude application and middleware patches which are not bundled as part of the Solaris OS.  

Caveat: Note, the interpretation of what is a Solaris OS patch used by the tool is currently about 97% accurate.  This is due to difficulties interpreting patches for applications and middleware which may be bundled with the Solaris OS.  (The old PatchFinder tool and SunSolve patch reports have the same issue.) Hence you can see anomalies between search results returned for "Recommended" and "Recommended & OS Patches Only" which should be identical.  A subsequent version will have a 100% accurate interpretation of what is a Solaris OS patch.

Advanced Search Capabilities

Click on "Show Advanced Search" for more options.

This gives you options such as searching by CR (Change Request, a.k.a. BugID) number, so if you suspect you've hit a particular bug, you can check whether a patch for that CR is available yet.

Or you can search for patches with particular words in the patch synopsis or keywords fields - e.g. ldap, "patch util", "package util", "pkg util", etc.  These options have limited value as it's difficult to guess the values.

The "Released Before" option is handy if your company has a policy of waiting for patches to "age" a specified number of days after release before you consider applying them.

The "Released After" option is useful to restrict the search to patches released since the last time you checked for patches.

The "README Modified After" option is subtly different to the "Released After" field and is a superset of the "Released After" results in that is also shows patches whose README or patchinfo metadata files have been updated since the patch was initially released - for example, Special Install Instructions may have been added to the README to specify workarounds for issues found post-release which do not warrant the patch being withdrawn from SunSolve (i.e. the patch still does more good than harm for the majority of customers).

You can filter the search further to see only those patches whose README file was modified since you last downloaded patches by using the following search filter combination: For example, if you downloaded patches 30 days ago, you can see which patches which were release 30 or more days ago have had their READMEs modified since then by using the combination: "Released Before" == 30 && "README Modified After" == 30

In all of these time related fields, you can specify actual dates instead of a specified number of days.

If you don't have a support contract, click on the "Public Patches Only" box to restrict the search to patches which are available for download without a support contract.

The "Patch Property" field enables you to search for things like "Interactive" patches which require manual intervention during installation, "NonStandard" which means they aren't applied using the standard 'patchadd' utility (e.g. firmware patches), or patches which require downtime (Single User Mode, Reboot*) if applied to the live boot environment.  (Remember, Live Upgrade can be used to minimize the downtime and risk associated with applying patches by applying the patches to an inactive boot environment, thereby avoiding such downtime requirements during or immediately after patch installation.  You can reboot to set the inactive boot environment live at a time that suits you.)

By default, only patches which are currently available for download (i.e. patches which haven't been withdrawn due to issues) are returned in the search results.    You can select "Withdrawn" patches instead to get a list of patches which have been withdrawn from SunSolve due to serious issues.   This is useful to ensure you don't have any withdrawn patches installed on your systems.  I recommend you also select "Show Obsoletes" along with "Withdrawn" so withdrawn patches which have been superseded by replacement good patches aren't masked.  (Note, a Sun Alert is issued whenever a patch is withdrawn, so if you keep abreast of Sun Alert notifications as is advisable, this step is simply a check and balance.)

Fields such as "OS Release", "State", etc., allow multiple options to be selected concurrently from the drop down menu.

Patch Metrics Gathering 

The new PatchFinder tool is also useful for helping you to calculate patch metrics - e.g. the number of Solaris 10 SPARC OS patches released in the last year.

Or, if you feel so inclined, you can use the new PatchFinder tool to calculate the percentage of Solaris OS patches which are publicly available.   Select an OS Version, select "OS Patches only", and search with and without the "Public Patches Only" box selected to get the number of publicly available patches and the total number of available patches respectively.   To save you the trouble, the percentage of Solaris OS patches which are publicly available without a support contract today (May 27th 2009) are 25% for Solaris 10 OS patches, 28% for Solaris 9 OS patches, and 31% for Solaris 8 OS patches.

Display and Bookmarking Options

You can also select the number of patches to display in each page of search results returned (default 20), hide the search form so that only the results are displayed (the option is in the top right hand corner of the tool), and order the results by PatchID, Released date, or Synopsis, in either ascending or descending order (by clicking on the appropriate column heading of the results returned).

You can click on a PatchID in the search results returned to display the Patch README.

You can also bookmark the search results returned for future reference.  This is handy if you wish to run the same query regularly. 

Help! 

There's a "Help" summary in the top right hand corner and each search field has it's own help summary marked "?".

What's next ? 

I hope you find this initial version of the new PatchFinder tool useful.

This is a start, not the finished article.   In future versions we plan to provide options to resolve patch dependencies and patch installation order, enable patch download, etc.  

Feedback - what else would you like to see ?

Feel free to provide feedback on features which you'd like to see to the software-update-finder-feedback@sun.com alias or directly to me, Gerry.Haskins@sun.com .  

Our goal is to improve your patching experience.

My colleague, Enda O'Connor, has published 3 more patching articles on Big Admin which I hope you will find useful:

• Analyzing a patchadd or patchrm Failure in the Solaris OS

  How to gather and analyze the necessary information to determine why a 'patchadd' or 'patchrm' session failed.
  

• How to Split a Root Mirrored With Solaris Volume Manager Prior to Updating Software

  How to split a Solaris Volume Manager mirror prior to updating the software through upgrades or patches.

• How to Remove a Solaris Patch While Booted From a Network or CD-ROM

  How to run 'patchrm' from a network or CD-ROM against a non-live system that is mounted from such media, and how to mount file systems under the mount point after the system is booted from the media.

I think the first article is particularly useful to help customers and support engineers understand what data to gather to enable analysis of a patching issue.  Even if you are not able to analysis the issue yourself, providing this data to Sun Support when you log a call will help speed up the issue analysis by Sun.

Solaris 10 5/09 (Update 7) and subsequent Kernel PatchIDs 

The Kernel patch included in Solaris 10 5/09 (Update 7) has now been released to SunSolve.  The PatchIDs are 139555-08 (SPARC) and 139556-08 (x86).  The rest of the patches included in Solaris 10 5/09 are either released or in the process of being released over the next week or so. 

I've updated the Solaris Kernel PatchID sequence listed in http://blogs.sun.com/patch/date/20080416 to reflect this, including the PatchIDs of the post Solaris 10 5/09 (Update 7) sustaining Kernel PatchID and the Solaris 10 Update 8 Kernel PatchID.

We will be releasing a patch bundle containing the set of patches included in Solaris 10 5/09 in the next couple of weeks.  This will be available from the "Solaris Updates Patch Bundle" section on the new look SunSolve Patch Cluster and Patch Bundle page, http://sunsolve.sun.com/show.do?target=patches/patch-access, which now includes a description of the purpose, contents, and update frequency of each patch cluster and bundle.

As always, customers need to have a valid support contract in order to download Solaris patch clusters and bundles.

New PatchFinder tool coming

We plan to release a new PatchFinder tool on SunSolve at the end of May.   This leverages the patch metadata in our release database to provide a much richer customer experience to search for patches.  Further enhancements are planned after the initial deployment.  The old PatchFinder tool will remain available for a transition period.

Zones Parallel Patching Performance Enhancement

The Zones Parallel Patching performance enhancement continues on schedule.  It has been successfully beta tested by a number of key customers who confirm a 3x performance improvement patching zones.   It is on schedule to be released in a revision of the patch utilities patch (119254 SPARC / 119255 x86) in June.

Solaris 10 "Recommended" and Sun Alert Patch Clusters

Improvements to the Solaris 10 "Recommended" and Sun Alert Patch Clusters are on schedule to be deployed before the end of June.  The improvements include an improved install_cluster script (currently available in the Solaris 10 Live Upgrade Zones Starter Patch Bundle and the Solaris 10 Updates Patch Bundles) and other process improvements designed to improve quality.

Solaris 8 Vintage Patch Support Program

The Solaris 8 Vintage Patch Support Program is now up and running.  The first Vintage Solaris 8 patches have been released.

Patches which fix SunAlerts which were issued prior to the April 1 start date of the Solaris 8 Vintage Patch Support program will be released as "normal" (i.e. non-vintage) Solaris 8 patches. 

But all other Solaris 8 patches created after April 1, including patches which fix security issues, require customers to have a Solaris 8 Vintage Patch Support contract to use them.

See http://www.sun.com/bigadmin/topics/vintagepatch/  for further information.

A software glitch caused a number of empty patches to be accidentally released last week.   The PatchIDs are:

• 139555, revisions -01 to -07: Solaris 10: Kernel Patch
• 139556, revisions -01 to -07: Solaris 10_x86: Kernel Patch
• 139981-01: Solaris 10_x86: md patch

The above patches have been removed from SunSolve.   All of these patches were empty (i.e. they contained no payload packages), so they are incapable of being installed or causing any problems on a customer system.   This notice is simply to clear up any confusion.

The correct Kernel patch revisions are now available.  These are:

• 139555-08, which is the Solaris 10 Kernel patch included in Solaris 10 5/09 (Update 7)
• 139556-08, which is the Solaris 10 Kernel patch included in Solaris 10_x86 5/09 (Update 7)

The correct "md" patch revision will be released shortly.  This is:

• 139981-03: Solaris 10_x86: md patch

On behalf of Sun, I apologize most sincerely for any confusion or inconvenience caused. 

My colleague, Jeff Victor, has written an excellent and informative blog posting on how to improve patching performance, "patching zones goes zoom!".   Enjoy!

Here's a Solaris patch presentation (updated July 28th 2009) which I hope will be of use to customers, partners, and field engineers.

It distills much of the information contained in this blog, including the recommended patching strategy, best practices, news on patching enhancements, background information on Solaris and the bug fix process, etc.

Patching isn't a one-size-fits-all operation, but I hope you find the information provided a useful guideline.

My team and I have been working with the SunSolve team on improvements to the Patch Cluster pages on SunSolve.  These improvements went live on April 20, 2009.

The old "Recommended Patch Clusters" and "Recommended and Security Patches" pages have been combined into a single Patch Cluster & Patch Bundle Downloads page.

A Notice Board section at the top of the page will be used to alert customers to current issues.

Click on the cluster headings to see a brief description of the purpose of the cluster, with links to view the cluster README as well as a download link.   The date the cluster was last updated and the size of the cluster are also shown.

No change has been made to the underlying cluster file names, so scripts using 'wget' to access the patch clusters should be unaffected.

This is part of an ongoing effort to improve our patch presentation to customers.

As before, customers need a valid support contract in order to be able to access patch clusters. 

If you are not registered in Member Support Center, simply log into SunSolve and associated one or more support contracts with your Sun Online Account using the "Change Contract" option in the top right hand menu.

If you are registered in Member Support Center, your contracts will be automatically associated with your account (and the "Change Contract" option will not be shown when you log into SunSolve).

A fix to the unzip utility is available in recent patch utility patch revisions.  This fix is required in order to be able to successfully unzip very large files such as the Solaris 10 Recommended and Sun Alert Patch Clusters.

Please download the latest revision of the patch utilities patch first and install it, before attempting to unzip the Solaris 10 Recommended or Sun Alert Patch Clusters.

The fix was incorporated in the putback to CRs 6344676 and 6464056.

The following are the earliest revisions of the patch utilities containing the fix:

• Solaris 10 SPARC: 119254-46 or above
• Solaris 10 x86:        119255-46 or above
• Solaris 9 SPARC:   112951-14 or above
  
• Solaris 9 x86:          114194-11 or above
  
• Solaris 8 SPARC:   108987-19 or above
  
• Solaris 8 x86:          108988-19 or above
  

Without the fix to unzip provided by the above patches, the following error will be seen when attempting to unzip the Solaris 10 Patch Clusters:

# unzip -q 10_Recommended.zip

note:  didn't find end-of-central-dir signature at end of central dir.
  (please check that you have transferred or created the zipfile in the
  appropriate BINARY mode and that you have compiled UnZip properly) 

In addition, do not unzip Solaris patch clusters on Windows. Solaris patch clusters, and solaris patches more generally, can contain case-sensitive file names. Consequently clusters and patches must be unzipped on a case-sensitive filesystem (corruption can occur if unzipping on filesystems that are not case-sensitive). 

The above information is now published in Infodoc http://sunsolve.sun.com/search/document.do?assetkey=1-62-252447-1

I've been asked to post a clarification: 

You cannot patch Solaris 10 from Solaris 8 or 9 as the version of 'patchadd' in Solaris 8 and 9 is totally unaware of how to handle Zones and other Solaris 10 specific features.

If using Live Upgrade to upgrade an inactive boot environment from Solaris 8 or 9 to Solaris 10, you must activate and boot into the Solaris 10 boot environment before patching it.  For example, activate and boot into the Solaris 10 boot environment, and either patch the live boot environment or create another inactive boot environment, and then apply patches to the inactive boot environment.

See http://www.sun.com/bigadmin/features/articles/live_upgrade_patch.jsp for further information.

Thanks to my colleague Enda O'Connor, who has made p7zip available for Solaris 8 SPARC, it's now possible to upgrade directly from Solaris 8 SPARC to the latest Solaris 10 Update releases such as Solaris 10 5/08 and Solaris 10 10/08.  See http://sunsolve.sun.com/search/document.do?assetkey=1-9-250526-1 and http://sunsolve.sun.com/search/document.do?assetkey=1-61-72099-1 for details.

Previously, due to the lack of p7zip on Solaris 8, customers needed to perform an interim upgrade to Solaris 9 or an earlier Solaris 10 release before upgrading to the latest Solaris 10 release.

The following is now published as Infodoc 208057:

To whom it may concern,

Over time, customers may experience space issues in /var as patch "undo" and "obsolete" data, which is stored for use during patch backout, builds up as more patches are applied to the system.

It is best to plan for this expansion of /var in advance,  and allocate plenty of free space to this partition. 

How much space to allocate to /var depends on a number of variables, such as how many patches are likely to be applied to the system, which is dependent on frequency of patching, strategy of which patches will be applied (all, sun alert, security only, etc.), products patched; whether the system has zones which implies 'pspool' space requirements, etc.  A figure of 10Gb to 20Gb or even more is not unreasonable.   In my Patch System Test lab, we currently have Solaris 10 systems with >7GB used in /var and this will continue to grow over the lifetime of Solaris 10.

If you have insufficient space in /var of an existing system, the recommended solution is to extend the size of the /var partition.  This can be accomplished by backing up /var, creating a bigger /var partition on another disk, restoring the /var backup onto that partition and then updating the /var entry in the /etc/vfstab file.  If /var is part of the "/" (root) partition, then relocating /var to a separate partition would be a good solution.  The relocation process is documented in Infodoc 215988.

Sun strongly discourages manual modifications to the /var/sadm/pkg directory and its file structure.  That directory structure should only be modified by the Solaris[TM] patching and packaging utilities.  Corruption in this directory structure will prevent any future packaging or patching operations.

Below is information on how to free up space in /var by deleting certain patch related files.  This should only be used when there are no other ways of gaining space in /var.

Having a full and current backup of the OS is highly recommended before deleting files from /var.

On behalf of Sun Microsystems, I hereby confirm that unneeded "obsolete" and "undo" files may be removed from the /var/sadm/pkg/<pkgname>/save/<PatchID> and "undo" files from the /var/sadm/pkg/<pkgname>/save/pspool/<pkgname>/save/<PatchID> directories in order to free up space in /var.

This will not invalidate support contracts.

The "undo" files are used during the patch removal process. Removing the "undo" files associated with a patch means that it will no longer be possible to uninstall the patch.

Once a later revision of a patch is applied, or a patch which obsoletes a patch is applied, the "undo" files in /var/sadm/pkg/<pkgname>/save/<PatchID> are renamed "obsolete".  These "obsolete" files may also be removed.

It is strongly recommended to only remove the "obsolete" and "undo" files for patches which the customer is confident will not need to be backed out - for example rejuvenated Kernel patches associated with Solaris 10 Update releases such as 118833-36 (SPARC) / 118855-36 (x86), 120011-14 (SPARC) / 120012-14 (x86), etc. and other older Kernel patch revisions.  See the sequence of Solaris 10 Kernel PatchIDs listed on http://blogs.sun.com/patch/date/20080416

As patches containing subsequent fixes to the objects contained in these rejuvenated patches will have a SUNW_REQUIRES dependency on the these rejuvenated patches, it's extremely unlikely that older versions of these rejuventated Kernel patches will ever be backed out.   Therefore, their "undo" files are prime candidates for removal to free up space in /var if required and it's not possible to extend /var.

All the "undo" or "obsolete" files for a particular patch should be removed from the /var/sadm/pkg/<pkgname>/save/<PatchID> directories. For example:

rm /var/sadm/pkg/*/save/118833-36/undo*

The system also keeps a copy of "undo" files for use during creation of new non-global zones. These may also be removed if the customer is confident that the patches will not need to be backed out. For example:

rm /var/sadm/pkg/*/save/pspool/*/save/118833-36/undo*

The "undo" files in the pspool directory are not renamed to "obsolete" when a later revision is installed.

Further Information:

1. The "undo" files are specific to the zone of the system on which they were created during application of a patch by patchadd. Therefore, do not copy the "undo" file from one zone to another or from one system to another.

2. One could potentially archive the "undo" files for each zone of a system and restore it to that zone if desired.

Best Wishes,

Gerry Haskins
Director, Software Patch Services