Wednesday March 21, 2007

Are you suffering from patch pain?

Patching is a hot issue for me right now. More and more of our customers are struggling to install ever larger and more complex patches, especially in a production environment. In this blog entry, I'll guide you through what's happened, why it's happened, what Sun is doing about it and how you can make your own life easier too!

Prior to Solaris 10, when a version of Solaris was released, the scale of change allowed in that release was very constrained. Usually the only changes we'd make to the code were for bug fixes and, very occasionally, support for a new platform. As a result, the Solaris patches released by Sun contained predominantly corrective code.

That all changed for Solaris 10.

Now we have significant features being integrated into Solaris, primarily intended for Solaris 10 updates...except, of course, many of these features also get delivered through the corresponding patches because they're built from the same Solaris 10 source code.

Unfortunately, there have been two serious ramifications from this approach...

1. Some of the features touch a significant proportion of the code, creating a big web of dependencies that force all those dependencies into the same patch. That's why the kernel patch is so big!

2. Some of the features introduce interface and/or behavioural changes that don't play nicely with the running system performing the 'patchadd', this makes it inordinately difficult for us to build a stable, installable KU. It's also the main reason for the 79 special notes at the end of the kernel patch README :(

So what are we doing about this?

Firstly, we're "rejuvenating" the kernel update which splits out all the dependencies so that, with the last KU installed, subsequent patches can be smaller once again.

Secondly, we're reviewing the integration review processes to ensure that appropriate verification is in place to ensure that features are tested in the "patch install" context prior to integration.

Thirdly, we're identifying enhancements and bug fixes to the patch and packaging utilities that will help them cope better with features being delivered in patches.

There's also something else that you can do for yourself - configure your production systems for Live Upgrade.

To be honest, Sun has not done a good job at encouraging the adoption of live upgrade and that's a shame. Anyone running in a production environment should be taking a very long, hard look at live upgrade because it enables you to apply patches or upgrade your Solaris release while still in production.

Because live upgrade applies the changes to an "alternate boot environment" (effectively an offline clone of your currently booted Solaris disk image), you don't need to be in single user mode to apply the KU - in fact, you can install a whole cluster of patches with your server still running in full production mode...just reboot into the updated alternate boot environment and you're away!

Furthermore, should you encounter any problems, you can reboot back to your original boot environment straight away - no lengthy downtime while you repair the damage. A huge amount of the pain associated with installing patches and managing your maintenance windows goes away when you use Live Upgrade.

I strongly urge everyone to reconfigure to enable Live Upgrade. You'll be glad you did!

(2007-03-20 23:30:39.0) Permalink Comments [2]