Albert White's Weblog

Opensolaris users may be familiar with browsing repositories in firefox. To look through the latest Develpoment repo for example you just open up http://pkg.opensolaris.org/dev in your browser.

Things are a little more complicated for the extras and support repos though.

Firstly you need to register to get access to these repos. Anyone can get access tot he extra repo, only supported customers can get access to the support repo. Go to http://pkg.sun.com/register and follow the instructions there to get your key and certificate and verify that you can connect to the repo through the pkg command.

To set up firefox to be able to browse the repo take a little more work. Danek Duvall from the IPS team provided these instructions on how to do it:

Run:

openssl pkcs12 -in /var/pkg/ssl/OpenSolaris_extras.certificate.pem \
    -inkey /var/pkg/ssl/OpenSolaris_extras.key.pem -export > \
    /tmp/OpenSolaris_extras.certificate.pkcs12

In the case of the support repo use the support key and cert in place of the extras ones above instead. That will prompt you for a password with which to encrypt the pkcs12 file.

Now in firefox add the  pkcs12 file: Edit -> Preferences -> Advanced -> Encryption -> View Certificates -> Your Certificates -> Import -> choose file (/tmp/OpenSolaris_extras.certificate.pkcs1) -> enter password.

Then point your browser at https://pkg.sun.com/opensolaris/extra/ (or https://pkg.sun.com/opensolaris/support for the support repo).  There's a dialog box that pops up saying that the site has requested you identify yourself with a cert, and gives you a list of possible certs to use. Choose the right one, click OK, and then you can browse the repo.

I hit an interesting problem tonight with jumpstart. Or old timeserver has gone away and the jumpstart clients are now going into interactive installs asking for the user to set the time. We rely heavily on automated installs so this needed to be fixed.

The solution was obvious I thought. I'll just set up one of our servers as a ntp server and tell the jumpstart clients to query that in the sysidcfg files.

The only problem is that jumpstart doesn't query ntp. After snooping on the server for a while it was clear that the packets reqesting the time were not NTP, they were TIME.

Heres how I diagnosed it.

First snoop the install.

snoop -v -o /tmp/snoop.op clientname

Then once your install has gone interactive you can convert that to a readble format:

snoop -i /tmp/snoop.op -v > /tmp/snoop.op.as

Examining the file you can find the time request:

TCP:  Source port = 32773
TCP:  Destination port = 37 (TIME)

<snip>
TCP:
TIME:  ----- TIME:   -----
TIME:
TIME:  ""
TIME:

So, whats port 37 exactly? /etc/services tells us that the time server runs there. (duh!)

The service that runs this is in Solaris 10 svc:/network/time:stream

On solaris 10 you need to do

svcadm enable svc:/network/time:stream

To check that is working ok you can telnet to the server and see if you get any output; if its not running you will get connection refused. This is basically what your jumpstart client is doing.

$ telnet patchtest-231 37
Trying 129.156.231.103...
Connected to patchtest-231.
Escape character is '^]'.
���Connection to patchtest-231 closed by foreign host.

We are now back to fully automated jumpstart installs!

Recently I had a discussion with some folks about ways to identify change in a workspace. In particular if there were ways where we could judge the risk associated with changes without needing to know the specifics of the changes or being told by the engineers.

In Opensolaris for example there are flag days. These coincide with putbacks where a project team has identified major change and tells you about it. We have something similar for Solaris Update releases. Sometimes this is great, if there is a big zones or zfs change for example we know to check patching extra carefully on systems using zones or zfs. However this isn't always enough. Every now and again there will be a putack that causes a regression somewhere and catches us all by surprise.

Before getting to involved in looking into this problem in detail we did what all good engineers do. Go and see if someone else has solved the problem already! And that's when I got distracted. You see I started wondering if there was some way to visualise the changes to a workspace and literally see where risk was introduced.

That led me to Michael Ogawa's page. There he has several videos produced from code swarm. In the videos the names of the engineers are displayed and the files that they are hanging are represented by dots that swarm around them. Now while this isn't really what I started out looking for it does allow you to see the number of files changed over time. More importantly Michael's videos looked cool so I thought I'd give it a go for Opensolaris.

Codeswarm is available from http://code.google.com/p/codeswarm . It will generate lots of png files which you can then use ffmpeg to make into a movie.

There was one problem though; it doesn't work with mercurial workspaces out of the box. However  Baptiste Lepilleur worked out a way to get a compatible xml file from a mercurial repository.

Anyway here are a couple of videos I made. The first is of the Image Packaging System. The music is from Dom The Bear (CC by-sa)

Image Packaging System Code Swarm.


Next up the ON gate! Music this time from Alexander Blu (CC by-sa). Vimeo will only let me embed the SD version here - visit it's Vimeo page if you want to see the HD version; its worth watching in HD imo. While you are there you can search for other code swarm videos - there are nearly 100 up there.


Opensolaris Code Swarm.

There is a blueprint available on Patching mirrored systems using live upgrade. This document will take you through the steps needed to create an alternate boot environment and how to patch it.

Why bother? Well by using Live Upgrade you can drastically reduce downtime as you are applying patches to a non-running boot environment. You just need one reboot to make everything active. If you run into problems you can easily reboot back into the unpatched environment.

A couple of comments that didn't make it into the final doc:

1. I would always recommend using the '-c' option to lucreate to label your current boot environment. lucreate will try and give it a sensible name (d0 in the blueprint examples), but naming this yourself makes things clearer.

2. Solaris 10 can order patches automatically for you. So if you just want to add all the patches in a directory you don't need the order file. 'cd /path/to/patch; luupgrade -t -n "New_BE" -s /path/to/patch *' will do the job.

3. The blueprint focusses on the EIS CD, however you can also use LU to just apply a single patch. eg 'luupgrade -n newroot -t -s /export/patches 987654-32'

4. If for some reason you need to remove a patch LU can do this for you also using 'luupgrade -T'.

Even if you don't currently use mirrored root filesystems it is worth taking a look at this blueprint as the Live Upgrade methodology is the same for all systems.

There was an interesting segment on RTE's "Capital D" program tonight about the work that Shamrock Rovers do in the community. The video feed is at http://www.rte.ie/news/2007/0426/capitald.html The initiative from Shamrock Rovers include providing scolarships for payers to gain 3rd level qualifications. Just in case they dont make millions playing football! Also some schoolkids are featured whose training gear, school books and uniforms are paid for by Shamrock Rovers. Shamrock Rovers - Building The Future in Tallaght!

Speaking of Tallaght  the Tallaght Stadium saga rolls on. The next court date is in early May. Since my last blog about the stadium the GAA club Thomas Davis have been making some extraordinary moves. In an 11 page memo they informed their members that the local TD Conor Lenihan was barred from their club. Conor found out about this while members were buying him drinks at the clubhouse to celebrate a €200,000 grant from the government which Conor helped secure! Needless to say banning an elected representative from their club got quite a bit  of media coverage. The coverage prompted this reply in the Independents letters pages:

The Tallaght stadium saga


I would like to congratulate your paper for its balanced and comprehensive coverage of the on-going Tallaght stadium saga (Irish Independent, April 18).


It is about time the public were given a chance to take on board the true facts of the situation.


While local GAA activists are determined to portray victimisation at the hands of the local authority and the Minister for Sport, the
reality is that it is, and never was, practical to accommodate senior Gaelic games in the as-yet completed stadium. Trying to squeeze a square peg into a round hole springs to mind.


Its anchor tenants, Shamrock Rovers, have no objection to the playing of underage GAA events there as this will not contravene plans, which date back more than ten years, to develop the ground into a modern community facility for all the people of Tallaght.


For those who live or work close to the partially completed stadium and who are curious as to why it remains an ugly concrete shell, it is important that they are made aware that if it wasn't for the continued court actions of local GAA club Thomas Davis, then maybe the sports loving people of the area would have a top-of-the-table football match at the venue to look forward to this weekend.


Unfortunately, for now, it remains to be seen how long they will have to wait before they see Tallaght Community Stadium finally put to good use.

This question came up on the solarisx86 yahoogroup recently:

What about a patch for a large package, say the video drivers. Might some files patched in an earlier release not have needed further patching and thus have been skipped over by applying the most current release?

So if you apply say rev-03 of a patch are you getting all the fixes from -01 and -02 also?

The answer is yes. All patch revisions are cumulative so the files and fixes from previous revisions will be in the latest revision.

This also holds for accumulated/obsoleted patches. If a patch obsoletes another patch it will then contain all the files that the obsoleted patch contains. Not only that but if there are any scripts that need to be ran the obsoleting patch will have to merge in these scripts.

This was actually the source of a lot of problems for the latest s10 KU patch,  118833-36. It had accumulated so much change from other patches that it was almost impossible to get it to install. Many of the patches that comprised the KU were easy enough (relatively!) to install on their own, but once they all accumulated into the KU it became much harder to get the patch to work. A Quick glance at the readme of KU-36 will give you an idea of the complexity involved. There are good reasons why all these other patches get accumulated into the KU and they most boil down to interdependencies - a zones patch for example will need the KU but changes in the KU will only work if the zones patch is installed also - so if you use zones you need to merge the patches. We are working on ways to ensure that adding a patch like 118833/55-36 will not be so painful for customers in future!

This situation became rather silly in Solaris 8 when we had a situation where almost everything was included in the KU. When your Kernel Update patch starts patching apache you know things are going out of control. At that point we decided to split the KU back up again in a process called rejuvenation.

With rejuvenation the KU is effectively split into smaller patches again, but each requires the previous KU. The new patches have new patchid's. So you will have to install the latest KU, but subsequent KU's will have different patchid's and require that the old KU be running on the system. The catch is that it is not possible to ever uprev the old KU since we would risk overwriting files delivered in the rejuvenated patches.

The following press release has just been issued by Shamrock Rovers F.C. It touches on some of the points I made in my last post, and its good to see the club officially responding to allegations made recently by some journalists.

Shamrock Rovers and the Tallaght Community Stadium

For all press related matters please contact Shamrock Rovers Press Officer John Byrne
by mobile on 087-768-1408 or by emailing
johnbyrne@shamrockrovers.ie.

Live Upgrade is a technology in Solaris that allows you to easily upgrade a system while not risking change to your running environment. You can easily revert back to your previous environment if you need to. BigAdmin has a great guide on setting up Live Upgrade.

I've mentioned before about the problems of patching live systems. With the latest Solaris 10 KU for example most of the effort went into crafting the patch scripts so that the patch could be applied to a running system. When you patch a live system for example you have to take care about replacing libraries so that the commands you are delivery and the ones on the running system will continue to work, 118833-36 does some clever loopback mounting of libraries for this. With Live Upgrade this problem goes away.

Thats why patch testers and developers like LU. For users the benefits are that patching is easier - you don't have to be in single user mode. Its safer - you can instantly return to your previous environment. And its quicker - the system can be patched while running and your downtime is just the time it takes to reboot to the new environment.

So how do you do it?

Set up an alternate boot environment.
For
this you need to have some space allocated for the boot environment
(BE), obviously it needs to be big enough to store the OS! lucreate(1M) is used to create our BE.

# lucreate -c "Current_BE" -m /:/dev/dsk/c0t0d0s1:ufs -n "New_BE"

This names our current BE "Current_BE" and the BE we want to create as
"New_BE". The root filesystem is installed on /dev/dsk/c0t0d0s1 that
will be the root filesystem of the New_BE. The Bigadmin LU guide explains more detailed options for lucreate.

Now to patching!

We use the luupgrade(1M) command to do patching. The simplest way to add a patch to our New_BE is to do:

# luupgrade -t -n "New_BE" -s /path/to/patch patchid

The -s argument indicating the patch directory is required. Adding multiple
patches is easy too. If you want to add all the patches in a directory
you can just do:

# luupgrade -t -n "New_BE" -s /path/to/patch *

luupgrade does allow you to use an order file, but since the patches
will be ordered automatically in Solaris 10 this is redundant. For
older OS's you could do this s recommended by Big Admin:

# luupgrade -t -n "New_BE" -s /path/to/patch -O  "-M /path/to/patch patch_order"

The -O option arguments are passed directly to patchadd. To remove patches we use the -T option:

# luupgrade -T -n "New_BE" patchid

Under the Hood

luupgrade re really just mounting the alternate BE, New_BE in our case, and then calling patchadd -R <mountpoint> to install the patches.

The -R argument causes, in the simplest cases, causes reloc directory from each package in the patch to be copied to /<mountpoint> rather than /.

In patch scripts the value of the -R argument is stored as $ROOTDIR, it is / is -R is not specified. So if you want your patch to say update an editable file you operate on ${ROOTDIR}/etc/file. Its good practice to prefix all filenames with ${ROOTDIR}.  Also if you are only interested in changing the running system, for example stopping and starting a service you need to check that $ROOTDIR is / before performing any operation.

At the package level the mount point is exported as $BASEDIR and $PKG_INSTALL_ROOT to the package level scripts. Similarly all paths in these scripts should be prefixed with $BASEDIR or $PKG_INSTALL_ROOT.

A wile ago some of our test suites were failing due to erroneous requests to port 80 on the test machines. The simplest thing to do was to just block requests to anything other than our webserver in the lab using ipfilter.

 pass in quick on bge2 proto tcp from any to <webserver IP> port = 80
 block in quick on bge2 from any to any port = 80

bge2 is our external interface.

The bit I got stuck on was logging any attempts. ipfilter allows you to log against any rule. When using syslog the default facility is local0 and the default levels are

LOG_INFO

Packets logged using the log keyword as the action rather than pass or block.

LOG_NOTICE

Packets logged that are also passed.

LOG_WARNING

Packets logged that are also blocked.

LOG_ERR

Packets that have been logged and that can be considered “short”.

You can alter these defaults in the rule for example as:

block in log level auth.info quick on bge2 from any to any port = 80

but the defaults are fine for me so we just use:

block in log quick on bge2 from any to any port = 80

Turning on logging will cause ipfilter to log to  /dev/ipl. You can use ipmon(1M) to monitor this, or use it to log to syslog via 'ipmon -Ds'.

 If you use syslog you need to define where the log should go as not everything automatically goes into /var/log/syslog. You need to add a line like this into /etc/syslog.conf:

local0.debug                                    /var/log/ipflog

You cannot say 'local.*' in syslog.conf, * only is valid for facilities. If you try you'll get an error of 'unknown priority name "*"'. You also cannot use spaces you must use tabs, otherwise you'll get an error like 'unknown priority name "debug /var/log/ipflog"'. Once this is defined correctly, touch /var/log/ipflog and restart syslog. You should then see connection attempts logged to this file. Since we are using the default logging above for our block rule they will be logged as local0.warning. The tabs and the * had me stuck for a while so hopefully this post will save someone else from some head scratching.

Some of the regulars from geocachingireland.com were featured doing what they do best on RTE's Nationwide TV Programme tonight. The 7 minute clip from the programme is on the Nationwide website.

When I did an interview for my current job in Sun I was asked what I didn't like about the company. Having been the sysadmin for some machines in college I hated Suns security patching policy. A vulnerability would be posted to bugtraq and the students would soon start trying to exploit the vulnerability. Some source code patch would come out for other OS's but the best you would get from Sun was a workaround. Eventually, often weeks/months later, a patch would come out.

A couple of years back we came up with IDR's. The initial idea was to provide a way for engineers to deliver diagnostic binaries to customers to help solve their issues in a way that would be recorded on the system. Rather than giving the customer a tarball the customer could now get a 'patch'. IDR's show up in 'patchadd -p' and also block any patches from being installed on top of them. It was quickly realised that this method solved the problem of getting quick security fixes out to customers.

On Sunday a telnet vulnerability came up on opensolaris-discus. Alan and Dan  have described a bit about how the issue was fixed. The code in opensolaris was fixed within hours and posted to opensolaris-discus. By this-morning Irish time patches for Solaris 10 were submitted and ready for testing and soon afterward were pushed to the team responsible for getting them onto sunsolve. The patches are now available - 12006[89]-02.

Two things have struck me from this experience:

1) Opensolaris. Someone posted the vulnerability and a Sun engineer was online and acted on it. Having the discoverer contact Sun privately would have been preferred I guess. But once the vulnerability was out there opensolaris was ready to fix it!

2) Sun has got a hell of a lot better at patching security vulnerabilities. It's gone from the months that I remember to 48 hours for a fully tested and supported patch. And there are probably places where a couple more hours could have been shaved off. Congrats to all involved.

ok 3 things.

3) stop using telnet. Use ssh. Then run 'netservices limited' ! :-)

The Patch System Test lab, like many other labs, has a mixture of different machines, some with different means of connecting up the serial console to them. The older Sun boxes have 25 pin connectors, some others have 9 pin serial connectors, and newer machines have RJ45 connectors. For the most part people will use some sort of terminal server for getting the consoles on these machines. To get the console you basically need to telnet to a port on the device and ensure that it is correctly wired to the host. A map of  server:port to host can be maintained so users know how to get to the host without needing to bother about the connection it uses.

Then along come domains on 4800's, network management ports, and system controllers etc. Now in order for users to get access to a hosts console they need to know what type of machine it is and connect appropriately.

A couple of years ago I installed conserver to replace several scripts that we had previously. Using this people can issue one command to get a console on a machine and not need to worry about the kind of machine it is. Granted if they connect to say an X4200 they will still need to know how to use it's SC, but they don't need to know how they are supposed to connect to it.

The conserver service runs on a regular machine. It has a configuration file that tells it how to connect to the hosts, more on that later. When the service starts it connects to all of the hosts in the configuration file. A user who starts the conserver client program is then connected to the host they require through the conserver service. The client program in our lab resides on the server, but I run it from my desktop and connect to the service host in the lab.

Aside from the benefits I've already mentioned there are a couple of other useful features.

You can 'view' a console. Multiple people can have the same console open at a time, but only one is attached in write mode. This is useful for keeping an eye on machines and also as a training aid.

Conserver also logs all serial traffic so you can see why a machine crashed for example.

That should be enough to convince you that this software is worth looking at, so lets look at the implementation.

Installing conserver is a fairly straightforward './configure;make;make install'.

The tricky bits, and they aren't that tricky, come with configuration.

Firstly you'll need to add

           console      782/tcp    conserver    # console server

bearass(5.9)$ head /export/PST/etc/conserver.passwd
albertw:
john:

### set up global access
default full    { rw *; }

# Default Settings
default * {        
	# The '&' character is substituted with the console name        
	logfile /var/consoles/&;        
	# timestamps every hour with activity and break logging        
	timestamp 1hab;        
	# include the 'full' default        
	include full;        
	# master server is localhost        
	master localhost;
}

# Basic Settings for the perle CS9000's
default perle {        
	type host;        
	baud 9600;        
	parity none;        
	portbase 10000;        
	portinc 1;
}

# Basic Settings for MRV console boxes
default mrv {        
	type host;        
	baud 9600;        
	parity none;        
	portbase 2000;        
	portinc 100;
}

default pst-console-03 {include mrv; host pst-console-03;}
default pst-console-04 {include perle; host pst-console-04;}

console beetle.ireland.sun.com { include pst-console-03; port 3;}
console cocaine.ireland.sun.com { include pst-console-04; port 4; }

console patchtest-x4200-4 { type exec;  exec /bin/ssh patchtest-x4200-4-sp -l root; }

Ah the participation age!

Here (live!) at the IFAS stand at the
Connacht Star
Party we have nearly live 3D images from Mars.


Midnight
Mars Browser is  Java application that can download
images from the two Nasa rovers on Mars.  So we downloaded images
from the surface for the past couple of days, some of which were taken
only hours previously. And they were in 3D. We run the risk of making
3D glasses popular again!


All naturally enough running on my solaris laptop. Leo Enrights talk
today I noticed was also presented in OpenOffice.

Anyway, the comment was about the future of patching now that ZFS is alive.

One thing that ZFS allows you to do is manage snapshots of your filesystem. A snapshot is described as a read only copy of the filesystem. Unlike a traditional backup which would require twice the space, a snapshot only get blocks of data copied over when they change on the live filesystem.

Writeable snapshots are called clones apparently (I'm new to ZFS ok!). Actually you clone an existing snapshot.

Finally there are the backup and restore options. You can create a full or incremental backup of a snapshot. Consequently you can restore a full directory or just the incremental changes.

What is interesting for patching is that in the future it may be possible to use ZFS to handle backing out patches. If you were to take a snapshot, then apply a patch, then the snapshot contains all the files that have changed since you applied the patch. Including all the metadata changes to the package database. You would then take a backup of that snapshot. To backout the patch, just restore this partial snapshot.

Thats an oversimplification. More care would be needed not to restore something that another patch had changed, like the pkginfo file of a package. But for such defined shared files (/var/sadm/pkg/*/pkginfo for example) it should be possible to write a tool to correctly merge the running and restored versions to correctly reflect the system state.

Everything you need to know about ZFS is linked from http://www.opensolaris.org/os/community/zfs/. Of particular interest are zfs(1M), one manpage is really all you need, and this pdf explains the basics of the filesystem, why its cool, and how it works - read this first.

Congratulations to the ZFS team! And I look forward to many lengthy email conversations when we have to generate patches for this!

Technorati Tag: Solaris
Technorati Tag: OpenSolaris
Technorati Tag: zfs