Albert White's Weblog

Hi,

Now that we are all in Oracle it seems appropriate that the personal entries on this blog should go. At least I don't see any blogs over at http://blogs.oracle.com/ talking about Astronomy, Football, Photography, etc. !

So I've removed all non Sun related posts on this blog and migrated everything over to my personal blog on http://www.cademuir.eu.

Cheers, ~Albert 

We've had some eventful days. We had fun and we kicked butt as Scott would say. We changed the world. We were part of the greatest technology company the world has ever seen. We changed computing forever - repeatedly.

And in a few minutes we'll learn what the future holds with Oracle. I hope Oracle is ready for Sun!

Opensolaris users may be familiar with browsing repositories in firefox. To look through the latest Develpoment repo for example you just open up http://pkg.opensolaris.org/dev in your browser.

Things are a little more complicated for the extras and support repos though.

Firstly you need to register to get access to these repos. Anyone can get access tot he extra repo, only supported customers can get access to the support repo. Go to http://pkg.sun.com/register and follow the instructions there to get your key and certificate and verify that you can connect to the repo through the pkg command.

To set up firefox to be able to browse the repo take a little more work. Danek Duvall from the IPS team provided these instructions on how to do it:

Run:

openssl pkcs12 -in /var/pkg/ssl/OpenSolaris_extras.certificate.pem \
    -inkey /var/pkg/ssl/OpenSolaris_extras.key.pem -export > \
    /tmp/OpenSolaris_extras.certificate.pkcs12

In the case of the support repo use the support key and cert in place of the extras ones above instead. That will prompt you for a password with which to encrypt the pkcs12 file.

Now in firefox add the  pkcs12 file: Edit -> Preferences -> Advanced -> Encryption -> View Certificates -> Your Certificates -> Import -> choose file (/tmp/OpenSolaris_extras.certificate.pkcs1) -> enter password.

Then point your browser at https://pkg.sun.com/opensolaris/extra/ (or https://pkg.sun.com/opensolaris/support for the support repo).  There's a dialog box that pops up saying that the site has requested you identify yourself with a cert, and gives you a list of possible certs to use. Choose the right one, click OK, and then you can browse the repo.

I hit an interesting problem tonight with jumpstart. Or old timeserver has gone away and the jumpstart clients are now going into interactive installs asking for the user to set the time. We rely heavily on automated installs so this needed to be fixed.

The solution was obvious I thought. I'll just set up one of our servers as a ntp server and tell the jumpstart clients to query that in the sysidcfg files.

The only problem is that jumpstart doesn't query ntp. After snooping on the server for a while it was clear that the packets reqesting the time were not NTP, they were TIME.

Heres how I diagnosed it.

First snoop the install.

snoop -v -o /tmp/snoop.op clientname

Then once your install has gone interactive you can convert that to a readble format:

snoop -i /tmp/snoop.op -v > /tmp/snoop.op.as

Examining the file you can find the time request:

TCP:  Source port = 32773
TCP:  Destination port = 37 (TIME)

<snip>
TCP:
TIME:  ----- TIME:   -----
TIME:
TIME:  ""
TIME:

So, whats port 37 exactly? /etc/services tells us that the time server runs there. (duh!)

The service that runs this is in Solaris 10 svc:/network/time:stream

On solaris 10 you need to do

svcadm enable svc:/network/time:stream

To check that is working ok you can telnet to the server and see if you get any output; if its not running you will get connection refused. This is basically what your jumpstart client is doing.

$ telnet patchtest-231 37
Trying 129.156.231.103...
Connected to patchtest-231.
Escape character is '^]'.
���Connection to patchtest-231 closed by foreign host.

We are now back to fully automated jumpstart installs!

Recently I had a discussion with some folks about ways to identify change in a workspace. In particular if there were ways where we could judge the risk associated with changes without needing to know the specifics of the changes or being told by the engineers.

In Opensolaris for example there are flag days. These coincide with putbacks where a project team has identified major change and tells you about it. We have something similar for Solaris Update releases. Sometimes this is great, if there is a big zones or zfs change for example we know to check patching extra carefully on systems using zones or zfs. However this isn't always enough. Every now and again there will be a putack that causes a regression somewhere and catches us all by surprise.

Before getting to involved in looking into this problem in detail we did what all good engineers do. Go and see if someone else has solved the problem already! And that's when I got distracted. You see I started wondering if there was some way to visualise the changes to a workspace and literally see where risk was introduced.

That led me to Michael Ogawa's page. There he has several videos produced from code swarm. In the videos the names of the engineers are displayed and the files that they are hanging are represented by dots that swarm around them. Now while this isn't really what I started out looking for it does allow you to see the number of files changed over time. More importantly Michael's videos looked cool so I thought I'd give it a go for Opensolaris.

Codeswarm is available from http://code.google.com/p/codeswarm . It will generate lots of png files which you can then use ffmpeg to make into a movie.

There was one problem though; it doesn't work with mercurial workspaces out of the box. However  Baptiste Lepilleur worked out a way to get a compatible xml file from a mercurial repository.

Anyway here are a couple of videos I made. The first is of the Image Packaging System. The music is from Dom The Bear (CC by-sa)

Image Packaging System Code Swarm.


Next up the ON gate! Music this time from Alexander Blu (CC by-sa). Vimeo will only let me embed the SD version here - visit it's Vimeo page if you want to see the HD version; its worth watching in HD imo. While you are there you can search for other code swarm videos - there are nearly 100 up there.


Opensolaris Code Swarm.

Recently my Manager started blogging. Despite my initial cynicism it's actually turning out to be a pretty good blog, and the comments are great.

Communication about patches is an area that Sun could improve in. What they are, how they are created, how they can be installed, when they can be installed, and when and what you should patch are all areas that I've received customer queries about.

Patch Automation Tools is Gerry's most commented post to date. And I'm not surprised. And to be honest I agree with most of the comments - pca is damn good. Hopefully Sun Connection Satellite will be a big improvement on previous offerings.

Live Upgrade is a feature of Solaris that lets you create alternate boot environments. This makes it easy to switch between OS builds at boot time, but also make upgrading much easier, less risky, and quicker. This extends to patching too.  

I recently received a query from a customer asking how we ensure that patches installed via live upgrade do not interfere with the running system. As well as ensuring that the patch applies correctly to your alternative boot environment you need to be sure that the patch is not changing any files or killing processes on tour running system.

 In Solaris 8 and 9 we use an interposition library to check this. We check all the open*, creat*,*link* calls to ensure that they are dealing with files on the correct boot environment; we allow changes in /tmp etc. and commands also need to load libraries from the running environment so we make exceptions for these. We also check the kill calls to ensure that processes are not being killed on the running system. An interposition library is one that is usually preloaded using LD_PRELOAD so that when a call is searched for the call as defined in our library will be matched rather than the system call. Heres a snippit of how we check for creat calls:

 
int
creat(char *path, mode_t mode)
{
        char *cwd;
        char *cmdname="creat";
        typedef (*realcreat_t)(char *p, mode_t m);
        static realcreat_t prealcreat;
        if (prealcreat == NULL){
                prealcreat=  (realcreat_t)dlsym(RTLD_NEXT, "creat");
                if (prealcreat== NULL){
                        (void) printf("dlopen: %s\n", dlerror());
                        return (0);
                }
        }
        parsepstname(path,cmdname);
        return ((*prealcreat)(path, mode));
} 
  
 

Our creat() call takes the same arguments as the system call. The first thing we do is look for the real system call by calling dlsym(3C) and we store it. We then write out the file thats being created to a log file and call the real creat() call. The parsepstname() function works out the full path to the file and then filters out our exceptions (/tmp etc).

Similar functions need to be written for any calls that we want to examine.

One issue we came up against when designing this was that shell script often call /sbin/sh when they need to run other scripts. /sbin/sh is statically linked so our interposition library will not work. In the case of pkgadd the environment was also being cleared. We get around these problems by catching the call to execute /sbin/sh, reloading our environment variables from a file and then execing /bin/sh instead. It works but it's a bit invasive. Also if we need to make changes to the test we need to recompile the library and reinstall it on the test machines. If only there was some way to dynamically trace what was happening on the system...

Well in s10 we can use dtrace for this. The procedure is basically the same; we check for certain system calls, filter out exceptions and flag an error if something is happening that should not be. Heres the dtrace script 

#!/usr/sbin/dtrace -qs

int x;
BEGIN{
/* set it to something that wont match a pid for
   the syscall prov. below */
x=-1;
}

/* The process that we are interested in */
proc:::create
/execname == "patchadd" || execname == "patchrm"/
{
        x=pid;
        self->called_proc_create = 1;
}

syscall::open*:entry,
syscall::creat*:entry,
syscall::unlink*:entry,
syscall::link:entry,
syscall::symlink:entry
/progenyof(x)/
{
     self->path = copyinstr(arg0);
     printf("%s:%s:%s:%s\n", probefunc, self->path, cwd, execname);
}

We check for patchadd and patchrm processes being started and note the pid. Although you use the luupgrade command to do the patching it ultimately calls patchadd and patchrm to do the work. Then when we examine a system call we check that it is from the patchadd process tree with the progenyof() test. If it is we log the function and arguments. Rather than having dtrace handle the parsing we have a perl script in our test harness that filters out the exceptions and warns us of any errors.

We also check for kill calls in Solaris 10, but if a patch needs to start or stop a process it should really do so by svcadm. So we check expecially for any calls to that:

proc:::exit
/execname == "svcadm"/
{   printf("%s:%d:%s:%s\n", probefunc, arg0, execname,execname);
}

The dtrace is much more straightforward and easier to implement. It's also tracing everything so we don't have to worry about someone clearing the environment or calling statically linked commands.

This test has caught quite a few problems in patches. The majority of these are down to errors in the patch and package scripts where patch creators are allowed to write their own scripts; sometimes these are written by product teams that have not considered patching in a live upgrade scenario.  We rarely see any issues with this test anymore. It seems that once we introduce a test we get an initial peak in test fails, the issues are fed back upstream and corrected and we then see a steady tailoff in failures.

There is a blueprint available on Patching mirrored systems using live upgrade. This document will take you through the steps needed to create an alternate boot environment and how to patch it.

Why bother? Well by using Live Upgrade you can drastically reduce downtime as you are applying patches to a non-running boot environment. You just need one reboot to make everything active. If you run into problems you can easily reboot back into the unpatched environment.

A couple of comments that didn't make it into the final doc:

1. I would always recommend using the '-c' option to lucreate to label your current boot environment. lucreate will try and give it a sensible name (d0 in the blueprint examples), but naming this yourself makes things clearer.

2. Solaris 10 can order patches automatically for you. So if you just want to add all the patches in a directory you don't need the order file. 'cd /path/to/patch; luupgrade -t -n "New_BE" -s /path/to/patch *' will do the job.

3. The blueprint focusses on the EIS CD, however you can also use LU to just apply a single patch. eg 'luupgrade -n newroot -t -s /export/patches 987654-32'

4. If for some reason you need to remove a patch LU can do this for you also using 'luupgrade -T'.

Even if you don't currently use mirrored root filesystems it is worth taking a look at this blueprint as the Live Upgrade methodology is the same for all systems.

This question came up on the solarisx86 yahoogroup recently:

What about a patch for a large package, say the video drivers. Might some files patched in an earlier release not have needed further patching and thus have been skipped over by applying the most current release?

So if you apply say rev-03 of a patch are you getting all the fixes from -01 and -02 also?

The answer is yes. All patch revisions are cumulative so the files and fixes from previous revisions will be in the latest revision.

This also holds for accumulated/obsoleted patches. If a patch obsoletes another patch it will then contain all the files that the obsoleted patch contains. Not only that but if there are any scripts that need to be ran the obsoleting patch will have to merge in these scripts.

This was actually the source of a lot of problems for the latest s10 KU patch,  118833-36. It had accumulated so much change from other patches that it was almost impossible to get it to install. Many of the patches that comprised the KU were easy enough (relatively!) to install on their own, but once they all accumulated into the KU it became much harder to get the patch to work. A Quick glance at the readme of KU-36 will give you an idea of the complexity involved. There are good reasons why all these other patches get accumulated into the KU and they most boil down to interdependencies - a zones patch for example will need the KU but changes in the KU will only work if the zones patch is installed also - so if you use zones you need to merge the patches. We are working on ways to ensure that adding a patch like 118833/55-36 will not be so painful for customers in future!

This situation became rather silly in Solaris 8 when we had a situation where almost everything was included in the KU. When your Kernel Update patch starts patching apache you know things are going out of control. At that point we decided to split the KU back up again in a process called rejuvenation.

With rejuvenation the KU is effectively split into smaller patches again, but each requires the previous KU. The new patches have new patchid's. So you will have to install the latest KU, but subsequent KU's will have different patchid's and require that the old KU be running on the system. The catch is that it is not possible to ever uprev the old KU since we would risk overwriting files delivered in the rejuvenated patches.

Live Upgrade is a technology in Solaris that allows you to easily upgrade a system while not risking change to your running environment. You can easily revert back to your previous environment if you need to. BigAdmin has a great guide on setting up Live Upgrade.

I've mentioned before about the problems of patching live systems. With the latest Solaris 10 KU for example most of the effort went into crafting the patch scripts so that the patch could be applied to a running system. When you patch a live system for example you have to take care about replacing libraries so that the commands you are delivery and the ones on the running system will continue to work, 118833-36 does some clever loopback mounting of libraries for this. With Live Upgrade this problem goes away.

Thats why patch testers and developers like LU. For users the benefits are that patching is easier - you don't have to be in single user mode. Its safer - you can instantly return to your previous environment. And its quicker - the system can be patched while running and your downtime is just the time it takes to reboot to the new environment.

So how do you do it?

Set up an alternate boot environment.
For
this you need to have some space allocated for the boot environment
(BE), obviously it needs to be big enough to store the OS! lucreate(1M) is used to create our BE.

# lucreate -c "Current_BE" -m /:/dev/dsk/c0t0d0s1:ufs -n "New_BE"

This names our current BE "Current_BE" and the BE we want to create as
"New_BE". The root filesystem is installed on /dev/dsk/c0t0d0s1 that
will be the root filesystem of the New_BE. The Bigadmin LU guide explains more detailed options for lucreate.

Now to patching!

We use the luupgrade(1M) command to do patching. The simplest way to add a patch to our New_BE is to do:

# luupgrade -t -n "New_BE" -s /path/to/patch patchid

The -s argument indicating the patch directory is required. Adding multiple
patches is easy too. If you want to add all the patches in a directory
you can just do:

# luupgrade -t -n "New_BE" -s /path/to/patch *

luupgrade does allow you to use an order file, but since the patches
will be ordered automatically in Solaris 10 this is redundant. For
older OS's you could do this s recommended by Big Admin:

# luupgrade -t -n "New_BE" -s /path/to/patch -O  "-M /path/to/patch patch_order"

The -O option arguments are passed directly to patchadd. To remove patches we use the -T option:

# luupgrade -T -n "New_BE" patchid

Under the Hood

luupgrade re really just mounting the alternate BE, New_BE in our case, and then calling patchadd -R <mountpoint> to install the patches.

The -R argument causes, in the simplest cases, causes reloc directory from each package in the patch to be copied to /<mountpoint> rather than /.

In patch scripts the value of the -R argument is stored as $ROOTDIR, it is / is -R is not specified. So if you want your patch to say update an editable file you operate on ${ROOTDIR}/etc/file. Its good practice to prefix all filenames with ${ROOTDIR}.  Also if you are only interested in changing the running system, for example stopping and starting a service you need to check that $ROOTDIR is / before performing any operation.

At the package level the mount point is exported as $BASEDIR and $PKG_INSTALL_ROOT to the package level scripts. Similarly all paths in these scripts should be prefixed with $BASEDIR or $PKG_INSTALL_ROOT.

A wile ago some of our test suites were failing due to erroneous requests to port 80 on the test machines. The simplest thing to do was to just block requests to anything other than our webserver in the lab using ipfilter.

 pass in quick on bge2 proto tcp from any to <webserver IP> port = 80
 block in quick on bge2 from any to any port = 80

bge2 is our external interface.

The bit I got stuck on was logging any attempts. ipfilter allows you to log against any rule. When using syslog the default facility is local0 and the default levels are

LOG_INFO

Packets logged using the log keyword as the action rather than pass or block.

LOG_NOTICE

Packets logged that are also passed.

LOG_WARNING

Packets logged that are also blocked.

LOG_ERR

Packets that have been logged and that can be considered “short”.

You can alter these defaults in the rule for example as:

block in log level auth.info quick on bge2 from any to any port = 80

but the defaults are fine for me so we just use:

block in log quick on bge2 from any to any port = 80

Turning on logging will cause ipfilter to log to  /dev/ipl. You can use ipmon(1M) to monitor this, or use it to log to syslog via 'ipmon -Ds'.

 If you use syslog you need to define where the log should go as not everything automatically goes into /var/log/syslog. You need to add a line like this into /etc/syslog.conf:

local0.debug                                    /var/log/ipflog

You cannot say 'local.*' in syslog.conf, * only is valid for facilities. If you try you'll get an error of 'unknown priority name "*"'. You also cannot use spaces you must use tabs, otherwise you'll get an error like 'unknown priority name "debug /var/log/ipflog"'. Once this is defined correctly, touch /var/log/ipflog and restart syslog. You should then see connection attempts logged to this file. Since we are using the default logging above for our block rule they will be logged as local0.warning. The tabs and the * had me stuck for a while so hopefully this post will save someone else from some head scratching.

 

When I did an interview for my current job in Sun I was asked what I didn't like about the company. Having been the sysadmin for some machines in college I hated Suns security patching policy. A vulnerability would be posted to bugtraq and the students would soon start trying to exploit the vulnerability. Some source code patch would come out for other OS's but the best you would get from Sun was a workaround. Eventually, often weeks/months later, a patch would come out.

A couple of years back we came up with IDR's. The initial idea was to provide a way for engineers to deliver diagnostic binaries to customers to help solve their issues in a way that would be recorded on the system. Rather than giving the customer a tarball the customer could now get a 'patch'. IDR's show up in 'patchadd -p' and also block any patches from being installed on top of them. It was quickly realised that this method solved the problem of getting quick security fixes out to customers.

 

On Sunday a telnet vulnerability came up on opensolaris-discus. Alan and Dan  have described a bit about how the issue was fixed. The code in opensolaris was fixed within hours and posted to opensolaris-discus. By this-morning Irish time patches for Solaris 10 were submitted and ready for testing and soon afterward were pushed to the team responsible for getting them onto sunsolve. The patches are now available - 12006[89]-02.

Two things have struck me from this experience:

1) Opensolaris. Someone posted the vulnerability and a Sun engineer was online and acted on it. Having the discoverer contact Sun privately would have been preferred I guess. But once the vulnerability was out there opensolaris was ready to fix it!

2) Sun has got a hell of a lot better at patching security vulnerabilities. It's gone from the months that I remember to 48 hours for a fully tested and supported patch. And there are probably places where a couple more hours could have been shaved off. Congrats to all involved.

ok 3 things.

3) stop using telnet. Use ssh. Then run 'netservices limited' ! :-)

 

 

 

The Patch System Test lab, like many other labs, has a mixture of different machines, some with different means of connecting up the serial console to them. The older Sun boxes have 25 pin connectors, some others have 9 pin serial connectors, and newer machines have RJ45 connectors. For the most part people will use some sort of terminal server for getting the consoles on these machines. To get the console you basically need to telnet to a port on the device and ensure that it is correctly wired to the host. A map of  server:port to host can be maintained so users know how to get to the host without needing to bother about the connection it uses.

Then along come domains on 4800's, network management ports, and system controllers etc. Now in order for users to get access to a hosts console they need to know what type of machine it is and connect appropriately.

A couple of years ago I installed conserver to replace several scripts that we had previously. Using this people can issue one command to get a console on a machine and not need to worry about the kind of machine it is. Granted if they connect to say an X4200 they will still need to know how to use it's SC, but they don't need to know how they are supposed to connect to it.

The conserver service runs on a regular machine. It has a configuration file that tells it how to connect to the hosts, more on that later. When the service starts it connects to all of the hosts in the configuration file. A user who starts the conserver client program is then connected to the host they require through the conserver service. The client program in our lab resides on the server, but I run it from my desktop and connect to the service host in the lab.

Aside from the benefits I've already mentioned there are a couple of other useful features.

You can 'view' a console. Multiple people can have the same console open at a time, but only one is attached in write mode. This is useful for keeping an eye on machines and also as a training aid.

Conserver also logs all serial traffic so you can see why a machine crashed for example.

That should be enough to convince you that this software is worth looking at, so lets look at the implementation.

Installing conserver is a fairly straightforward './configure;make;make install'.

The tricky bits, and they aren't that tricky, come with configuration.

Firstly you'll need to add

           console      782/tcp    conserver    # console server

to /etc/services.

Next you'll need to create a password file, regardless of whether or not you intend to use passwords. So our password file just lists usernames

bearass(5.9)$ head /export/PST/etc/conserver.passwd
albertw:
john:

Next comes the actual configuration script that defines where the consoles are and how to get to them.

Well start with the default settings that define the basic operation:

### set up global access
default full    { rw *; }

# Default Settings
default * {        
	# The '&' character is substituted with the console name        
	logfile /var/consoles/&;        
	# timestamps every hour with activity and break logging        
	timestamp 1hab;        
	# include the 'full' default        
	include full;        
	# master server is localhost        
	master localhost;
}

In PST we have two brands of terminal server. Some are from MRV and others are Perle CS9000's. The both work the same way, you telnet to a specific port, but the port numbers each uses is different. What we do next in the configuration script is define how the ports on these units are numbered so that later on when we list a host we can just say that it is plugged into say port 5 of a unit, and not have to worry about what port it is.

# Basic Settings for the perle CS9000's

# Basic Settings for the perle CS9000's
default perle {        
	type host;        
	baud 9600;        
	parity none;        
	portbase 10000;        
	portinc 1;
}

# Basic Settings for MRV console boxes
default mrv {        
	type host;        
	baud 9600;        
	parity none;        
	portbase 2000;        
	portinc 100;
}

So the CS9000 starts at port 10000 and increments in 1's ¿ 10001, 10002 etc. The MRV ports are 2000, 2100, 2200 etc.

Now we specify what type of unit each of our terminal servers is:

default pst-console-03 {include mrv; host pst-console-03;}
default pst-console-04 {include perle; host pst-console-04;}

Finally we can define the hosts themselves:

console beetle.ireland.sun.com { include pst-console-03; port 3;}
console cocaine.ireland.sun.com { include pst-console-04; port 4; }

Eventhough the hosts are connected to different terminal servers we use the same syntax to list them.

There are other machines, such as X4200's and v20z's that have network management ports that you ssh into to get console access. In those cases its just a matter of getting the servers ssh keys on the host SP so that ssh logins without the need for passwords work. Then the host can be added to the configuration as:

console patchtest-x4200-4 { type exec;  exec /bin/ssh patchtest-x4200-4-sp -l root; }

That covers our basic usage of conserver. The documentation also mentions being able to compile in support for tcp_wrappers and openssl for more secure connections, but thats not something I've played with.

www.cybuscorporation.com - Runs the SunOne webserver.
Cybus founded in 1982, Sun Microsystems founded 1982.
Participation age eh?

Perhaps its too easy to get too stuck into Dr. Who! :-)

Long live the Daleks!

I had a bit of a cold this weekend, so didn't leave the house much. In fact I didn't leave my bed much. To pass the time I decided to learn Java. As you do.

First stop was netbeans.org to download the IDE. I could have just started writing the code in vi but I wanted to see if these IDE's are as cool as people tell me. After running through the quickstart guide I see some of why people rave about IDE's. You can rename a class and its updated everywhere! You can move classes around and their packages get updated! This probably sounds very obvious to many of you. But I've been writing in php and ksh for the last while and neither have a netbeans. With php you have vi and var_dump() as your debugger.

Second stop was to topcoder.com. Topcoder is basically a code competiton site. However all the previous competitons are archived and you can view and submit code for the problems. So I took a stab at one. And hours later I managed to get code that passed all the tests. It was a simple test, that I got right on nearly my first attempt. In theory. In practice I got a lesson in number types in Java.
1/2=0.5 right?
What I was doing was:
       
        int a=1;
        int b=2;
        float c=0;
        c=a/b;
        System.out.println("c: " + c);

And got c: 0.0 !

It turns out that if you divide an integer by an integer in Java, you get an integer, regardless of what type you are assigning the answer to. The integer then appears to be typecast into whatever you declared. This confused me at first but it makes some sense. If you declare integers Java assumes that you want to work in integers. A lot of integer arithmetic would get slow, and possibly wrong, if it was converting to floats. This way you are sure that integer arithmetic works.

The solution to my problem. Declare a and b as floats or doubles, then you get the right answer.

Hopefully this will help someone from falling into the same issue!

Technorati Tag: Java