John Brady's Weblog

Tuesday August 30, 2005

Weddings & Harps

I've just spent a long weekend in Northern Ireland attending a wedding. Everything went well, and everyone had a good time. One of the nice things during the wedding ceremony itself, was that there was a piece of music played on a harp at one point. This made a nice change from the organ traditionally played in churches, and the acoustics of the harp gave a different feel to the church. The harp player was Rachel Hair, who has been playing the Scottish harp since a child.

We made a weekend of the visit over from England, and had a day doing some of the tourist stuff. We visited the Giant's Causeway which is quite a spectular piece of coastline, and Bushmills Distillery which makes Irish Whiskey.

( Aug 30 2005, 12:39:57 PM BST ) Permalink Comments [0]

Wednesday August 10, 2005

DTrace is great!

DTrace is really good. I'm sure most of you know that by now, so I'm not telling you anything new. But I was in an internal presentation on DTrace the other day, and I was just blown away by the ease with which the presenter created D scripts (the DTrace language) to find out what was happening on a system, and then drill down further.

Up to now I have always viewed Solaris Containers as the most useful feature of Solaris 10 i.e. the feature that you would benefit from the most, and would be the one that made you migrate to Solaris 10. But after this presentation I have changed my mind. DTrace gives you so much observability on what is happening on your system. And it does it immediately, non-intrusively (no need to recompile your application), in real time (meaning as it happens with no special set up), gives you as much detail as you want (including access to all arguments to functions and system calls), and is totally safe and always ready to use in Solaris 10.

Presuming that most application environments have some inefficiency or other in them, DTrace just opens up everything for you to find out where your bottlenecks are, and help identify the root causes of them. And you don't need the source code of any application to do this.

Want to see what is consuming up all the I/O on the system? Then write a small DTrace script to count the calls to the read and write system calls. In fact, also look at the size of each I/O too, and show the distribution and count the I/Os by process.

I know that this is all standard DTrace stuff, and that it has all been described many times in the DTrace documentation and other articles and blogs. So I won't waste time trying to describe what DTrace is capable of. I just wanted to say, that if I was an administrator or developer I would be begging for Solaris 10, so that I could use DTrace to investigate any anomaly with the system.

Containers would take some time to do all the necessary work: design the final configuration, set it all up, migrate the applications over from other systems and environments into each container, and require monitoring to ensure that performance of each application was acceptable. But DTrace I could use immediately on Solaris 10, on all applications, regardless of their configuration and set up.

( Aug 10 2005, 10:57:55 AM BST ) Permalink Comments [0]

Tuesday August 02, 2005

System Activity Snapshot or Performance Profile Baseline

Taking a snapshot of the activity on a system, or a baseline profile of its performance, is a useful tool for dealing with future performance problems. By establishing a baseline, and recording all of its associated details, we have a reference point for comparison at any point in the future. Should any performance problems be reported about the system in the future, then we can compare the current profile of the system to its previous baseline. Any differences will help us identify what has changed and from that the cause of the change in behaviour on the system. How else are we find out what has changed, and is causing the change in performance behaviour of the system? Without a baseline for reference, you literally do not know what has changed since the system was last working properly.

A performance profile consists of saving information about the utilisation of all of the resources on the system (CPU, Memory, Disk, Network, etc.), and all of the processes on the system (the consumers of the resources). This information is normally taken as a series of snapshots over a period of time. Examples range from every hour throughout a 24 hour day, to every minute during the peak hour of the day.

If you have some kind of performance management or monitoring tool, then it should be capable of capturing this data for you over your representative period, and then saving it away somewhere permanently for later use. If you don't have such a tool, then you can achieve something similar using standard UNIX tools like sar, ps, System Accounting and maybe even top (or prstat on Solaris), saving the outputs to a set of files. Of course it will require some manipulation to turn this raw data into a profile of the system and the applications running on it. But it must be better to have some data, no matter how raw it is, than no data at all.

Once this snapshot of the system behaviour has been established, we now have a point of reference for what we consider to be normal activity on the system. Should anything appear to be wrong at any point in the future, then we can compare it to this baseline snapshot, and find out what is different.

Performing this baseline does not require a lot of effort, yet has enormous potential benefits:

• We have a direct comparison between the actual behaviour of the system then and now.
• We can identify the actual changes on the system immediately, and so identify the cause of the effects we are observing.
• We can quickly drill down into a single area that needs further investigation, thereby eliminating the rest of the system.
• Our efforts can be focussed in on one target area only, increasing productivity and improving the time to a resolution.
• Anytime we want to, we can compare the current behaviour of the system to what it was.
• Regular captures of performance profiles can be used for trend analysis, establishing any regular change over time.
• Regular comparison can identify small changes before they become big enough to impact the overall performance of the system.

The cost of this is very little in real terms – some disk space to store the performance profile data, and some software tools to capture that data. Note that these tools would be needed to undertake any performance problem analysis anyway. So if you are serious about performance management on your systems, and have such tools, then there is no real extra expense to using them to baseline your systems.

( Aug 02 2005, 02:03:08 PM BST ) Permalink Comments [0]