Using Sun tech at Strathclyde Sun@Strathclyde

I woke up today to find that one of the disks in my home server had failed overnight. I was actually able to work this out while still in bed shortly after waking up, because I could hear it clicking and whirring pathetically as it tried to spin up - not a nice way to start your day. As I write this I'm in the process of filing an RMA to get the disk replaced, which promises to be a painful, drawn-out process, but hey - at least my data is still safe thanks to ZFS (so long as none of my other disks decide to break - not inconcievable, seeing as they're all identical...).

However, hardware faults aren't always audible, so I was pleased to see that my script for detecting hardware faults and then emailing me had triggered. Here's what I got sent:

-------- Original Message --------
Subject: Hardware failed on zebedee
Date:    Sat, 25 Apr 2009 13:54:02 +0200
From:    lamsey@zebedee

--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Apr 25 09:32:07 43d4b6e4-1219-e9d5-bac5-f829b8fb2f2a  ZFS-8000-D3    Major    

Fault class : fault.fs.zfs.device

Description : A ZFS device failed.  Refer to http://sun.com/msg/ZFS-8000-D3 for
              more information.

Response    : No automated response will occur.

Impact      : Fault tolerance of the pool may be compromised.

Action      : Run 'zpool status -x' and replace the bad device.

Logging into my server and running zpool status -x showed me which disk was at fault (c4d0), and a bit of searching in the output of prtconf -v allowed me to work out the serial number of the affected disk (more specifically, it allowed me to work out the serial numbers of the disks which were still working, meaning I could work out which physical disk was broken by a process of elimination after cracking the box open).

So, how do I achieve the above? The answer is actually incredibly simple. The content of the email is just the output of fmadm faulty, a command which interrogates Solaris' FMA (Fault Management Architecture) feature to see if there's any hardware issues on a system. Wrap it up in a script (the below is based on one I found on the 'net eons ago and can no longer find), and you end up with something like:

lamsey@zebedee:bin$ cat check_hardware.ksh
#!/bin/ksh
# Public domain. Use as you wish.

EMAIL=liam@lamsey.co.uk
TMPFILE=/tmp/fmadm.output.$$

# run fmadm and cut away the first two lines (headers)
/usr/bin/pfexec /usr/sbin/fmadm faulty | /usr/bin/sed 1,2d > $TMPFILE

# Check if the file size is greater than zero. This means we got
# some output from fmadm and therefore some hardware may be bad.
# Using HTML here means we can use <pre> to preserve formatting.
if [ -s $TMPFILE ]; then
    (
        /usr/bin/echo "Subject: Hardware failed on `hostname`"
        /usr/bin/echo "From: lamsey@zebedee"
        /usr/bin/echo "MIME-Version: 1.0"
        /usr/bin/echo "Content-Type: text/html"
        /usr/bin/echo "Content-Disposition: inline"
        /usr/bin/echo
        /usr/bin/echo '<pre>'
        # don't just use the temp file, it's missing headers
        /usr/bin/pfexec /usr/sbin/fmadm faulty
        /usr/bin/echo '</pre>'
    ) | /usr/local/bin/msmtp -a 1and1 $EMAIL
fi

# clean up the temp file
/usr/bin/rm -f $TMPFILE

Simply slap a call to the above script into your crontab, ideally running at least once a day, and you're good to go. Note that I use msmtp for sending emails automatically as it's a heck of a lot easier to configure than sendmail (which is important if you use an ISP like o2 which blocks outgoing SMTP traffic, preventing you from using sendmail in its out-of-the-box configuration). It doesn't come with Solaris though, so you'll need to compile it if you want to do the same (very simple, works fine with configure / make / make install).

Edit (01/5/09): I received the replacement disk today (took them long enough...). Slammed it into the server, issued a quick zpool replace c4d0 command, and all is good with the world again :-)

lamsey@zebedee:~$ zpool status shared
  pool: shared
 state: ONLINE
 scrub: resilver completed after 3h13m with 0 errors on Fri May  1 17:12:23 2009
config:

        NAME        STATE     READ WRITE CKSUM
        shared      ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            c3d0    ONLINE       0     0     0  217M resilvered
            c3d1    ONLINE       0     0     0  217M resilvered
            c4d0    ONLINE       0     0     0  236G resilvered
            c4d1    ONLINE       0     0     0  217M resilvered
            c6d1    ONLINE       0     0     0  217M resilvered

errors: No known data errors

OK, first off: I'm an idiot. You'd think that after forgetting to take pictures/videos at my last few tech demos, I'd have managed to remember to do so today during the Sun@Strathclyde OSUM's visit to Linlithgow. Sadly, no - completely forgot. Had the camera in my bag and everything, just completely failed to use it. Epic lose :-(

The day, however, did go pretty well - we were entertained by the manager of Sun's Executive Briefing Centre at Linlithgow, who kindly showed us around the campus, including a tour of Project Blackbox and the cavernous Solution Centre lab. We also, after a brief spell of warbling from myself, enjoyed a great talk from two experienced Sun engineers about science, technology and engineering from a Sun perspective - really interesting stuff, and plenty of useful food for thought for the future.

 All in all, a pretty good day, and a chance for students to get a glimpse of the inner workings of a high-tech computing company. I think we'll have to see about doing this again next year, so don't worry if you missed it!

I was asked today for some info on how to get to Sun's Guillemont Park location by train from Glasgow. This is a journey I've made many times, so I actually know pretty well off the top of my head how to negotiate the national rail network to the area around GMP. With that in mind, here's the info I put together - hopefully it'll be useful to other people at some point.

Here are the rail instructions. Obviously, if you're coming from somewhere other than Glasgow, you'll have a different starting station, but if it's anywhere north of Birmingham the below should still apply.

When you're wanting to check times or book tickets, go to somewhere like http://www.nationalrail.co.uk and search for "from: Glasgow (all stations)" to either "Farnborough (all stations)" or "Camberley". Between those end stations, you'll get all of the three train lines coming into the area around Guillemont Park.

Usually, it'll be a direct line from Glasgow to London Euston (if it's the West Coast main line, usually Virgin) or King's Cross / St. Pancras (if it's the East Coast main line, usually National Express). The east coast route takes longer as it goes via Edinburgh, but National Express have free wifi on their trains whereas Virgin don't.

From either London station, it'll then be a tube transfer to Waterloo. From Euston it's about half a dozen stations on the Northern line, no changes needed. I think you might need one change coming from King's cross, can't remember off the top of my head which lines.

From Waterloo, you'll get a South West Trains service to whichever end station you chose. If it's Farnborough Main, it'll most likely be a direct service. Camberley will most likely need a change at either Ascot or Ash Vale. Farnborough North will involve a change at Reading or Woking, I think - not sure, I've only been on this line once.

There are some other routes to these stations other than London - I've been through Reading and Birmingham before, rather than the big London terminals. I'd go via London for preference, but you might find the website gives you other options.

Once you get to your chosen end station, though, you need to get to Guillemont Park itself. Unfortunately, GMP is roughly equidistant from all the surrounding train stations, meaning you'll need to walk a few miles to get there on foot from the station.

Handily, however, Sun and Nokia run a shuttle bus service between their campuses and most of the surrounding rail stations (with Fleet the notable exception - take note if you're moving down there and don't have transport for getting to GMP!). Here's a Google Map I threw together which shows GMP's location relative to the shuttle bus routes:

The map above can be found here, while the shuttle bus timetables can be found here.

I hope this is of use to someone at some point. Feel free to leave a comment if you want any more info!

Today saw the last tech demo before the easter break, this time about MySQL. I managed to forget to take videos of the past few demos, sadly (I blame Jason for being busy with other stuff - he's supposed to play cameraman, dammit), so you'll just have to take my word for it that they were really interesting!

I've got some news about our upcoming Linlithgow visit - more accurately, I have a date: Friday 17th April (that's the second week of Strathclyde's Easter holidays). I'm still in the process of working out an agenda with the folks at Linlithgow, so watch this space - and the Sun@Strathclyde OSUM page - for more info.

Update (10/4/09): I've now got the agenda and travel instructions up on the OSUM website - check there for more details as I receive them!

So I've just gotten home from a day's field work for my final-year project, and I find that my guest speaker for tomorrow's tech demo has had to cancel because the train service screwed up and he got stranded in Crewe.

Just what I need after an exhausting day - now I've got to spend the evening preparing a talk for tomorrow, not to mention telling everyone that there's been a change of plans. Worst of all, there's no way I can give the specific talk Michael was planning to give, so it's all a bit crap really :-(

Oh well, onwards and upwards. Time to crank up OpenOffice Impress!

I had initially planned to just talk about OpenSolaris Zones today, but I decided to squeeze in some VirtualBox goodness too since they're two different approaches to the same problem. As a result, today's talk was a pretty decent introduction to virtualisation in general, so that's the subject of this week's demo clips:

Next week should be entertaining on the video front, as there is some potential for mass destruction when Michael comes visiting. Watch this space for more details...

Another week, another Sun tech demo at Strathclyde. This week, I was talking about ZFS, the revolutionary filesystem that comes as standard with the OpenSolaris distribution. It's pretty cool technology, so I've been looking forward to doing this talk.

As I've done before, I recorded some video clips at the demo. Below you can watch the first twenty minutes or so of the talk. It has to be said that this only scratches the surface of what ZFS can do - I discussed a lot more in the talk (ZFS properties like quotas and reservations, transparent compression, snapshotting, and so on), and even then there is a whole load of other funky stuff that ZFS does which I didn't have time to cover...

We're now four weeks into my programme of weekly events at Strathclyde, and I'm not letting up. Next week I'll be talking about Zones, another OpenSolaris technology, and I should hopefully be welcoming Michael Clarke to Strathclyde the following week for an entertaining introduction to device driver development using NetBeans (I'm waiting for absolute confirmation on that, so watch this space!).

After accidentally leaving it in university over the weekend, I've now retrieved my camera and pulled off some clips from last Friday's Sun@Strathclyde tech demo, which was all about OpenSolaris. I've had to split the clips up a bit since YouTube doesn't like videos longer than ten minutes in length, but thankfully there's a custom player option which allows me to combine them into one:

The next two demos I'll be running are also going to have an OpenSolaris theme, so if you're interested in hearing more about ZFS and Zones, make sure you remember to come along!

Today saw the second Sun@Strathclyde OSUM tech demo of 2009, this time about Sun SPOTs. We had a lower turnout than last time around, which is a shame, because those who did come along seemed to enjoy the demo. It was certainly fun getting some audience members up to throw a SunSPOT around between each other while I graphed the accelerometer data on my laptop (and I think they enjoyed the free t-shirts too!).

Sadly, the videos I took today haven't turned out well; I had hoped to be able to share them online, but my camera decided it didn't want to play ball. The fact that Jason wasn't around to play cameraman didn't help (he was away seeing a lawyer about sorting his US taxes out or something - can't see how that's more important than helping out at my demos, but there you go :-D ).

There seemed to be a fair bit of interest in the tech from the CES students who arrived in the room for a tutorial afterwards, so if there are more people who'd like to see this talk given again, I'd be happy to oblige. Just get in touch - preferably through the OSUM website ;-) - and I'll see what can be done!

My next demo is an introduction to OpenSolaris - Sun's free-to-download open-source operating system. As always, it's next Friday at 1pm in room M4.09 (the James Weir Building). See you there!

Here's another short clip from the Sun@Strathclyde OSUM Kickoff event last Friday - this one's an introduction to SAI, Sun's program offering students and academics free access to online training resources (plus deeply-discounted certifications!).

Unlike the previous video, there's no walkthrough showing how to sign up. This is because you need a special code and program ID to sign up to SAI (this is so we can make sure that only students and academics are actually signing up). If you're interested in getting access to SAI (and why wouldn't you be?!), give me a shout and I can provide you with a card which has detailed instructions on how to sign up.

I've now ploughed through all the feedback forms from last Friday and worked out what I'm going to be talking about in my weekly tech demos over the course of this semester. It was a surprisingly close-run thing in places, but the areas of interest were mostly as I expected. I'll have to see about getting in a guest presenter for one or two of the events, but that hopefully won't be a problem :-)

Here's the list of topics and dates. Unless otherwise specified on the OSUM website, all my tech demos will be held on Fridays at 1pm in room M4.09 (that's the James Weir Building). You can click the links below for more details and to register for events, just so long as you've got an osum.sun.com account (and if you don't, see my previous post for a run-through of how to sign up!).

• Fri 06/2/09 - Sun@Strathclyde OSUM Kickoff 2009
• Fri 13/2/09 - Sun SPOT: Small, Programmable Object Technology
• Fri 20/2/09 - Introduction to OpenSolaris
• Fri 27/2/09 - ZFS: The Most Advanced Filesystem on the Planet
• Fri 06/3/09 - Zones: Virtual Operating Systems
• Fri 13/3/09 - Introduction to NetBeans
• Fri 20/3/09 - Web Development with NetBeans
• Fri 27/3/09 - MySQL: The Open-Source Web DB
• Fri 03/4/09 - No event (final-year project submission day!)
• Fri 24/4/09 - Sun Studio (to be confirmed)
• Fri 01/5/09 - Project Darkstar (to be confirmed)

In addition to the above events, I've got some other stuff in the pipeline to sort out, including an organised tour of Sun's Linlithgow campus. I'll probably do that one during the Easter break, so we don't interfere with anyone's class schedules, so I'll talk about that more once I've made some arrangements - as always, watch this space!

It's been a pretty hectic weekend for me - immediately after finishing my first tech demo of the year on Friday, I hopped straight onto a train to go visit a friend for the weekend. Then, with it being my mum's birthday today, I had to high-tail it back home to join my parents for dinner at The Bay Tree (a very nice wee restaurant on Great Western Road, highly recommended). I was going to finalise my demo programme tonight but my brain's not working so I guess it'll have to wait till tomorrow :-p

The kickoff event on Friday went fairly well - we got 46 students along, which is not too bad for the first event of the year. Hopefully word-of-mouth and the continued allure of free stuff will keep people coming along. Below you can see a short clip from the talk in which I describe Sun's open-source strategy and how the OSUM programme fits into it. There's also a run-through of how to sign up to the website and join my group (hint, hint) :-D

So that's one down and ten to go - we've got ten weeks left before the end of the semester and I'm hoping to fit ten demos in, one per week. Watch this space for news of what I'll be covering!

It's been a while since I've posted any updates here. Despite appearances, I haven't actually disappeared off the face of the Earth (I've actually been sequestered in my room working away on my final-year project for approximately the last four months) and I've now sorted out the schedule for the Sun@Strathclyde OSUM tech demos in 2009.

What's an OSUM? Well, it's kind of like an on-campus club; it stands for Open-Source University Meetup and is, somewhat embarrasingly, pronounced "awesome" (the person who thought that one up is an American, obviously!). There's a whole social networking site devoted to all the OSUMs around the world at osum.sun.com and the Sun@Strathclyde group is here.

So when's the first tech demo? The answer is next Friday (that's the 6th of February) in room M4.09 at 1pm till 2pm. We'll be covering: a general intro to the OSUM program, instructions on how to access free online training resources through the Sun Academic Initiative, and attendees will have the chance to vote for the topics and technologies they want to see covered at the subsequent weekly demos. That's right, this is a weekly thing - same time, same place, every week.

For more info or to get involved, please feel free to join the Sun@Strathclyde OSUM group. I'll see you on there!

As I've previously mentioned, the main reason I'm a Sun Campus Ambassador is because I spent 18 months with them on placement as part of my degree course. Well, by the end of this summer that'll be 21 months, as I'm back down in Camberley just now for a summer contract with my old team in the Global Labs.

My desk at Guillemont Park, complete with boxes of broken server hardware

It's an interesting experience, as I'm now one rung off the bottom of the pecking order - still a long way from CEO, but at least I get to boss the new guys about help bring the new year's placement interns up to speed. It's a good opportunity to build up a bit more professional experience before leaving uni, and the extra money doesn't hurt either :-D

I'll be starting up the Campus Ambassador program again when I return to Strathclyde this autumn, and part of that will be promoting the Sun placements available to Strathclyde students, which will probably include the chance to work the US Global Labs team in California as in previous years. Watch this space for more information!

Today saw the second Sun@Strathclyde tech demo, this time a talk about the innovative Sun SPOT wireless sensor technology. There's a fair bit of interest in using these at the university, especially within the EEE department, but sadly since that department still hasn't gotten back to me about arrangements to promote my events I wasn't able to get anyone from EEE along.

That, combined with the fact that it's the end of term, meant I only welcomed along 12 CS students and lecturers to the demo today - not a great turnout given last week's attendance, but those who did come seemed to be quite interested in using the technology - not to mention earning themselves some Sun T-shirts by getting involved (thanks guys!).

Below you can see a short video clip from one of the live demo sections. Sadly the 'Air text' effect doesn't show up very well on video (it works by 'tricking' the naked eye), but you can get a decent idea of what it looks like here.

Since tomorrow's the last day of term, that pretty much wraps up the tech demos for this semester, but I'm still going to be watching the forums and checking my email, so if any readers need to know anything about Sun@Strathclyde, please just give me a shout!