Jon Greaves: Service Science, Gadgets and BBQ (not necessarily in that order...)

One way to start collecting some basic availability data is to poll SysUpTime from SNMP. Sysuptime shows the number of timeticks since the machine last rebooted and is part of a standard branch of SNMP that’s available on pretty much anything that supports SNMP. This has the benefit over other values in that it’s a counter, so for instance if you poll on a 5 minute interval and the box reboots at 2mins there is a good chance if you use absolute value that you will miss the reboot. With SysUpTime you can detect a reboot by the counter not incrementing between polls. We tend to use SysUpTime along with other metrics to give a proxy for certain types of OS availability. Why the Star Trek reference? Well while working on a customer issue we seem to be experiencing a temporal anomaly where time can slow down or speed up! The customer is running on a shiny new dell server running windows2k. If you poll every two minutes you would expect a constant number of timeticks to pass (with a small variance to cover overhead of the poll etc). In the box in question we have noted time slow down or speed up. This begged deeper investigation as I wondered if Dell had come up with a new way to improve the throughput of their hardware by utilizing a worm hole. It turns out that the explanation was a lot simpler and didn’t require a call to Dr Hawking. When the box gets busy its not providing enough cycles to keep the counter used by SNMP updated. Then when it gets a couple of extra cycles it catches back up. We managed to prove this by instrumenting the box with a management agent and dumping the cpu load every few seconds and cross reference. Makes me start to wonder what other “neat” things this hardware can provide. When you look at the specs of the box, it looks pretty reasonable; it has many of the items you would expect in a server. One gapping hole appears when you dig deeper into the architecture and you note that the box has obviously not been designed to be managed. Living in the world of remote management I look for various elements that allow me to perform my job in harmony with the hardware. Typically the manageability components of a server are referred to as a service processor. This is much more that a KVM switch, it’s a way to poll or be alarmed on fault conditions that can be pushed to an upstream management console for processing. Many high end service processors are really a specialized computer in their own right with a dedicated cpu, memory, network stack and a real-time OS. This allows an independent view of the hardware that’s not influenced by the running OS. Adding a lights out board after market is certainly an option but an ugly hack. The manageability needs to be designed into the hardware to gain the full benefit. One little know fact about the sunfire v20z (and 40z) servers is they have a strong service processor embedded into the hardware. This is not simply an after thought but designed into the system for manageability. The SP runs on its own CPU (Motorola MPC855T), 64megs of ram, has a built in three port network switch and even an auxiliary power source so you can access the SP functions when the main unit is powered down. Take a look at the whitepaper to see what a well designed SP looks like. Certainly not all data center hardware is created equally (even the entry-level server market), I would strongly recommend anyone considering a purchase to ask their vendor for more info on manageability of their server before spending a cent.