Scott Davenport @ Sun: (Mainly) Fault Management

You've all seen this before...actually, maybe you haven't. In which case....good! You've got a healthy system. But, if your box did have issues, FMA would report something like:

# fmadm faulty --------------- ------------------------------------ -------------- --------- TIME EVENT-ID MSG-ID SEVERITY --------------- ------------------------------------ -------------- --------- May 13 15:00:02 04837324-f221-e7dc-f6fa-dc7d9420ea76 AMD-8000-AV Major Fault class : fault.cpu.amd.dcachedata Affects : cpu:///cpuid=0 degraded but still in service FRU : "CPU 0" (hc://:product-id=Sun-Ultra-20-Workstation:chassis-id= 0604FK401F:server-id=hexterra/motherboard=0/chip=0) ...

Most people are immediately interested in the FRU label - the "CPU 0" in the above output. Answers the question of what needs to be replaced to fix my system. But where does that label come from? Today, FRU labels are put in via platform specific code (an example). And not all Sun platforms have support for FRU labels coded. Running Solaris on HP, Dell, or IBM gear? Unless you've modified OpenSolaris, you don't have FRU labels.

You'll still have the FMRI (the hc://... stuff above). And while the example above seems simple enough, it's not always straightforward determining a FRU from an FMRI (try doing it for IO). Solaris FMA, specifically the topology, is supposed to do the translation for you.

A recently added project, the x86gentopo project, is working to revamp topology creation on x86 platforms. The concept is that a platform, Sun or otherwise, can describe its topology and FRU information to Solaris FMA. And the goal is to use existing industry standards to do so. My colleague Tom Pothier is leading the project, and the first go-round is looking to use SMBIOS information to build topology.

SMBIOS already provisions for representation of baseboard FRUs, processors, memory and the like. And with some extensions can provide the glue to connect the structures together and associate them with the devices as Solaris sees them. A very basic example is depicted at right. There are several fields in the structures that allow for the various structures to be arranged in a logical hierarchy. FMRI names can be derived either directly from the structure type ('chassis' for Type 3 seems obvious), or by a mapping (such as using "Board Type" in Type 2s). The picture at right could produce a topology like:

hc:///chassis=0
hc:///chassis=0/cpuboard=0
hc:///chassis=0/cpuboard=0/chip=0

And as for labels, cpuboard=0 is labeled per the Type 2 "Location" field. The Processor, if it is a FRU, is labeled per the Type 4 "Socket Designation" field. Something else Sun is including in the FMRIs is details on product serial numbers, part numbers, serial numbers, and chassis serial numbers. If you're using Sun's ASR for Systems offering, including this level of detail allows automatic checking for service entitlement, case creation, and replacement part ordering.

This is merely a simple example, and there's many more complexities with having a generic mechanism for generating FMA topology on Solaris x86. If you're interested in the project, become an observer and/or join the x86gentopo-discuss alias. We'd welcome the help :)

:wq

A lot of work has been put into devising a common rule set for PCI/PCIE devices in Solaris FMA. Known as I/O Fault Services, there's a thorough document detailing how a developer goes about hardening a device driver for Solaris FMA. If you're working on a device driver, definitely check it out.

Here's just a few of the drivers hardened to integrate with FMA that I've noticed integrate into OpenSolaris recently:

Driver	Description	Integration
emlxs	Emulex 2.40 driver	snv_114 via 6794530
hermon	Mellanox device driver	snv_107 via 6747341 (open sourced in snv_115 via 6808773)
mega_sas	MegaRAID SAS controllers	snv_99 via 6808773
ixgbe	Intel 82598 10GbE	snv_90 via 6574882
igb	Intel 82575 PCI-E Gigabit NIC	snv_90 via 6656301
hxge	Hydra 10G ethernet	snv_88 via 6656720

:wq

OpenSolaris 2009.06 released today. One of the key features is support for the SPARC product line. A colleague and I gave one of the pre-releases a whirl on a midrange system, an M3000

Now, a silent demo of a system booting is about as exciting as watching paint dry. So being a fault management guy, I added a basic error injection to the demo, triggering an FMA response....which makes the video asymptotically approaching exciting. In other words, be prepared to fast forward at times.

The flash video below (approximately 8 minutes) shows a few things:

• Boots OpenSolaris b109 on an OPL M3000
• ssh into the M3000, confirm OpenSolaris is running, confirm 8 processors online
• Force an uncorrectable CPU problem, causing a system crash
• Observe FMA automatically diagnose the CPU problem
• Run a few CLIs to examine processor state and FM status
• Repair the (contrived) fault
• Confirm offlined processors returned to on-line state

Note: If the video is blurry or text is too small, try viewing directly on MediaCast.