SPARC Enterprise M-class Servers The Secrets of Olympus

The new Sun External I/O Expansion Unit, or IO Box, supports PCI-X and PCI-Express hot plug. However, early on we realized that with the IO Box being remote from the host, it would be a challenge figuring out what the hot plug attachment points would be.

Normally, PCI card AP IDs (attachment point IDs) are labeled based on the physical location of the card -- its I/O Unit (IOU) and slot. For example, on Sun SPARC Enterprise M-class servers, the AP ID "iou#0-pci#1" is the slot PCI#1 on I/O Unit IOU#0.

With IO Box, however, there is no fixed physical location for the IO Box slots. The IO Box does connect to a host slot (like iou#0-pci#1), so one could label the AP IDs something like "iou#0-pci#1:iob.pci3" to show that it's PCI slot 3 in an IO Box attached to IOU#0-PCI#1.

On the other hand, this introduces issues when the IO Box is physically remote from the server -- it might not be obvious where this box conencts to the host. We don't want customers tracing cables, and a simple mistake could cause you to power off or power on the wrong slot. Something better, something more reliably was needed.

So we augmented the AP ID to include the serial id of the IO Box boat. With this approach, someone can look at an IO Box, write down the serial id and slot they wanted to power off, then go back to Solaris and power off that slot based on serial id. Similarly, if you powered off a slot and wanted to go remove the card, you can write down the serial id and slot, then go find the IO Box boat with matching serial id, and have confidence that you're removing the right card.

The resulting AP ID is a combination of physical location of the host slot, and serial id of the IO Box boat. An example of the AP ID format looks like this "iou#0-pci#4:iobE00E7.pcie1", which slows PCIE slot 1 in the IO Box boat with serial id ending in "E00E7". From the AP ID, it also is clear that the IO Box boat is connected to the host using a link card in host slot IOU#0-PCI#4.

I should also note that the IO Box boat serial id is prominently featured on handle, in plain view. There's no need to remove the boat to get to the product nameplate.

Here's some sample output from 'cfgadm -a' showing just the PCI slots:

# cfgadm -a
Ap_Id                          Type         Receptacle   Occupant     Condition
...
iou#0-pci#0                    unknown      empty        unconfigured unknown
iou#0-pci#1                    unknown      empty        unconfigured unknown
iou#0-pci#2                    unknown      disconnected unconfigured unknown
iou#0-pci#3                    pci-pci/hp   connected    configured   ok
iou#0-pci#3:iobX00FC.pci1      unknown      empty        unconfigured unknown
iou#0-pci#3:iobX00FC.pci2      fibre/hp     connected    configured   ok
iou#0-pci#3:iobX00FC.pci3      scsi/hp      connected    configured   ok
iou#0-pci#3:iobX00FC.pci4      unknown      empty        unconfigured unknown
iou#0-pci#3:iobX00FC.pci5      fibre/hp     connected    configured   ok
iou#0-pci#3:iobX00FC.pci6      unknown      empty        unconfigured unknown
iou#0-pci#4                    pci-pci/hp   connected    configured   ok
iou#0-pci#4:iobE00E7.pcie1     unknown      empty        unconfigured unknown
iou#0-pci#4:iobE00E7.pcie2     etherne/hp   connected    configured   ok
iou#0-pci#4:iobE00E7.pcie3     etherne/hp   connected    configured   ok
iou#0-pci#4:iobE00E7.pcie4     pci-pci/hp   connected    configured   ok
iou#0-pci#4:iobE00E7.pcie5     pci-pci/hp   connected    configured   ok
iou#0-pci#4:iobE00E7.pcie6     unknown      empty        unconfigured unknown

In the above output, IOU#0-PCI#0 through IOU#0-PCI#4 are the host slots; IOU#0-PCI#4 is connected to a PCI-X IO Box boat, while IOU#0-PCI#4 is connected to a PCI-Express IO Box boat.