SPARC Enterprise M-class Servers The Secrets of Olympus

Physical System Boards and the Sun Fire 6800

In the past (using the Sun Fire 6800 as an example, since I happen to have one handy), an SB could be configured into a domain, and the resources on that SB were identified to Solaris based on the board number; similarly, if you knew the resource id, you could infer the physical system board it is on. For example, if psrinfo in Solaris showed

    % psrinfo
    0       on-line   since 03/01/2007 12:16:43
    1       on-line   since 03/01/2007 12:16:44
    2       on-line   since 03/01/2007 12:16:44
    3       on-line   since 03/01/2007 12:16:44
    8       on-line   since 03/01/2007 12:16:44
    9       on-line   since 03/01/2007 12:16:44
    10      on-line   since 03/01/2007 12:16:44
    11      on-line   since 03/01/2007 12:16:44

    % ls -1d /devices/ssm@0,0/pci@*000
    /devices/ssm@0,0/pci@18,600000
    /devices/ssm@0,0/pci@18,700000
    /devices/ssm@0,0/pci@19,600000
    /devices/ssm@0,0/pci@19,700000
    /devices/ssm@0,0/pci@1c,600000
    /devices/ssm@0,0/pci@1c,700000
    /devices/ssm@0,0/pci@1d,600000
    /devices/ssm@0,0/pci@1d,700000

If a system board experienced a fault and needed to be replaced, or worse, the system board slot was at fault so you could not simply replace the system board, you could reconfigure the system from the System Controller to add CPUs, memory or IO from a different system board to restore the domain to full power. You could, for example, configure SB0 out of the domain, and configure SB1 into the domain. At that point, the domain would be running with CPU IDs 4 through 11 (4 through 7 on SB1, and 8 through 11 on SB2). Similarly, you could replace IB6 with IB7, and the PCI hostbridges would change from pci@18 and pci@19 to pci@1a and pci@1b.

That's all fine, unless your boot device was hanging off IB6. Even if you moved the boot device to IB7, the device paths would all be different. The boot device that was "/devices/ssm@0,0/pci@18,700000/pci@1/SUNW,isptwo@4/sd@0,0:a" would change to "/devices/ssm@0,0/pci@1a,700000/pci@1/SUNW,isptwo@4/sd@0,0:a".

In effect, the CPU IDs and hostbridge bus addresses are physical addresses -- they are calculated based on the physical location of the board.

Logical System Boards and the Sun SPARC Enterprise Servers

When an XSB is assigned to a domain, it is given a logical system board number. In effect, the LSB number is a virtual address. And as the Sun Fire 6800 assigns CPU IDs based on physical system board number, the Sun SPARC Enterprise assigns CPU IDs based on logical system board number. The same is true for hostbridge bus addresses. The CPUs on LSB 0 are assigned CPU IDs from the range 0 through 31, and the CPUs on LSB 1 are assigned CPU IDs from the range 32 through 63, regardless of the physical system board hosting the CPU chips. Similarly, the first hostbridge on LSB 0 is pci@0,600000, while the first hostbridge on LSB 1 is pci@10,600000, and so forth.

The mapping from LSB-to-XSB is user configurable; you can choose the LSB number for any XSB almost entirely at will, and LSB numbers can be re-used for different XSBs in different domains. As a result, it is possible that every domain in a chassis could have a CPU with cpuid 0. And every domain could have its boot device below /devices/pci@0,600000. You could have a domain that includes SB 0, 1 and 2 assigned as LSBs 0, 1 and 2, but for some reason it is necessary to replace SB 0. You could then assign SB 3 to the domain as LSB 0. The domain would continue to have cpuid 0 and /devices/pci@0,600000. If you move your boot device over to SB 3's I/O unit (either move the PCI-Express card, or simply move the internal SAS disk), you could boot the domain, and device paths and processor sets would remain unaffected.

If we use the analogy of virtual memory, the domain is the context, the LSB is the virtual address, and the SB (or more specifically, the XSB) is the physical address.

Configuring LSBs

setupfru

    XSCF> setupfru -x 1 sb 0
    XSCF> setupfru -x 1 sb 1

setdcl

• Domain 0:
  • LSB 0 => XSB 00-0 (system board 0)
  • LSB 1 => XSB 01-0 (system board 1)
• Domain 1:
  • LSB 15 => XSB 00-0 (system board 0)
  • LSB 0 => XSB 01-0 (system board 1)

    # For domain 0, map LSB 0 to XSB 00-0 and LSB 1 to XSB 01-0
    XSCF> setdcl -d 0 -a 0=00-0 1=01-0
    # For domain 1, map LSB 15 to XSB 00-0 and LSB 0 to XSB 01-0
    XSCF> setdcl -d 1 -a 15=00-0 0=01-0

addboard

    XSCF> # Assign XSB 00-0 to domain 0
    XSCF> addboard -c assign -d 0 00-0
    XSB#00-0 will be assigned to DomainID 0. Continue?[y|n] :y
    XSCF> # Assign XSB 01-0 to domain 1
    XSCF> addboard -c assign -d 1 01-0
    XSB#01-0 will be assigned to DomainID 1. Continue?[y|n] :y

    # Try to assign XSB 00-0 to domain 1 also
    XSCF> addboard -c assign -d 1 00-0
    XSB#00-0 is already assigned to another domain.

We can use showboards to see what we've done:

    XSCF> showboards -a
    XSB  DID(LSB) Assignment  Pwr  Conn Conf Test    Fault
    ---- -------- ----------- ---- ---- ---- ------- --------
    00-0 00(00)   Assigned    n    n    n    Unknown Normal
    01-0 01(00)   Assigned    n    n    n    Unknown Normal

Proof

    XSCF> console -yq -d 0

    {0} ok show-disks
    a) /pci@0,600000/pci@0/pci@8/pci@0/scsi@1/disk
    q) NO SELECTION
    Enter Selection, q to quit: q
    {0} ok
    {0} ok exit from console.

    XSCF> console -yq -d 1

    {0} ok show-disks
    a) /pci@0,600000/pci@0/pci@8/pci@0/scsi@1/disk
    q) NO SELECTION
    Enter Selection, q to quit: q
    {0} ok