Tuesday June 14, 2005

Hardware interrupts overview for Solaris X86

Welcome to OpenSolaris, and the wonders of Solaris 10.

This paper provides a brief introduction to hardware interrupts on x86 platforms. It is relevant for Intel and AMD based platforms. Interrupt handling is done via interrupt controller hardware in the system which are mostly sideband signals. Inband interrupts, for e.g. Message Signalled Interrupts (MSIs), introduced with PCI v2.2 specification onwards, will be discussed in another blog. MSIs are becoming mainstay with advent of new interrconnects like PCI-Express.

However, there are mainly two kinds of hardware interrupt controllers which are commonly used on x86 platforms:

1. 82c59(A) PIC(Programmable Interrupt Controller)

This is supported by the Solaris uppc(7d) module and its source is located at usr/src/uts/i86pc/io/psm/uppc.c.  Each PIC can handle 8 vectored priority interrupting sources and there are two PICs cascaded together to provide 16 interrupts on x86 systems.  However, one of the pin - IRQ2 of the 1st PIC is used to cascade the 2nd PIC and so there are only 15 interrupt sources.  This can not be used for multiprocessor (MP) systems without any major modifications.

2. APIC (Advanced Programmable Interrupt controller)

This is supported by the Solaris pcplusmp(7d) module and its source is located at usr/src/uts/i86pc/io/pcplusmp/apic.c.  It consists of two components - I/O APIC and Local APIC.  The Local APIC is embedded in the CPU while the I/O APIC is used for connecting the interrupting sources.  The Local APIC also has the capability to send interprocessor interrupt from one cpu to another and so APIC is widely used on all the x86 MP systems.  Each system can have multiple I/O APICs and each I/O APIC can have 4, 16, 20 or 24 interrupt pins.  Since the Local APIC is embedded in the CPU and the I/O APIC can handle more than 16 interrupt sources, even the single-CPU systems uses APIC as well instead of some other hardware.

There are many systems, which have I/O APICs with 4 inputs(this is typically done for PCI-X slotted systems, where each slot is given a dedicated I/O APIC, enabling INTA-INTD for each of the slots to have a dedicated input).

Solaris supports multiple I/O APICs.

             processor #1         processor #2
            +-----------+        +-----------+
            |           |        |           |
            |    CPU    |        |    CPU    |
            |           |        |           |
            +-----------+        +-----------+
            | local APIC|        | local APIC|
            +-----------+        +-----------+
                  ^                    ^
                  |                    |
                  |                    |
                  |                    |
                  v                    v         Processor system bus
    <----------------------------------------------------------------->
                            ^
                            |
                            |
                            |
                +-----------------------------+
                |           |                 |
                |           v                 |
                |       +--------+            |
                |       |        |            |
                |       | Bridge |            |
                |       |        |            |
                |       +--------+            |
                |           ^                 |
                |            |                |
                |            v    PCI         |
                |     <------------------>    |
                |                ^            |
                |                |            |
                |                v            |
                |           +----------+      |
                |           |          |      |
                |           | I/O APIC |<-----|---External
                |           |          |<-----|---Interrupts
                |           +----------+      |
                |                             |
                +-----------------------------+
                        System Chip set

Solaris x86 interrupt handling overview

When a device driver adding the interrupt through ddi_add_intr(9f), it eventually gets to uppc_addspl() in uppc(7d) for PIC or apic_addspl() in pcplusmp(7d) if using APIC.  The interrupt pin will be identified and then enabled.  It is quite simple for the uppc(7d) case, the interrupt pin to enable on the PICs is basically 1-1 mapped to the "IRQ#" or the "interrupt" property of the device on Solaris.  But for APIC (pcplusmp(7d)), it is a lot more complicated as internally it either uses the MP Spec. 1.4[2] or ACPI specification[3]  to locate the right interrupt pin of the right I/O APIC for the device. The system BIOS sets up how the interrupts are routed and saves that information in either the MP Specification table or somewhere that ACPI can easily access. pcplusmp(7d) then access that information to initialize and add the interrupts.

During the ddi_add_intr(9f) call, the device interrupt handler's entry point is stored in the autovect[] and the interrupt pin will be enabled through uppc_addspl (for PCI) or apic_addspl(for APIC).  Also, before the interrupt pin is enabled, an interrupt "vector" (refered to as "vector" from now on) will have to be selected for the CPU to trigger when that particular interrupt comes in.  For Intel CPUs, there are total 256 vectors and the first 32 vectors are reserved for special functions and so the first available vector for devices is 32 (or hex 0x20). 

For uppc(7d), vectors are set up such that the 1st pin or IRQ#0 is mapped to 32, IRQ1 to 33 and so on.  As for pcplusmp(7d), it is not as simple.  Solaris handles interrupts based on interrupt priority and each device is assigned a unique priority (can be modified by the device driver).  Say, if a device "abc" is assigned  priority 5, then all other interrupts at 5 or lower can NOT be triggered when the interrupt handler of "abc" is executing.  However, an interrupt of priority 6 or higher is allowed to trigger.  Since APIC has mechanism to prioritize interrupts, pcplusmp(7d) needs to select the vectors accordingly.

In order to handle the interrupt priority properly, there are few internal interface calls provided by uppc(7d) and pcplusmp(7d). They are the uppc_intr_enter()/uppc_intr_exit() for the uppc(7d); and apic_intr_enter()/apic_intr_exit() for pcplusmp(7d).  After the interrupt is triggered but before the interrupt handler is called, uppc_intr_enter() or apic_intr_enter() will be called to setup the interrupt priority accordingly to block all other interrupts with the same or lower priority.  After the interrupt handler is completed, then uppc_intr_exit() or apic_intr_exit() is called to restore the interrupt priority.

On the x86 platform, all the local variables of the interrupt handler are on stack. Also, if the interrupt handler needs to call another function, the parameters that are passed to the function are on stack too. i.e. all the interrupt handlers should use the stack one way or the other.

Solaris code that handles interrupts

usr/src/uts/common/io/avintr.c

#define MAX_VECT        256
struct av_head autovect[MAX_VECT];

usr/src/uts/common/sys/avintr.h

struct autovec {

        /*
         * Interrupt handler and argument to pass to it.
         */
        struct autovec *av_link;        /* pointer to next on in chain */
        uint_t  (*av_vector)();
        caddr_t av_intarg;
        uint_t  av_prilevel;            /* priority level */

        /*
         * Interrupt handle/id (like intrspec structure pointer) used to
         * identify a specific instance of interrupt handler in case we
         * have to remove the interrupt handler later.
         *
         */
        void    *av_intr_id;
	dev_info_t *av_dip;
};

av_vector is the device interrupt handler.

struct av_head {
        struct  autovec *avh_link;
        ushort_t        avh_hi_pri;
        ushort_t        avh_lo_pri;
};

- All interrupts run at some priority which has a ceiling of LOCK_LEVEL.
Interrupts below LOCK_LEVEL run as threads.

usr/src/uts/intel/sys/machlock.h

#define CLOCK_LEVEL     10
#define LOCK_LEVEL      10

- The following sequence shows what is done for each CPU to allocate
  enough interrupt threads to handle the interrupts. Since interrupts
  are prioritized, one interrupt thread per priority should be sufficient.

usr/src/uts/i86pc/os/mp_startup.c

void
start_other_cpus(int cprboot)
{
        for (who = 0; who < NCPU; who++) {
		mp_startup_init(who);
		...
}

void
mp_startup_init(void)
{
        init_intr_threads(cp);
	...
}

usr/src/uts/i86pc/os/intr.c

/*
 * Allocate threads and stacks for interrupt handling.
 */
#define NINTR_THREADS   (LOCK_LEVEL-1)  /* number of interrupt threads */

void
init_intr_threads(struct cpu *cp)
{
        int i;

        for (i = 0; i < NINTR_THREADS; i++)
                (void) thread_create_intr(cp);
	...
}

usr/src/uts/common/disp/thread.c

void
thread_create_intr(struct cpu *cp)
{
	- Here is the actual code handling the interrupts on x86.  
        *setlvl is the wrapper for uppc_intr_enter() or apic_intr_enter().
}



NOTE: Calling the interrupt handler is done in low level assembly code
which is not discussed here.

Reference:

Using Cfgadm with InfiniBand Welcome to OpenSolaris, and the wonders of Solaris 10.

This provides a brief introduction to using InifniBand Device management with cfgadm(1m).

Introduction

• IB Port devices
• IB HCA service (HCA_SVC) devices
• IB Virtual Physical Point of Attachment (VPPA) devices
• I/O Controller (IOC)
• IB Pseudo devices
  

Device Type

Attachment Type format

Static attachment

Attachment type format

# cfgadm -a ib hca:2C90109764440 
Ap_Id                          Type         Receptacle   Occupant     Condition
hca:2C90109764440              IB-HCA       connected    configured   ok
ib                             IB-Fabric    connected    configured   ok
ib::2C90109764440,0,svch       IB-HCA_SVC   connected    unconfigured unknown
ib::2C90109764441,0,psvc       IB-PORT      connected    unconfigured unknown
ib::2C90109764441,ffff,ipib    IB-VPPA      connected    unconfigured unknown
ib::2C90109764442,0,psvc       IB-PORT      connected    unconfigured unknown
ib::2C90109764442,ffff,ipib    IB-VPPA      connected    unconfigured unknown
ib::daplt,0                    IB-PSEUDO    connected    configured   ok
ib::rpcib,0                    IB-PSEUDO    connected    configured   ok
# 

# cfgadm -x list_clients  hca:2C90109764440
Ap_Id                          IB Client                 Alternate HCA
ib::daplt,0                    daplt                     no
ib::rpcib,0                    nfs/ib                    no
-                              ibmf                      no
-                              ibdm                      no
# 

# cfgadm -a ib::daplt,0             
Ap_Id                          Type         Receptacle   Occupant     Condition
ib::daplt,0                    IB-PSEUDO    connected    unconfigured unknown
# cfgadm -yc configure ib::daplt,0  
# cfgadm -a ib::daplt,0           
Ap_Id                          Type         Receptacle   Occupant     Condition
ib::daplt,0                    IB-PSEUDO    connected    configured   ok
# 

# cfgadm -a ib::daplt,0
Ap_Id                          Type         Receptacle   Occupant     Condition
ib::daplt,0                    IB-PSEUDO    connected    configured   ok
# cfgadm -yc unconfigure ib::daplt,0
# cfgadm -a ib::daplt,0             
Ap_Id                          Type         Receptacle   Occupant     Condition
ib::daplt,0                    IB-PSEUDO    connected    unconfigured unknown

# cfgadm -x unconfig_clients hca:2C90109764440 
Unconfigure Clients of HCA /devices/ib:2C90109764440
This operation will unconfigure IB clients of this HCA
Continue (yes/no)? yes <<<<<<< 
# 

# cfgadm -x list_services ib   
PORT communication services:
                psvc

VPPA communication services:
                ipib
HCA communication services:
                svch
# cfgadm -o comm=port,service=srp -x add_service ib
# cfgadm -x list_services ib                       
PORT communication services:
                srp <<<<<<<<<<<<<<
                psvc

VPPA communication services:
                ipib
HCA communication services:
                svch
#

# cfgadm -x list_services ib                       
PORT communication services:
                srp
                psvc

VPPA communication services:
                ipib
HCA communication services:
                svch
# cfgadm -o comm=port,service=srp -x delete_service ib
# cfgadm -x list_services ib                          
PORT communication services:
                psvc

VPPA communication services:
                ipib
HCA communication services:
                svch
#