Tuesday June 14, 2005

OpenSolaris - The 1394 Software Framework

When I started at Sun about seven years ago, the Solaris 1394 Software Framework was my first project. A team of four, we designed, implemented, tested and putback to Solaris 8 (Update 2) in about a year and a half. Since then the code has been transitioned to others (like Alan Perry and Artem Kachitchkine), who continue to maintain and extend its functionality even today. Now, almost six years later, I finally get a chance to share this code with the world (and, hopefully, at least some interested readers.)

So I figured I'd start by giving a brief overview of the stack and the features that it provides and then talk a bit about the source files for the modules: how they're organized, what's in them, etc. Then (over the course of several posts) I'll get into the specifics of how to use each of interfaces, gotcha's for potential developers working with the source, some of the little bits of which I'm most proud, and maybe some discussion of what could be improved or extended in the existing code (and I'll be interested to hear what others think). This will not be an intro to the IEEE 1394 specification or the technology, nor will it be an introduction to writing Solaris device drivers (though I'll try to help anyone in any way I can). What follows will assume a certain familiarity with IEEE 1394 and with writing Solaris device drivers.

The Solaris 1394 Software Framework

The framework itself consists of a central module (s1394) called the "1394 Services Layer", an OpenHCI-compliant HAL driver (hci1394), and numerous target drivers (currently, av1394, scsa1394, and dcam1394 - which I'll say more about later.) If you are familiar with Solaris's SCSA framework (for SCSI drivers), you'll recognize this stacking of modules. This arrangement is a typical way to abstract hardware-specific details (below the Service Layer) from the target drivers (above the Services Layer).

The Solaris 1394 Software Framework Device Driver Interface provides a set of kernel interface routines to facilitate access to devices on an IEEE 1394 bus. The interface routines are intended for use by 1394 device drivers, also referred to as target drivers.

There are two kinds of target drivers: class drivers and vendor specific drivers. Class drivers adhere to a general standard for a particular kind of device and can drive any vendor's device that adheres to the same standard. For example, a class driver for the IEEE 1394 Digital Conferencing Camera specification can drive video conferencing cameras manufactures by a variety of vendors (even though each may have a different set of features). Vendor-specific drivers are built to drive a specialized non-standard device. The 1394 Software Framework supports both kinds of target drivers.

Features of the 1394 Framework

The 1394 Software Framework provides several features to support target drivers using the IEEE 1394 bus.

Asynchronous I/O

There are two sides of asynchronous I/O: issuing outgoing requests and handling incoming requests.

Outgoing Requests - The 1394 Framework provides the ability to send the basic set of IEEE 1394 asynchronous requests; read, write and lock. In addition to the IEEE 1394 defined set of lock request options, the Framework lock request interface also provides a set of bit and arithmetic functions.

For asynchronous requests, the 1394 Framework automatically determines the device's destination ID, sends the request using the local host's 1394 hardware interface, tracks the status of the request until the transaction completes, and supplies response information as needed.

Target drivers choose whether to: 1) block while waiting for the transaction to complete, 2) poll on the request completion status, or 3) have the 1394 Framework call a specified callback routine when the transaction completes. Using the poll and callback mechanisms, target drivers can issue several outstanding requests and poll or be notified for each completion.

Incoming Requests - The primary role of the 1394 Framework with respect to incoming asynchronous requests is to dispatch the request to the appropriate target driver or to handle the request on behalf of the appropriate target driver.

To support this, the 1394 Framework provides an allocation mechanism that target drivers use to reserve ranges of addresses within the 48-bit local node address space. Target drivers have several options available when allocating 1394 address space including the ability to specify a kernel virtual buffer to map to the allocated space. When the 'destination_offset' of an incoming request falls within an allocated address range, the 1394 Framework fulfills the request if possible, notifies the target driver if desired, and sends the response. Target drivers may also allocate 1394 address space with the characteristic that an incoming request to that space will be handled by hardware. In this case hardware directly accesses host memory bound to that address space, and transmits the appropriate response.

Isochronous I/O

Due to the potentially large volume of isochronous data and the critical isochronous timing needs, the 1394 Software Framework provides a mechanism designed to reduce call overhead and to maximize throughput.

Before starting isochronous I/O, a target driver sets up the overall sequence and structure of receive or transmit buffers, indicating other needs such as when the 1394 Framework should invoke a target driver's callback. Once isochronous I/O is started, the target driver can focus most of its time on handling the data.

The 1394 Framework mechanism for configuring isochronous I/O is the Isochronous Transfer Language (IXL). The IXL is a hardware independent set of control blocks that the target driver uses to direct isochronous DMA. The 1394 Software Framework converts the hardware independent IXL into the appropriate DMA directives for the local host 1394 interface hardware changes, the impact to the target driver is minimal or non-existent.

The 1394 Software Framework also facilitates peer to peer communication by tracking all target drivers with an interest os a particular isochronous stream, allocating a channel number and bandwidth as needed, and coordinating the target driver notification of stream starts and stops.

Bus Reset, Isochronous Resource Manager, Bus Manager

In addition to complying with the IEEE 1394 requirements for bus reset processing, such as cancelling pending asynchronous requests, the 1394 Framework provides several bus reset related features.

One of the most severe effects of a bus reset is the re-enumeration of all the nodes on the bus. The 1394 Software Framework assesses the post bus reset topology and determines the new node_IDs for all target driver instances. It can then reissue any uncompleted outgoing asynchronous requests on behalf of the issuing target driver, and each target driver can continue on without concern for their new node number.

As part of the topology evaluation, the 1394 Software Framework also creates a speed map to determine the maximum packet speed between any two nodes. The Framework uses the speed map to select the most efficient speed for target driver outgoing asynchronous requests.

In addition to the topology map and speed map which are part of 1394 bus manager duties, the 1394 Software Framework also contends for isochronous resource manager and bus manager. If it is bus manager, the Framework will ensure that the root is cycle master capable and optimize the gap count.

Hotplug

Another aspect of the Framework's topology evaluation is that it determines which devices, if any, have been removed from the bus and which ones have been added to the bus. For removed devices, the 1394 Software Framework calls into the Solaris Hotplug Framework to notify it that the device is offline. For added devices, the Framework reads the device's configuration ROM to determine the pertinent information, the Global Unique ID and often the Unit_Spec_Id and Unit_Sw_Version, and creates the Solaris "/devices" node using the Solaris Hotplug Framework interfaces.

Building a target driver

To ensure that the Solaris 1394 Software Framework is loaded, target drivers must link with a dynamic dependency on the Framework misc module. This is done using the '-N' flag with ld:

        ld -r -dy -Nmisc/s1394 -o target target1.o target2.o -o target

1394 Device /devices pathname

Solaris device entries for IEEE 1394 devices are created based on the device's global unique ID. The format of the name uses a prefix of "unit@" followed by the GUID in hexadecimal. An example /devices pathname for device A above is as follows ("tdA" is target driver A's minor name):

        /devices/pci@1f,4000/firewire@4,2/unit@0800460200000016,0:tdA

Adding a driver

Although the /devices name for the device is based on the GUID, the device driver itself is bound to the device(s) based on the first pair from the following list to exist in the device's configuration ROM:

        1. Unit_Spec_Id, Unit_Sw_Version
        2. Node_Spec_Id, Node_Sw_Version
        3. Node_Vendor_Id, Node_Hw_Version
        4. Module_Spec_Id, Module_Sw_Version
        5. Module_Vendor_Id, Module_Hw_Version

For further information on the layout of configuration ROM and the meaning of these values, refer to IEEE 1212-1994 Section 8 and IEEE 1394-1995 Section 8.3.2.5. For specific information about configuration ROM for a particular device class, refer to the device class specification.

After parsing configuration ROM and locating one of the pairs as shown above, the Solaris 1394 Software Framework provides the information to the hotplug framework. If a driver is configured to bind to the designated pair, the /devices and /dev entries are created and the driver's attach() routine is invoked. For example, to add a driver for a video conferencing camera which adheres to the 1394 Digital Camera Draft 1.04 (note that hexidecimal letters must be in lower case):

        add_drv -n -i \"firewire00a02d,000100\" tdA

Where firewire is the hardware interface, 00a02d is the Unit_Spec_ID and 000100 is the Unit_Sw_Version.

How the source is organized

All the source and headers for the Solaris 1394 Framework can be found under:

        usr/src/uts/common/io/1394
        usr/src/uts/common/sys/1394

The files themselves break down this way:

        * t1394.c - 1394 Target Driver Interfaces
        * s1394.c - 1394 Services LAyer Initialization and Cleanup Routines
        * s1394_addr.c - 1394 Address Space Routines
        * s1394_asynch.c - 1394 Services Layer Asynch Communications Routines
        * s1394_bus_reset.c - 1394 Services Layer Bus Reset Routines
        * s1394_csr.c - 1394 Services Layer CSR and Config ROM Routines
        * s1394_dev_disc.c - 1394 Services Layer Device Deiscovery Routines
        * s1394_hotplug.c - 1394 Services Layer Hotplug Routines
        * s1394_isoch.c - 1394 Services Layer Isoch Communications Routines
        * s1394_misc.c - 1394 Services Layer Miscellaneous Routines
        * nx1394.c - 1394 Services Layer Nexus Support Routines
        * h1394.c - 1394 Services Layer HAL Interfaces
        * t1394_errmsg.c - Utility function that targets can use to convert an
                           error code into a printable string.
        * s1394_cmp.c - 1394 Services Layer Connection Management Procedures Support Routines
        * s1394_fa.c  - 1394 Services Layer Fixed Address Support Routines
                        (Currently used only for FCP support)
        * s1394_fcp.c - 1394 Services Layer FCP Support Routines

        * t1394.h
            This is the primary header file for the 1394 Framework and it includes
            all other header files listed below.  In addition, it contains all 1394
            Framework interface routine prototypes as well as all data structure and
            defines beginning with the "t1394_" prefix.  (n.b. This one's pretty
            well-commented, if I say so myself.)
        * cmd1394.h
            This file contains all structures and defines for handling asynchronous
            commands.
        * id1394.h
            This file contains all structures and defines for managing a local
            isochronous DMA resource.
        * ieee1394.h
            This file contains general IEEE 1394 defines.
        * ieee1212.h
            This file contains general IEEE 1212 defines.
        * ixl1394.h
            This file contains all structures and defines for utilizing IXL programs.
        * h1394.h
           This file contains the structure and error codes used to communicate between
           the HAL and the rest of the 1394 Software Framework
        * s1394.h
           This file contains all of the structures used (internally) by the 1394
           Software Framework.
        * s1394_impl.h
           This file contains typedefs and defines used by all 1394 Software Framework
           files.

The source for our OpenHCI-compliant HAL driver can be found in:

        usr/src/uts/common/io/1394/adapters
        usr/src/uts/common/sys/1394/adapters

The files here are numerous, so I will hold off saying more about this driver until some later entries.

        hci1394_extern.c       hci1394_misc.c         hci1394.c
        hci1394_ioctl.c        hci1394_ohci.c         hci1394.conf
        hci1394_async.c        hci1394_isr.c          hci1394_s1394if.c
        hci1394_attach.c       hci1394_ixl_comp.c     hci1394_tlabel.c
        hci1394_buf.c          hci1394_ixl_isr.c      hci1394_tlist.c
        hci1394_csr.c          hci1394_ixl_misc.c     hci1394_vendor.c
        hci1394_detach.c       hci1394_ixl_update.c
        hci1394_isoch.c        hci1394_q.c

        hci1394.h              hci1394_extern.h       hci1394_state.h
        hci1394_async.h        hci1394_ioctl.h        hci1394_tlabel.h
        hci1394_buf.h          hci1394_isoch.h        hci1394_tlist.h
        hci1394_csr.h          hci1394_ixl.h          hci1394_tnf.h
        hci1394_def.h          hci1394_ohci.h         hci1394_vendor.h
        hci1394_descriptors.h  hci1394_q.h            hci1394_drvinfo.h
        hci1394_rio_regs.h

And the source for the existing target drivers (mentioned above) can be found in:

        usr/src/uts/common/io/1394/targets/av1394
        usr/src/uts/common/io/1394/targets/scsa1394

OK... So what's next?

So my basic plan is to continue with some discussion of driver attach() and detach() interfaces (see t1394_attach() and t1394_detach() in the source) and basic 1394 event processing. Then to move on to talk about the asynch interfaces, the isoch interfaces, and finally some of the miscellaneous interface routines. At this point, I thought I'd change the focus from a description of the interfaces and how to use them to a more detailed examination of some specific bits of internals code.

But I'm open to suggestions too. If you've read this far and feel like you may be interested in reading more, going through the code yourself, sharing you thoughts and comments, and end up with something specific you'd like to hear about... lemme know.

(2005-06-14 09:00:00.0) Permalink