Weblog

Tuesday June 14, 2005

Solaris USB Architecture (USBA)

Solaris USB Architecture (USBA)

USB Architecture (USBA) was first integrated into Solaris 8 and its main focus was support for USB keyboard and mouse. At the time, it was felt that USB mass storage devices and USB printers had not much of a future.

The Human Interface Device driver (hid) was the first driver that was developed. Since it was a streams driver, it made sense to use mblks (msgb(9S)) for moving data from client drivers to host controller drivers (ohci, uhci). It turned out to be a convenient mechanism, even for non-streams drivers and you will find that mblks are widely used in USBA.

When the first USB mass storage devices appeared (eg. IOMEGA ZIP), a bridge nexus driver called scsa2usb was developed which allowed us to use sd(7D) to actually manage the disk and eventually support st(7D) as well. Although conceptually easy (just converting SCSA requests into USBA requests), it turned out to be a troublesome driver. Many USB mass storage devices did not handle common SCSI commands such as Mode Sense or Start/Stop well and just died. We started a blacklist of troublesome devices and provided work arounds in scsa2usb.conf. The most significant and far-reaching decision we made was that all USB mass storage devices should be treated as "removable media" devices. This provided a consistent model for all USB devices and allowed vold to manage them. Unfortunately making vold hotplug aware took a few more years.

Other drivers developed for USBA were USB audio (usb_ac and usb_as), hub driver (hubd) and USB serial (usbser_edge).

					   +----------+
					   |   sd     |
+------+--------+--------+--------+--------+----------+-------------+
| hid  |  hubd  | usb_ac | usb_as | usbprn | scsa2usb | usbser_edge |
+------+--------+--------+--------+--------+----------+-------------+
| 		USB Architecture (USBA 0.0)                         |
+---------------------------------------+---------------------------+
|       ohci                            |     uhci    		    | 
+---------------------------------------+---------------------------+

These drivers exposed many shortcomings in the USBA framework. The error handling was very poorly defined and it was not easy to write a robust USBA client driver. Because client drivers received callbacks in interrupt context where blocking is prohibited, complicated state machines were necessary.

When USB 2.0 spec came out, we decided that the old USBA framework was not a good foundation for USB 2.0 device support and that the USBA interfaces to the client drivers were not suitable for making public and adding to the DDI.

A new framework was written from scratch, USBA 1.0. One of our requirements was that a misbehaving client driver (eg. blocking in a callback handler) should have no impact on other client drivers or the system. This was achieved by extensive use of threads and taskq(9F). Each pipe has its own taskq which is used for callbacks and asynchronous requests. This also allowed USBA to do most of the stall handling and pipe clearing.

By providing each client driver with a bunch of its own threads, the need for complex state machines was eliminated, each client driver was well-contained, and would almost never run in interrupt context. There is one exception (refer to hcdi.c) when taskq_dispatch fails which rarely happens fortunately.

ehci(7D) is the USB 2.0 host controller driver. A USB 2.0 controller chip usually has one ehci and 2 ohci or uhci companion controllers which handle the USB 1.x devices. The routing to ohci/uhci is done in hardware.

The new USBA framework was integrated early into Solaris 10 development gate (2001).

Marketing required USB 2.0 support in Solaris 8 and Solaris 9 but unfortunately it was not allowed to replace a framework in a patch release for SB1500 and SB2500. Therefore, we came up with the idea of a "Dual Framework" where we run the old USBA and new USBA 1.0 framework side by side with the option of running the new USBA 1.0 only. The USBA 1.0 drivers would bind to a PCI USB 2.0 card and the old USBA driver would bind to onboard USB 1.x hardware. Although somewhat confusing to users, it provided a lot of feedback and made Solaris 10 USBA much better.

When we added the USBA interfaces to the DDI, we created a new revision of USBA, called USBA 2.0 to emphasize that this framework was designed for USB 2.0.

					   +----------+
					   |   sd     |
+------+--------+--------+--------+--------+----------+-------------+------+
| hid  |  hubd  | usb_ac | usb_as | usbprn | scsa2usb | usbser_edge | ugen |
+------+--------+--------+--------+--------+----------+-------------+------+
| 		USB Architecture (USBA 2.0)	                           |
+---------------------------+----------------------+-----------------------+
|       ohci                |     uhci             |  ehci                 | 
+---------------------------+----------------------+-----------------------+

The most interesting USB client driver in Solaris 10 is ugen(7D) which allows the entire USB device management to be written in userland. Microsoft has announced a similar driver (winusb) and this is clearly the way to go for USB device support. Most USB device support does not really need to run in the kernel.

The most interesting work in USBA is done in usbai_req.c and usbai_pipe_mgt.c. The former file does all the request handling, the latter all the pipe management. These two files are good starting points in understanding how USBA works.

To get started on writing a new driver from scratch, usbskel.c may provide interesting reading.

Debugging USB client drivers is made easy by a circular trace buffer which can be managed using dcmds in mdb. Note the various DPRINT_Lx macros in the client drivers. L4 is for function tracing, L3 is for interesting information, L2 for recoverable errors, and L1 and L0 for serious problems (refer to usba.c).