Friday November 18, 2005
Friday November 18, 2005 Introduction
USB serial adapter is a USB device that tunnels asynchronous serial protocols, such as RS-232 and RS-485, through USB. It allows to add serial lines to computers with insufficient number of built-in serial ports. The most common need for these devices is on laptops to connect other machines' serial consoles, GPS devices, PDAs, etc.
Traditionally Solaris supported class devices only, such as printer class or mass storage class. A class driver can support devices from various vendors as long as they follow the class specification. Unfortunately, USB serial devices do not belong to any class: every device vendor has to invent a unique, often proprietary protocol, and therefore requires a separate driver.
Eventually a project was funded to support one type of USB serial adapters. We selected the Edgeport series from Inside-Out Networks. After a long break, drivers for Keyspan devices and those based on the Prolific PL2303 chip are coming soon to the next Solaris Express build. Due to the proprietary nature of device protocols, it may take some time for the driver sources to propagate to OpenSolaris.
This is the first in a series of articles about writing USB serial drivers for Solaris.
Devices
A typical USB serial device consists of a USB interface IC, firmware executed by an embedded CPU, and one or more UARTs (Universal Asynchronous Receiver-Transmitter). Firmware can be downloadable or upgradable by the driver. Firmware is responsible for bridging the two hardware interfaces, encoding outgoing and decoding incoming USB packets. At the other end, the driver similarly decodes/encodes USB packets and provides applications with the standard serial port API.
Interfaces
While each protocol is unique, functionality is largely the same: set serial parameters, such as baud rate and flow control, get modem status, set control lines, transmit and receive data. All these functions are available to applications via the standard UNIX termio(7I) programming interface.
The Solaris terminal subsystem, including the termio interface, is based on the STREAMS framework. STREAMS code is not easy to write, especially serial drivers, due to extreme asynchronicity and everything that it entails. It took some time for se(7D) and su(7D) to stabilize and we're still finding bugs in them once in a while.
It comes as no surprise that we decided to put USB termio implementation into a common module used by all USB serial drivers. We call it generic serial driver, GSD.
The bottom part of the driver that implements the vendor-specific USB protocol is called device-specific driver, DSD. The interface between these two parts is well-defined, we call it DSDI.
DSD in turn talks to the kernel USB framework, also known as USBA.
These layers are illustrated by the following diagram:
Note that GSD does not interact with USBA directly. In fact, GSD turned out generic enough to write any kind of serial driver, not just USB: it is possible, for example, to reimplement se(7D) and su(7D) using GSD. The only USB-specific features utilized in GSD are logging and hotplug events.
Files
- GSD is a misc module:
/kernel/misc/usbser(and the 64-bit subdir). - DSDs are driver modules:
/kernel/drv/usbs*(and the 64-bit subdir) - GSD sources:
usr/src/uts/common/io/usb/clients/usbser, usr/src/uts/common/sys/usb/clients/usbser - DSDI definition:
usr/src/uts/common/sys/usb/clients/usbser/usbser_dsdi.h - DSDs should be located in
usr/src/uts/common/io/usb/clients/usbser/<driver_name>directories.
Tags:
Solaris
OpenSolaris
USB
(2005-11-18 15:43:29.0)
Permalink
Comments [2]
Thursday November 17, 2005 UPDATE 19-Jul-2006:
It looks like this blog entry is still getting hits from search engines and the USB FAQ, so here's an update. Since I wrote the entry, the project I said was under way has now been integrated in Nevada Build 36, Solaris Express 4/06 and Solaris 10 6/06 Release. It is mentioned in Dan's what's new blog as:
* Hotpluggable drives are now better able to accomodate EFI-labels and device IDs, both of which are very important to supporting ZFS on USB and Firewire disk drives. [6348407]
If you use one of these releases or later, the hack described below is not necessary (and as Brian noted, may cause some error messages).
END UPDATE
ZFS is awesome. I'd bust some zeerleading moves for y'all, but I misplaced my pom poms and it wasn't going to be pretty anyway. Instead I will tell you how to try ZFS on your laptop (or any computer without a spare fixed disk). There is no supported way to create ZFS pools on USB disks yet, although a project is under way to rectify this (you can have limited ZFS functionality using lofi(7D)). What follows is a hack, use it at your own risk.
Before we proceed, you should get vold(1M) out of the way:
# svcadm disable volfs
An important thing to know in this context is that you can't create a pool on removable media devices, i.e. the ones whose storage media can be removed. DVD drives and flash readers are removable media, hard disks are not. So you'll need one or more USB hard disks. Most USB hard disks and IDE-to-USB enclosures should work. Thumb drives are not likely to work. In this experiment I'm using two USB 2.0 disks: a 20GB LaCie and a 40GB IOGEAR.
For the reasons that are beyond this blog entry, the Solaris USB driver presents any USB storage device as removable media. A simple command to list removable and non-removable disks is format(1M). Quoting the man page:
Removable media devices are listed only when users execute
format in expert mode (option -e).
Run format with and without -e and notice the difference:
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c1d0
/pci@0,0/pci-ide@8/ide@0/cmdk@0,0
Specify disk (enter its number): ^C
# format -e
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c1d0
/pci@0,0/pci-ide@8/ide@0/cmdk@0,0
1. c4t0d0
/pci@0,0/pci1297,a550@2,2/storage@6/disk@0,0
2. c5t0d0
/pci@0,0/pci1297,a550@2,2/storage@5/disk@0,0
Specify disk (enter its number): ^C
c4t0d0 and c5t0d0 are USB disks.
What we're going to do is tell the USB driver not to treat hard disks as removable media. This can be done by appending the following line to /kernel/drv/scsa2usb.conf file:
attribute-override-list = "vid=* removable=false";
Reboot for these changes to take effect. Now all your USB hard disks are going to be treated as fixed (but you can still hotplug them). You can verify that by running format without -e option - if it still doesn't list your USB disk, most likely it's one of the rare samples that pretends to be removable media and that's a hack for another day.
Now you can use these disks just like any fixed disk:
# zpool create usbpool mirror c4t0d0 c5t0d0
# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
usbpool 27.8G 33.0K 27.7G 0% ONLINE -
# zpool status
pool: usbpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
usbpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c4t0d0 ONLINE 0 0 0
c5t0d0 ONLINE 0 0 0
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
usbpool 32K 27.5G 8K /usbpool
Tags:
Solaris
OpenSolaris
USB
ZFS
(2005-11-17 16:52:19.0)
Permalink
Comments [7]
Thursday September 08, 2005 Free Code Graphing Project was an attempt to visualize Linux kernel source code. Now that Solaris is free code, too, why not try to graph it. This blog entry describes my little effort of debugging the thing into existence. I can't afford to spend any more time on this, so hopefully someone picks the baton.
The end result is a huge PostScript file that looks like this:
Sample close-up, common/vm/seg_kmem.c:
Roughly, the algorithm used:
- Function: sequential code is a line, loop is an arc, branch splits the line.
- Color: normal functions are blue, static functions are green.
- File: functions are put in boxes, sorted by size and arranged in a spiral.
- Ring: Files are put in boxes, sorted by size and arranged in a circle.
- Rings can be broken into sectors.
Here's how rings are defined for Linux:
RING1:=init lib mm kernel ipc RING2:=net fs RING3:=$(subst $(KERNEL_DIR)/,,$(wildcard $(KERNEL_DIR)/arch/*)) RING4:=drivers crypto sound security
Solaris has a bit different directory structure. Here's what I used:
RING1:=common/os common/vm common/c2 common/cpr common/disp common/dtrace common/exec common/krtld common/syscall RING2:=common/fs common/inet common/ipp common/ktli common/rpc RING3:=i86pc sparc sun sun4 sun4u sun4v RING4:=common/io
Steps to make it work in Solaris
1. Prerequisites:
- at least 400MB of disk space
- gcc
- gmake
- flex
- GNU indent
- OpenSolaris source
- lgp-2.6.0a with my fixes (original is here)
2. Set you KERNEL_DIR environment variable to usr/src/uts subdirectory of OpenSolaris source, e.g.:
KERNEL_DIR=$HOME/solgraph/usr/src/uts; export KERNEL_DIR
3. Preprocess OpenSolaris code, since the fcgp parser does not like Solaris coding style (ignore warnings):
chmod -R +w $KERNEL_DIR
find $KERNEL_DIR -name '*.c' -exec indent -npsl {} \;
4. Modify your path to look in /usr/xpg4/bin first: Alternatively you can replace all 'tail -n number' occurences in fcgp scripts with 'tail -number', but it's easier to just use /usr/xpg4/bin/tail.
PATH=/usr/xpg4/bin:$PATH; export PATH
5. If using X, turn off the bell. Fcgp spits out some escape sequences that would drive you nuts (they might not look pretty on some terminals either):
xset -b
6. Kick off the build process. Takes about 9 minutes on my Ferrari 3400.
cd lgp-2.6.0a-solaris gmake CC=gcc
7. Generate viewable/printable postscript files. Examples for a single file and for a 6x6 poster:
./posterize a4 1 ./posterize a4 6
See README for more information on printing and assembling a poster.
To do list
- Fix the parser to handle Solaris coding style.
- Some files, e.g. common/c2/audit_syscalls.c and common/disp/ts.c, are not rendered correctly, resulting in very large bounding boxes. The rest of the files are scaled down too much. This problem might take care of itself when parser is fixed.
- Replace the Tux in the center of the circle with OpenSolaris logo. See tux.ps.
- Figure out what's in 'codemap' and 'zoom' directories and make it work.
- Come up with an easy way to locate files and functions on the map.
I won't have time for these in the next two months, so feel free to pick up where I left. I can be reached at artem dot kachitchkin at sun dot com.
Tags:
Solaris
OpenSolaris
(2005-09-08 21:36:01.0)
Permalink
Comments [2]
Thursday July 14, 2005 A minor instance of driver confusion
Most driver writers are familiar with device minor numbers. But a notion of an instance number is specific to Solaris and it is often incorrectly assumed that minor == instance. Let's see if we can see clear that up.
Instance number
A driver handles devices of a certain type. There can be multiple instances of the same device type in the system. For example, there are two asynchronous serial devices (UARTs) on my system, handled by the asy(7D) driver and represented by two nodes in the device tree:
$ prtconf -D | grep asy
asy, instance #0 (driver name: asy)
asy, instance #1 (driver name: asy)
These are the two instances of the same driver: sharing binary code in the kernel address space, but maintaining separate data 1. Instances are numbered, in this case 0 and 1. Instance numbers are assigned permanently to physical devices in the system; the mapping is maintained in the /etc/path_to_inst file:
$ grep asy /etc/path_to_inst
"/isa/asy@1,3f8" 0 "asy"
"/isa/asy@1,2f8" 1 "asy"
There are also drivers without respective hardware, called pseudo drivers. An example of a pseudo driver is random(7D), the random number generator driver. Pseudo drivers have no physical device nodes to attach to. We have to synthesize artificial instances via driver.conf file:
$ cat /kernel/drv/random.conf | egrep -v '^#|^$'
name="random" parent="pseudo" instance=0;
The system obeys and creates a node for instance 0:
$ prtconf -DP | grep random
random, instance #0 (driver name: random)
Drivers can retrieve their instance number with ddi_get_instance().
Minor number
Outside the kernel, devices are identified by their {major, minor} pairs, also know as dev_t. The major number range is owned by the system. It assigns major numbers when drivers are installed and keeps the list in /etc/name_to_major:
$ grep asy /etc/name_to_major
asy 106
The minor number range is owned by the driver. A minor number becomes visible to the userland when the driver creates a minor node using ddi_create_minor_node() 2, which takes minor number as one of its arguments:
int ddi_create_minor_node(dev_info_t *dip, char *name, int
spec_type, minor_t minor_num, char *node_type, int flag);
Problem is, the minor number range is shared among all driver instances, so each instance can only use a subrange. If each instance needs exactly one minor node, then the node's minor number can equal instance number, i.e. the 4th argument in ddi_create_minor_node() is set to instance number.
Things get a bit fancier in case of multiple minor nodes per instance. Serial drivers typically create two nodes per node: one in /dev/term and one in /dev/cua:
$ ls -lL /dev/{term,cua}/[ab]
crw------- 1 uucp uucp 106, 131072 Jun 29 19:04 /dev/cua/a
crw------- 1 uucp uucp 106, 131073 Jun 29 19:04 /dev/cua/b
crw-rw-rw- 1 root sys 106, 0 Jun 29 19:04 /dev/term/a
crw-rw-rw- 1 root sys 106, 1 Jun 29 19:04 /dev/term/b
106, as we saw earlier, is asy major number. Minor numbers 0 and 131072 (0x20000) belong to instance 0, and 1 and 131073 (0x20001) to instance 1. The instance/minor mapping is not 1:1 here.
Conversion
The driver holds the algorithm for mapping instances into minors and vice versa. When the kernel needs to find out instance number from a minor number, it has to ask the driver by calling getinfo(9E) entry point with the DDI_INFO_DEVT2INSTANCE command. One example could be found in spec_open() which calls e_ddi_hold_devi_by_dev().
1 Per-instance data is called soft state and explicitly supported in DDI, see ddi_soft_state(9F) man page.
2 A whole lot of action can be triggered by ddi_create_minor_node(). See dacfc_match_create_minor() and /etc/dacf.conf file on your nearest Solaris system.
Tags:
Solaris
OpenSolaris
(2005-07-14 15:17:13.0)
Permalink
Comments [0]
Tuesday June 14, 2005 From a driver writer's perspective, CPUs are amazing devices: they manage to stay extremely reliable while implementing extremely complex logic. Neither is true of most peripheral devices. It is usually not practical to put huge effort into design and testing of cheap, short-lived devices. Which to some degree explains device drivers' reputation for being too obscure, overcomplicated and continuously changing: they reflect imperfections of their hardware... and sometimes of their programmers, but that is an altogether different subject - this blog is about a bug that exists in some IEEE 1394 (FireWire) mass storage devices and how it's been worked around in the scsa1394(7D) driver.
1394 mass storage protocol is based on the Serial Bus Protocol 2. SBP-2 allows to encapsulate arbitrary command sets on buses compliant with IEEE 1212. In case of scsa1394, the IEEE 1212 compliant bus is IEEE 1394 and it transports SCSI commands submitted by SCSI target drivers, such as sd(7D) and st(7D), or via uscsi(7I) interface.
For SCSI commands with data phase, scsa1394 has to map data buffers for DMA transfer. When a buffer cannot be mapped into contiguous I/O memory, the kernel attempts to map it into non-contiguous chunks described by the ddi_dma_cookie(9S) structure. Each cookie contains a DMA address of the memory chunk and its size. Cookies are then programmed into device's DMA engine. This type of DMA transfers is known as scatter-gather.
SBP-2 provides a facility for DMA scatter-gather called page tables. A page table is an array of entries of the following format 1:
31 0 +----------------+----------------+----------------+----------------+ | segment_length | segment_base_hi | +----------------+----------------+----------------+----------------+ | segment_base_lo | +----------------+----------------+----------------+----------------+
Page table itself must be located in contiguous I/O space. Once the driver prepares an ORB 2 and attaches a valid page table to it, it signals the device to read them in and perform data transfer. SBP-2 segments fit neatly into Solaris DMA cookie concept: segment length corresponds to cookie size and segment base to cookie address.
SBP-2 does not put any restrictions, other than bit width, neither on segment length, nor the total length of segments in a page table. However, some devices would not function correctly unless page tables meet two requirements:
- segment length == 4K;
- total length < 128K;
Evidently, not all vendors use dedicated test software for their devices, some limit themselves to compliance testing on a few target OS/CPU. As far as we know, this bug only exists in devices based on SYM13FW* chip series, hence known as the "symbios bug".
Solaris driver had to provide a workaround. Satisfying symbios page table limits requires that we divide cookies into 4K segments, which breaks the neat 1:1 mapping between DMA cookies and SBP-2 segments. So I had to introduce an additional mapping layer described by struct scsa1394_cmd_seg:
typedef struct scsa1394_cmd_seg {
size_t ss_len;
uint64_t ss_daddr;
uint64_t ss_baddr;
t1394_addr_handle_t ss_addr_hdl;
} scsa1394_cmd_seg_t;
ss_daddr here refers to segment's DMA address and ss_baddr to its 1394 bus address. scsa1394_cmd_dmac2seg() function is responsible for turning cookies into segments. Page table is then created from the segment array in scsa1394_sbp2_seg2pt().
If you ever used Solaris DDI DMA routines, you might think: why not just use ddi_dma_attr(9S) structure to limit cookie size by setting dma_attr_seg to 0xFFF? This is because dma_attr_seg cannot be less that base page size, so 4K would not work on SPARC (or x86 should the page size ever increase).
As if that wasn't enough, there seems to be no way to distinguish between devices with the symbios bug and devices without it. No combination of configuration ROM3 keys provides reliable indicator of a buggy device. This led me to a difficult decision to enable the symbios workaround by default, sacrificing performance for data integrity. Then I thought: we can't blacklist bad devices, but maybe we could whitelist the good ones. That is how the white list was invented:
scsa1394_bw_list_t scsa1394_sbp2_symbios_whitelist[] = {
{ SCSA1394_BW_ONE, 0x0a27 }, /* Apple */
{ SCSA1394_BW_ONE, 0xd04b } /* LaCie */
};
It contains vendor IDs of those companies that have never, to the extent of our knowledge, produced devices based on the buggy chip. We'll be adding more to the list in the future. In order to test a device for the symbios bug, the workaround can be disabled by setting scsa1394_wrka_symbios variable to 0:
temporarily in mdb: # echo scsa1394_wrka_symbios/W 0 | mdb -kw or permanently in /etc/system: set scsa1394:scsa1394_wrka_symbios = 0
1 These are so called unrestricted page tables. SBP-2 also defines normalized page tables, but at the time of this writing they are not used in scsa1394.
2 ORB: Operation Request Block, prepared by the initiator in I/O memory and read by the target. ORB is a polymorphic data structure: see usr/src/uts/common/sys/sbp2/defs.h for its multiple incarnations.
3 Every 1394 device contains a special address region called configuration ROM. It's a hierarchical key-value structure that describes various device attributes such as device class, vendor/device ID, etc. For those familiar with PCI, it is similar in purpose to PCI configuration space: it is basically what makes these devices self-identifying.
Technorati Tag: OpenSolaris
Technorati Tag: Solaris
Technorati Tag: FireWire
Technorati Tag: mdb
(2005-06-14 08:00:00.0)
Permalink
Comments [0]
Monday June 13, 2005
My name is Artem Kachitchkine2, I'm an engineer in the Operating Platforms Group. Since I joined Sun a few years ago (virtually straight out of college) I've been working on a number of I/O related projects. I started by fixing bugs in Solaris serial and parallel port drivers, then participated in the bringup of the Sun Blade workstation series. After that, I moved into the USB land and later took ownership of the 1394 (FireWire) framework in Solaris, during which I wrote av1394(7D) and scsa1394(7D) drivers and introduced framework extensions to support new drivers. I was also part of the team that ported Solaris Fibre Channel stack aka Leadville to x64 platforms. Lately I've been busy working on various aspects of mass storage and removable media management in Solaris.
Update Oct 2007: After finishing Tamarack and starting WWAN OpenSolaris projects, I joined the networking group and now contributing to Brussels.
1AT&C0 is the Hayes modem command that means "assume data carrier is present". I often feel that way during blogging, i.e. assuming someone is listening.
2 Do not attempt to pronounce or memorize my last name, it will hurt and I don't want to be liable.
(2005-06-13 16:33:46.0)
Permalink
Comments [8]
| archives | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| links | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| referers | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Today's Page Hits: 75 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||