My head hurts ...

Michael Tibbetts' Blog

Friday June 17, 2005

Scuzzy SCSI Configuration

The good, the bad, the dirty laundry it all comes out now with OpenSolaris. So I'll take the opportunity to air a small bit of the good, bad and dirty laundry associated with the Solaris SCSI and Fibre Channel disk drivers. I worked on these drivers in a prior life at Sun and even got a putback or two into Solaris 10 before moving on to different things.

The primary functions of a device driver are to adhere to an OS interface and control a device (duh!). The Solaris target disk drivers support a very wide range of devices that go beyond what you might expect, to include everything from an internal SCSI drive or CD ROM to an enterprise class multi-terabyte Fibre Channel array. Over time the need has arisen to control unique device behavior at the SCSI command layer which is administered via the Solaris target disk drivers (e.g. sd, ssd). By default these drivers will support any SCSI or Fibre Channel disk or CD ROM device. The device table is used to enable behavior that differs from the default.

This unique configuration is implemented via the driver device table (or .conf file, more on that below).

http://cvs.opensolaris.org/source/xref/usr/src/uts/common/io/scsi/targets/sd.c#507

Currently this table is used to primarily configure retries, throttling, and sundry behaviors. Take a look at the header file for the complete list of configurable behaviors:

http://cvs.opensolaris.org/source/xref/usr/src/uts/common/sys/scsi/targets/sddef.h#1636

So far so good. Using the device table you can configure a set of behaviors for a device matched on its vid (vendor), pid (product/model) and revision strings as defined by its response to the SCSI INQUIRY command. The implementation behind this is a pretty simple bit mask, each bit indicates that a particular behavior needs to be set uniquely from the default. Here's the source for that processing:

http://cvs.opensolaris.org/source/xref/usr/src/uts/common/io/scsi/targets/sd.c#3432

Now we move on to the bad and the downright ugly. Starting with this little gem of a comment:

   1635 /*
   1636 * Driver Property Bit Flag definitions
   1637 *
   1638 * Unfortunaltely, for historical reasons, the bit-flag definitions are
   1639 * different on SPARC, INTEL, & FIBRE platforms.
   1640 */

Nope, I'm not talking about the misspelling of "unfortunately" (although that's a shame), I'm talking about the fact that the bit-flag definitions are different between platforms and device types. This means that setting bit 3 to modify the Not Ready Retry value on the Sparc platform for a SCSI device is used to set the Busy retry value on the Sparc platform for a fibre channel device. Ugh!!! Why? This is the result of a long and sordid history having to do with separate and divergent source files for the sd and ssd driver that have been brought back together over time starting with Solaris 9 and now with Solaris 10. As the source files were brought back together, we ensured that when new configurable options were added they are done so using the same bit for all platforms and device types.

All of this results in a truly hideous nest of #ifdefs that you see in the device table, and the header file bit definitions. The resulting morass makes it very difficult to determine the proper bitset for a specific platform and device table. Having spent more time than I care to remember wading through this crud I put together the table you see below.

Bit	sd (Sparc)	ssd (Sparc)	sd/ssd (x86/x64)
0	throttle (max)	throttle (max)	throttle (max)
1	controller type	not ready retries	controller type
2	not ready retries	busy retries	fabricate device id
3	fabricate device id	fabricate device id	disable caching
4	disable caching	disable caching	play BCD
5	busy retries	controller type	read subchannel BCD
6	play BCD	play BCD	read TOC TRK BCD
7	read subchannel BCD	read subchannel BCD	read TOC ADDR BCD
8	read TOC TRK BCD	read TOC TRK BCD	no READ_HDR
9	read TOC ADDR BCD	read TOC ADDR BCD	read CD XD4
10	no READ_HDR	no READ_HDR	not ready retries
11	read CD XD4	read CD XD4	busy retries
12	reset retries	reset retries	reset retries
13	reservation release time	reservation release time	reservation release time
14	TUR check	TUR check	TUR check
15	throttle (min)	throttle (min)	throttle (min)
16	disable disksort	disable disksort	disable disksort
17	enable LUN reset	enable LUN reset	enable LUN reset

The header file has a brief description of what these behaviors mean and appropriate #defines for all, but to fully understand the functionality you'll need to walk through the driver to see how it modifies behaviors based upon these settings. Some are obvious (e.g. retry and throttle values), some are a bit more obscure and have to do with legacy device behaviors (e.g. disable caching, controller type, etc.). The Red entries above are associated with either legacy SCSI disk or CD ROM behaviors and are all but obsolete. They've been preserved to ensure no field regressions. Take another quick look at the device table and you can see that for the majority of entries we are setting the retry and throttle values as well as indicating support for LUN resets and disabling the use of disksort(9F) which is not as applicable for modern fibre channel arrays.

Can it get any uglier you ask? Well, yes it can. To make it easier to unit test and validate support for a new device as well as debug driver behaviors, the device configuration is exposed via the driver configuration file (e.g. sd.conf, ssd.conf). The purpose of this .conf file access was never for anything more than developer use. This is not a public documented interface of the driver. It is simply a more efficient means of verifying the proper behavior set for a device without having to continually modify, recompile, and reload the driver. However, over time partners and customers began to uncover the fact that they could set some of these options via the .conf file. While never a supported interface for public consumption the fact remains that there are .conf files out there being used and/or recommended so while we would have liked to clean all of this up as part of bringing the sd and ssd source back together, we've preserved it to ensure no customer experienced a regression based on a .conf file they may have been using. So this dirty little secret of backdoor configurability is a blessing and a curse. As a developer its useful to be able to quickly modify the behaviors without having to change the source, recompile, and reload every time. Unfortunately, for some time it wasn't just the developer using this option. So, now that our OpenSolaris community includes many more developers, here is how you can add an entry to sd.conf (for SCSI devices) or ssd.conf (for Fibre Channel devices) for a specific device.

The .conf entry for sd and ssd take the following form:

sd-config-list=
"VENDOR1 PRODUCT1","sd-ver1-tst-data";
sd-ver1-tst-data= 1,0x1F01F,12,1,5,0,0,8,0,0,0,0,0,0,5,60,0,2,0;

Where:
"VENDOR1 " is the contents of the vendor identification field of the device response to a SCSI INQUIRY.
"PRODUCT1" is the contents of the product identification field of the device response to a SCSI INQUIRY.
"sd-ver1-tst-data" is the property array name
sd-ver1-tst-data is the property array definition.

Note: use ssd-config-list for the fibre channel driver.
Note: the vendor and product fields must be 8 bytes in length. If the device response is less than 8 bytes the .conf file entry must be padded with spaces.
Note: the bit fields specified in the above table is based on the current driver implementation. These fields have been updated in legacy releases through patches. If the following error occurs, it means an older version of the driver is installed on the system which does not support all of the parameters. The user can either update to the latest driver patch, or change the field and flags based to the length specified in the message (i.e. n).

data property sd-ver1-tst-data list size is incorrect. No props set.
Size expected: version + 1 flagword + n props.

The property array is defined by version, flag, integer array. The version is always set to 1 and the flag is a bit mask indicating the members of the integer array to be used during device configuration. The user should set only those bits in the mask corresponding to the configurable parameters being set. For example if the user wishes to configure the maximum throttle for sd the flag bit mask should be set to 0x00001. Similarly if the user wishes to configure max throttle, and min throttle for ssd the flag bit mask should be set to 0x10001.

This is a sample sd config list entry for the sparc platform to set the reservation release time, min throttle, and enable LUN resets.

sd-config-list=
"VENDOR1 PRODUCT1","sd-ver1-tst-data";
sd-ver1-tst-data= 1,0x00015,0,0,0,0,0,0,0,0,0,0,0,0,0,60,0,2,0,1;

#
#NOTE: Field Definitions
# sd-ver1-tst-data= version, flags, throttle (max), ctype, not_ready_retry,
# fabricate devid, nocache, busy_retries, rsvd, rsvd, rsvd, rsvd, rsvd,
# rsvd, reset_retries, reservation_release_time, TUR_check,
# throttle (min), disable_disksort, LUN reset

This is a sample ssd config list entry for the sparc platform to set the reservation release time, min throttle, and enable LUN resets.

ssd-config-list=
"VENDOR1 PRODUCT1","ssd-ver1-tst-data";
ssd-ver1-tst-data= 1,0x00015,0,0,0,0,0,1,0,0,0,0,0,0,0,60,0,2,0,1;

#
# NOTE: Field Definitions
#ssd-ver1-tst-data= version, flags, throttle (max), not_ready_retry, busy_retries,
# fabricate devid, nocache, ctype, rsvd, rsvd, rsvd, rsvd, rsvd, rsvd,
# reset_retries, reservation_release_time, TUR_check,
# throttle (min), disable_disksort, LUN reset

I hope this is useful information on how devices are uniquely configured for the Solaris SCSI and Fibre Channel disk drivers.

Technorati Tag: OpenSolaris

Technorati Tag: Solaris

( Jun 17 2005, 05:24:23 PM CDT / Jun 17 2005, 05:28:36 PM CDT ) Permalink Comments [2]
Trackback: http://blogs.sun.com/mtibbets/entry/scuzzy_scsi_configuration

Trackback URL: http://blogs.sun.com/mtibbets/entry/scuzzy_scsi_configuration

Comments:

That's great piece of knowledge. Thanks!

Posted by Leon Koll on April 02, 2008 at 02:07 AM CDT #

Thanks for writing this.

Posted by passerby on June 25, 2009 at 04:25 AM CDT #