Saturday July 09, 2005

The 1394 Software Framework - Outgoing Asynchronous Requests (Part 1 of 2)

Continuing my discussion of the Solaris 1394 Software Framework, in this post I'm going to go into some detail on the methods by which a 1394 target driver can transmit outgoing asynchronous requests and receive the corresponding responses.

Command Allocation/Freeing

Handling of all asynchronous packets is acheived via commands. A command is a data structure which contains fields for the necessary packet components as well as command options and other fields used to specify the desired command processing.

To acquire a command structure for transmitting asynchronous requests and receiving responses, the 1394 target driver calls t1394_alloc_cmd(). This routine allocates and returns a pointer to a cmd1394_cmd_t structure (which I'll say more about below.) Note: This same command structure is also used for handling incoming asynchronous requests, although some fields are used differently.

A 1394 target driver can re-use a command after it has completed (see below), but it should not reissue or alter any fields for any command that is still pending. And when a target driver no longer needs a command, it will call t1394_free_cmd() to release the command structure back to the Framework. Note: 1394 target drivers are responsible for freeing all of their allocated commands before detaching (or the detach will not be allowed - see my previous blog entry.)

Command Action

After initializing the relevant parameters for the outgoing request, the 1394 target driver calls the appropriate asynchronous command routine; either t1394_write(), t1394_read(), or t1394_lock().

The command itself often takes some time to complete, since a packet must be sent over the bus to the destination node and the 1394 Framework must await the response. Therefore, the Framework gives the target driver the option to block or not block pending command completion (more on this below).

Once the command is "handed off" to the 1394 Software Framework, the target driver should not re-use or modify the same allocated command until the target driver can determine that the requested action has fully completed.

A command is "completed" when the bus transaction(s) used to perform the command have finished. The command's cmd_result field indicates either success (CMD1394_CMDSUCCESS) or failure, where a failure is indicated by an error code (See 'Error Codes' below for details).

Command Completion

Target drivers have three options for determining command completion status:

• completion callback - Within the command structure, the 1394 target driver may specify a completion_callback() entry point which the 1394 Software Framework will call once the command completes. The target driver may also specify a cmd_callback_arg for its own use to track per-command state or other pertinent information.
  When the 1394 Software Framework calls the command's completion_callback() it provides the address of the completed cmd1394_cmd_t command. The target driver can then retreive its own cmd_callback_arg and the completion cmd_result from the command structure.^[1]
• polling - Another option for target drivers is to poll for command completion status. By specifying a completion_callback() address of NULL, the 1394 Software Framework won't notify the target driver when a command completes. Although, there isn't any notification, the Framework still tracks the command and its completion status, which the target driver may retrieve at any time by reading the command's cmd_result field (defined as volatile). A cmd_result of CMD1394_NOSTATUS (more on this below) indicates that the command is still in progress and has not completed.^[2]
• blocking - The third option, enabled by setting the CMD1394_BLOCKING flag, causes the target driver to block in the t1394_ call pending completion of the issued command. When the command completes, the call returns and the target driver checks the cmd_result field to determine completion status. If a completion_callback() is specified for the command, it won't be called.^[3] If a blocking command is attempted, the 1394 Framework will return a cmd_result of CMD1394_INVALID_CONTEXT.

Bus Reset and Command Handling

The IEEE 1394 bus is reset when devices are added to or removed from the bus (or for a variety of other reasons). When this happens, all devices on the bus are re-enumerated and can possibly be assigned different bus addresses (referred to as "Node IDs") from the ones they each had prior to the bus reset.

From the IEEE 1394 protocol perspective, when a bus reset occurs all the pending and in-progress command requests are canceled. Target drivers have two options with respect to processing of any outstanding commands.

• Automatic Retry (default) - Following a 1394 bus reset, the 1394 Software Framework automatically determines the new Node ID for each device on the bus, and therefore for each target driver. Since the Framework always tracks the current Node ID for each device, it automatically reissues any pending (i.e. non-completed) commands using the update Node IDs.
• Cancelation - In some cases, a target driver may not want a command automatically reissued on its behalf. By setting the command's CMD1394_CANCEL_ON_BUS_RESET flag, it specifies that the Framework should cancel the command in this circumstance. The result is that when a bus reset occurs while the command is being processed, the command cmd_result is set to CMD1394_EBUSRESET and either the command completion routine is called (if one was specified) or if it is blocking, the issuing call will be unblocked.

Asynch Command Structure

/* cmd1394_cmd: cmd1394 - common command type */
typedef struct cmd1394_cmd
{
        int                     cmd_version;
        volatile int            cmd_result;
        cmd1394_flags_t         cmd_options;
        cmd1394_cmd_type_t      cmd_type;
        void                    (*completion_callback)(struct cmd1394_cmd *);
        opaque_t                cmd_callback_arg;
        uint64_t                cmd_addr;
        uint_t                  cmd_speed;
        uint_t                  bus_generation;
        uint_t                  nodeID;
        uint_t                  broadcast;
        union {
                cmd1394_quadlet_t       q;
                cmd1394_block_t         b;
                cmd1394_lock32_t        l32;
                cmd1394_lock64_t        l64;
        } cmd_u;
} cmd1394_cmd_t;

• cmd_version - This field is set by the 1394 Framework when the command is allocated and is read-only for target drivers. For the current release, this version field is always set to T1394_VERSION_V1.
• cmd_result - The cmd_result field is filled in by the Framework upon completion of processing for the request command. If the command completes due to a parameter error, the invoked command (for example t1394_write()) returns DDI_FAILURE and cmd_result contains a code indicating the reason for the failure. (See 'Error Codes' below for details.) The cmd_result field is defined as volatile to permit target drivers to poll for updates rather than receiving a completion_callback().
• cmd_options - The target driver sets cmd_options to tailor the Framework's handling of the command. (See 'Command Options' below for details)
• cmd_type - The target driver sets cmd_type to indicate to the Framework the type of request transaction to be performed. (See 'Command Types' below for details)
• completion_callback() - The target driver sets completion_callback() to direct the Framework to issue the callback upon command completion. If non-NULL, this function is invoked by the Framework upon completion of the command. If NULL or if the CMD1394_BLOCKING flag is set, completion_callback() is not called.
• cmd_callback_arg() - A value used only by the target driver. The target driver can use this field to track the command state information or any other pertinent information. The target driver can set this field on a per-command basis, i.e. can set a different value in each command.
• cmd_addr - The cmd_addr field is the IEEE 1394 address of the destination node. Typically the target driver sets cmd_addr to the 48-bit destination offset for the request. The 1394 Software Framework overwrites the high-order 16 bits by filling in the current node ID for the device. The target driver can also override this automatic addressing feature by setting the CMD1394_OVERRIDE_ADDR option.
• cmd_speed - The 1394 Framework's default behavior is to transmit the command request packet(s) at the fastest possible speed, given the current 1394 bus topology, between the Solaris host and the target device. To override this feature, the target driver may specify the CMD1394_OVERRIDE_SPEED, set cmd_speed, and the 1394 Framework will transmit the request packet(s) at the speed specified in cmd_speed. If the CMD1394_OVERRIDE_SPEED option is not enabled, the 1394 Framework ignores cmd_speed. Valid values for this field are IEEE1394_S100, IEEE1394_S200, and IEEE1394_S400.^[4]
• bus_generation - The Framework ignores bus_generation unless the target driver sets the CMD1394_OVERRIDE_ADDR option. The bus_generation number indicates for which bus reset generation the command is valid. The generation count is a counter that the Framework increments each time a bus reset occurs. Since target drivers use CMD1394_OVERRIDE_ADDR to specify a raw destination address, the bus_generation number is used to ensure that the node ID in the raw destination is not stale. The target driver obtains the current generation number initially from t1394_attach() and subsequently from the DDI_DEVI_BUS_RESET_EVENT callback. (See 'Attach, Detach, and Events').
• nodeID - This field is ignored for outgoing asynchronous requests.
• broadcast - This field is ignored for outgoing asynchronous requests.
• cmd_u - The cmd_u field contains that data specific to the cmd_type request. For read quadlet and write quadlet requests, the target driver uses cmd_u.q. For read and write block requests, the target driver uses cmd_u.b. For 32-bit and 64-bt lock requests, the target driver uses cmd_u.l32 and cmd_u.l64, respectively.

Command Types

/*
 * cmd1394_cmd.cmd_type
 *    Used to select/indicate the request packet type
 */
typedef enum {
        CMD1394_ASYNCH_RD_QUAD  = 0,
        CMD1394_ASYNCH_WR_QUAD  = 1,
        CMD1394_ASYNCH_RD_BLOCK = 2,
        CMD1394_ASYNCH_WR_BLOCK = 3,
        CMD1394_ASYNCH_LOCK_32  = 4,
        CMD1394_ASYNCH_LOCK_64  = 5
} cmd1394_cmd_type_t;

• CMD1394_ASYNCH_RD_QUAD - A target driver uses this type to specify a command for a read quadlet request. To issue this request, the target driver calls t1394_read().
• CMD1394_ASYNCH_WR_QUAD - A target driver uses this type to specify a command for a write quadlet request. To issue this request, the target driver calls t1394_write().
• CMD1394_ASYNCH_RD_BLOCK - A target driver uses this type to specify a command for a read block request. To issue this request, the target driver calls t1394_read().
• CMD1394_ASYNCH_WR_BLOCK - A target driver uses this type to specify a command for a write block request. To issue this request, the target driver calls t1394_write().
• CMD1394_ASYNCH_LOCK_32 - A target driver uses this type to specify a command for a 32-bit lock request. To issue this request, the target driver calls t1394_lock().
• CMD1394_ASYNCH_LOCK_64 - A target driver uses this type to specify a command for a 64-bit lock request. To issue this request, the target driver calls t1394_lock().

Command Options

A target driver uses these options to tailor the Framework's command processing behavior.

/*
 * cmd1394_cmd.flags
 *    Used to select the request's behavior, including
 *    how the destination address is determined, how
 *    a large request will be broken into smaller requests,
 *    whether the command should be resent after a
 *    bus reset has happened, etc.
 */
typedef enum {
        CMD1394_CANCEL_ON_BUS_RESET     = (1 << 0),
        CMD1394_OVERRIDE_ADDR           = (1 << 1),
        CMD1394_OVERRIDE_MAX_PAYLOAD    = (1 << 2),
        CMD1394_DISABLE_ADDR_INCREMENT  = (1 << 3),
        CMD1394_BLOCKING                = (1 << 4),
        CMD1394_OVERRIDE_SPEED          = (1 << 5)
} cmd1394_flags_t;

• CMD1394_CANCEL_ON_BUS_RESET - If the target driver sets this option, then if a bus reset occurs before this command has completed, the Framework completes the command by calling the command's completion_callback() routine. When this occurs, the Framework sets the command cmd_result to CMD1394_EBUSRESET.
  If the target driver doesn't set this option, then if a bus reset occurs before this command has completed, the Framework holds onto the command and reissues the request(s) when the target device's new node ID has been determined.
• CMD1394_OVERRIDE_ADDR - The target driver sets this option to bypass the automatic Framework node addressing. The Framework will not fill in the high-order 16 bits in the command's cmd_addr field. The address is transmitted on the bus as is. The Framework fragments the message as needed based on the current max_payload value. Commands issued with CMD1394_OVERRIDE_ADDR set are always canceled on a bus reset, regardless of the setting of the CMD1394_CANCEL_ON_BUS_RESET option.^[5]
  
• CMD1394_OVERRIDE_MAX_PAYLOAD - The target driver sets this option to direct the Framework to use the specified max_payload in place of the prevailing maximum payload size. This is used to determine how to split a read or write block request into multiple packets.
  Using this option, the target driver can specify a max_payload that is smaller than the current maximum payload size. But the 1394 Framework will not transmit a packet payload with a size greater than the current maximum payload size.
  Target driver writers are cautioned when using this option as a way to have the Framework automatically split a data buffer into multiple packets. For error handling reasons, the Framework transmits one packet at a time and only transmits the next packet after the current packet transaction has successfully completed. If an error occurs for one of these fragments, cmd_result reflects the error and bytes_transferred indicates the partial number of bytes transferred. In some circumstances, it is more efficient for the target driver itself to transmit several transactions in parallel, rather than using the serialized transmit that the Framework provides for multi-packet operations.
• CMD1394_DISABLE_ADDR_INCREMENT - When a request is larger than the destination node's prevailing maximum payload size, it is split into multiple packets, each with may_payload size. This bit affects how the Framework builds the destination address of each of the split packets.
  Normally, when this option is not set, and after the initial packet is sent, the Framework automatically increments the destination address accordingly for the subsequent transmitted packets. However, if the target driver sets this bit, the Framework does not adjust the destination address of the subsequent packets and instead sends each subsequent packet to the same destination address as it did for the initial packet.
• CMD1394_BLOCKING - If set, the t1394_read(), t1394_write(), or t1394_lock() call will not return until the command completes. In this case, the completion_callback() field is ignored.
• CMD1394_OVERRIDE_SPEED - The target driver may select the transmit speed of the ensuing asynchronous request packet(s) by enabling the CMD1394_OVERRIDE_SPEED option. When this option is enabled, the 1394 Framework transmits the request packet(s) at the speed specified in the command's cmd_speed field.
  When this option is not enabled (the default), the 1394 Framework transmits the request at the maximum possible speed acheivable between the Solaris host and the destination device.

Packet Data

The asynchronous command structure is used for issuing a variety of requests. And each kind of request requires a different set of parameters. There are four kinds of requests as shown below:

quadlet requests

The structure is used for both write quadlet requests and read quadlet requests. A target driver issues write quadlet requests using t1394_write() and issues read quadlet requests using t1394_read().

/* Asynchronous Command (Data Quadlet) */
typedef struct cmd1394_quadlet {
        uint32_t                quadlet_data;
} cmd1394_quadlet_t;

For t1394_write(), the quadlet_data field contains the 4 bytes to be written. For t1394_read(), the quadlet_data contains the 4-byte read data from the requested address.

block requests

The following structure is used for read block and write block requests. A target driver issues write block requests using t1394_write and issues read block requests using t1394_read().

/* Asynchronous Command (Data Block) */
typedef struct cmd1394_block {
        mblk_t                  *data_block;
        size_t                  blk_length;
        size_t                  bytes_transferred;
        uint_t                  max_payload;
} cmd1394_block_t;

• data_block - The data buffer(s) for the command are in the form of a STREAMS mblk_t structure. For t1394_write(), the target driver supplies a data_block containing the data to be written. For t1394_read(), the target driver supplies a data_block for the Framework to use to fill in the requested read data.
  Non-STREAMS drivers obtain an mblk_t from allocb(9F) or esballoc(9F). See msgb(9S) for more information on the mblk_t structure.
• blk_length - The target driver provides the blk_length to specify the number of bytes in data_block that the Framework uses for the requested operation. In t1394_write, blk_length indicates the number of bytes to write. In t1394_read(), blk_length indicates the number of bytes to request in the read. Note that the blk_length can be less than the total number of bytes comprising the data_block buffer(s). If blk_length is greater than the number of bytes in data_block buffer(s), the Framework rejects the command with CMD1394_EMBLK_TOO_SMALL.
• bytes_transferred - The 1394 Software Framework fills in the bytes_transferred field upon command completion, indicating the total number of bytes read or written.
  If an error occurs and the operation does not complete successfully, bytes_transferred will be zero if no bytes were transferred. For requests which have been split into multiple packets, bytes_transferred reflects the number of bytes transferred successfully.
  This field is needed in the case of a block write or read in which only part of the data transmits successfully.

lock requests

The following structure is used for 32-bit lock requests:

/* Asynchronous Command (Lock Cmd - 32 bit) */
typedef struct cmd1394_lock32 {
        uint32_t                old_value;
        uint32_t                data_value;
        uint32_t                arg_value;
        uint_t                  num_retries;
        cmd1394_lock_type_t     lock_type;
} cmd1394_lock32_t;

The following structure is used for 64-bit lock requests:

/* Asynchronous Command (Lock Cmd - 64 bit) */
typedef struct cmd1394_lock64 {
        uint64_t                old_value;
        uint64_t                data_value;
        uint64_t                arg_value;
        uint_t                  num_retries;
        cmd1394_lock_type_t     lock_type;
} cmd1394_lock64_t;

The Framework supports a large number of lock operations, each of which uses the lock values somewhat differently. The value fields are described here in a general way.

• old_value - Typically the Framework overwrites old_value with the data actually read from the device. If needed, the target driver is responsible for independently tracking the original value it supplied in old_value.
• data_value - The target driver provides data_value which typically is the value used to modify the contents at the destination address.
• arg_value - The target driver provides arg_value which typically is used in a comparison with the value at the requested device's destination address.
• num_retries - The num_retries field directs the Framework to retry the current operation the specified number of times to acheive successful command completion.
  Certain operations offered by t1394_lock() are actually performed by a series of transactions including a lock transaction. For example, the Framework performs a CMD1394_LOCK_BIT_AND by issuing a read and then a lock (compare-swap) transaction. Occasionally, due to the time that passes between these requests, the compare-swap step may fail. If that happens, the Framework will attempt the transaction sequence num_retries times. This field is not used to specify the number of retry attempts in the event of a device ACK_BUSY. For that the Framework uses a fixed global retry count.
• lock_type - The target driver sets lock_type to one of the values from the cmd1394_lock_type_t enumerated type.

Error Codes

There are two contexts under which an error may be reported.

An error might occur when the command is initially issued due to a faulty parameter or another immediately detected error. For example, if t1394_write() is called with a NULL data_block message pointer for a block write, it will return an error status of DDI_FAILURE, and the command's cmd_result is set to CMD1394_ENULL_MBLK. If no error is found, the function return value is DDI_SUCCESS and the command's cmd_result reflects the current status of the transaction(s).

An error might also occur during the ensuing bus transaction(s). In this case, the Framework sets the command's cmd_result appropriately and the command is completed as described above.

Error Codes for Immediately Detected Failures

After a target driver gets DDI_FAILURE from t1394_read(), t1394_write(), or t1394_lock(), the command's cmd_result can be one of the following:

• CMD1394_ENULL_MBLK - The command requires a data buffer and cmd_u.b.data_block is NULL.
• CMD1394_EMBLK_TOO_SMALL - The command specifies a transaction data length that is larger than the data buffer in cmd_u.b.data_block.
• CMD1394_ESTALE_GENERATION - The CMD1394_OVERRIDE_ADDR option is specified, but the provided bus_generation is out of date.
• CMD1394_EDEVICE_REMOVED - The target device was removed from the 1394 bus.
• CMD1394_EINVALID_CONTEXT - The command's CMD_BLOCKING option is set, but the call has been issued for an interrupt context.
• CMD1394_EINVALID_COMMAND - The command type is not valid for the function or an invalid lock type was specified.
• CMD1394_ENO_ATREQ - Asynchronous request transmit has been temporarily disabled. This may be due to the CSR_STATE.dreq bit being cleared or due to a temporary lack of tlabels. This error indicates a transient condition.
• CMD1394_EFATAL_ERROR - The 1394 Framework has detected a fatal error using the 1394 hardware interface and can no longer process commands.
• CMD1394_EUNKNOWN_ERROR - An error was detected that cannot be represented by any other defined error code.

For non-blocking commands, the command cmd_result after a target driver gets DDI_SUCCESS can be CMD1394_NOSTATUS.

• CMD1394_NOSTATUS - There are no command parameter errors and the command transaction(s) are in progress. After calling an outgoing request function, if the target driver polls for status (as described above), this value may been seen under certain timing conditions.

Error Codes for Transactions

• CMD1394_SUCCESS - The command completed successfully; the request was issued and a corresponding non-error response was received.
• CMD1394_EDEVICE_BUSY - After multiple attempts to issue the request the device remained busy and did not accept and respond to the request.
• CMD1394_ERETRIED_EXCEEDED - The requested operation was retried num_retries times and failed. This error code is not used when the device is busy (i.e. returns ACK_BUSY). This error code is possible only with multiple step operations using t1394_lock(), which have failed after the specified number of retries.
• CMD1394_EDEVICE_ERROR - The 1394 Framework has detected an error on the remote device. For example, this error will be returned if the device has sent a response using an incorrect tcode.
• CMD1394_ETYPE_ERROR - The device rejected the read, write, or lock to the location specified by cmd_addr. This corresponds to the IEEE1394 resp_type_error response code.
• CMD1394_EDATA_ERROR - Only for requests with a block data payload, this error indicates that the device rejected the request due to problems receiving the data. This may indicate a hardware error. This corresponds to the IEEE1394 resp_data_error response code.
• CMD1394_EDEVICE_REMOVED - The device was removed from the bus and the request could not complete. This error will not be returned if using the CMD1394_OVERRIDE_ADDR option.
• CMD1394_EADDRESS_ERROR - The destination offset in the request was set to an address no accessible in the destination node. This corresponds to the IEEE1394 resp_address_error response code.
• CMD1394_ETIMEOUT - The device received the request and either did not respond within SPLIT_TIMEOUT (see IEEE 1394-1995) or it did not ACK the receipt of the request.
• CMD1394_ERSRC_CONFLICT - The request was received intact, but the resources required to handle the request were not available. The request may succeed if reissued later. This corresponds to the IEEE1394 resp_conflict_error response code.

Outgoing Asynchronous Requests (Part 2 of 2)... upcoming

OK, so that's a ton of detail for one post. And that's just to talk about how the command structure is used for outgoing requests. But I'm gonna stop here for now and finish this discussion of outgoing asynch requests with another post in a couple of days to talk about the interfaces themselves.

---

[1]It's important to note that it is possible, due to various timing conditions, for the Framework to call the completion_callback() routine before the interface call returns. So target drivers using completion_callback() need to handle this possible sequence of events.
[2]Why polling? Why when we have callbacks and blocking commands already? Well the primary reason is for a purpose not yet fulfilled... being able to do a crash dump to a 1394 disk. When the system is crashed and dumping to disk, interrupts are disabled... which makes both blocking and callbacks impossible. If you're actually reading this, and you want a project, take a look at what it'd take to enable polling throughout the entire 1394 Software Framework and to enable dump to 1394 disk.
[3]It's a really big no-no to issue blocking commands while operating in the interrupt context, e.g. in a callback, as this could cause a deadlock.
[4]These defines can be found in ieee1394.h
[5]A word of caution: A target driver should be careful when sending requests using CMD1394_OVERRIDE_ADDR. Sending requests to devices other that its own could adversely impact the performance or function of the destination node.
(2005-07-09 20:57:00.0) Permalink