SteveJay's Weblog

All | FireWire | General | InfiniBand | PCI Express
« Previous month (May 2005) | Main | Next month (Jul 2005) »

20050710 Sunday July 10, 2005

The 1394 Software Framework - Outgoing Asynchronous Requests (Part 2 of 2)

OK... here's part 2 of my discussion on the Solaris 1394 Framework's interfaces for sending outgoing asynch requests. This continues my discussion of the Solaris 1394 Software Framework, and in this post I'm going to go over the functions used to allocate, free, and transmit asynch requests (read, write, or lock). In my previous entry, I described the details of the cmd1394_cmd_t structure that is used to encapsulate all the relevant information for an asynch command.




t1394_alloc_cmd()

/*
 * Function:    t1394_alloc_cmd()
 * Input(s):    t1394_hdl               The target "handle" returned by
 *                                          t1394_attach()
 *              flags                   The flags parameter is described below
 *
 * Output(s):   cmdp                    Pointer to the newly allocated command
 *
 * Description: t1394_alloc_cmd() allocates a command for use with the
 *              t1394_read(), t1394_write(), or t1394_lock() interfaces
 *              of the 1394 Software Framework.  By default, t1394_alloc_cmd()
 *              may sleep while allocating memory for the command structure.
 *              If this is undesirable, the target may set the
 *              T1394_ALLOC_CMD_NOSLEEP bit in the flags parameter.  Also,
 *              this call may fail because a target driver has already
 *              allocated MAX_NUMBER_ALLOC_CMDS commands.
 */
int
t1394_alloc_cmd(t1394_handle_t t1394_hdl, uint_t flags, cmd1394_cmd_t **cmdp)

Using this interface, a target driver allocates one command used to issue outgoing asynchronous requests. The Framework limits the total number of commands that a target driver may have outstanding. This limit is fixed and currently defaults to 256 commands.

Context:

Can be called from base kernel context or from interrupt context. If called from interrupt context, flags must be set to T1394_ALLOC_CMD_NOSLEEP.

Parameters:

Return Values:


t1394_free_cmd()

/*
 * Function:    t1394_free_cmd()
 * Input(s):    t1394_hdl               The target "handle" returned by
 *                                          t1394_attach()
 *              flags                   The flags parameter is unused (for now)
 *              cmdp                    Pointer to the command to be freed
 *
 * Output(s):   DDI_SUCCESS             Target successfully freed command
 *              DDI_FAILURE             Target failed to free command
 *
 * Description: t1394_free_cmd() attempts to free a command that has previously
 *              been allocated by the target driver.  It is possible for
 *              t1394_free_cmd() to fail because the command is currently
 *              in-use by the 1394 Software Framework.
 */
/* ARGSUSED */
int
t1394_free_cmd(t1394_handle_t t1394_hdl, uint_t flags, cmd1394_cmd_t **cmdp)

Using this interface, a target driver frees one cmd1394_cmd_t command.

Context:

Can be called from base kernel context or from interrupt context.

Parameters:

Return Values:


t1394_read()

/*
 * Function:    t1394_read()
 * Input(s):    t1394_hdl               The target "handle" returned by
 *                                          t1394_attach()
 *              cmd                     Pointer to the command to send
 *
 * Output(s):   DDI_SUCCESS             Target successful sent the command
 *              DDI_FAILURE             Target failed to send command
 *
 * Description: t1394_read() attempts to send an asynchronous read request
 *              onto the 1394 bus.
 */
int
t1394_read(t1394_handle_t t1394_hdl, cmd1394_cmd_t *cmd)

The t1394_read() command issues an asynchronous read quadlet request of read block request to the indicated address and places the response data into the provided cmd->cmd_u.b.data_block message buffer or cmd->cmd_u.q.quadlet_data

If the CMD1394_OVERRIDE_ADDR option is not set, the 1394 Software Framework fills in the appropriate high-order 16 bits of the cmd_addr field with the 10-bit bus number and 6-bit node ID for the attached device. If the CMD1394_OVERRIDE_ADDR option is set, the Framework uses the complete 64-bit cmd_addr field as provided.

The Framework splits the read request into multiple read requests if any of the following conditions are true.

  1. The specified data length exceeds the prevailing maximum payload size which is either the current maximum payload size or the overridden max_payload value.
  2. The specified data length exceeds the maximum packet size permitted by the current bus speed.

If the CMD1394_DISABLE_ADDR_INCREMENT option is not set, successive read requests are issued with the destination address advancing accordingly. If the CMD1394_DISABLE_ADDR_INCREMENT option is set, the Framework sends each read request to the same destination address.

Context:

Can be called from base kernel context or from interrupt context. If called from interrupt context, cmd_options can not be set to CMD1394_BLOCKING.

Parameters:

Return Values:

If the command succeeds (i.e. if cmd_result equals CMD1394_SUCCESS), the read data is in cmd->cmd_u.b.data_block or cmd->cmd_u.q.quadlet_data and bytes_transferred indicates the total number of bytes read. For block requests, the data block's b_wptr points to the byte following the last read byte.

Possible immediate errors (see 'Error Codes' in my previous blog entry):

Possible completion statuses (in cmd_result) are below. Note that if the operation fails, bytes_transferred indicates if part of the operation succeeded.


t1394_write()

/*
 * Function:    t1394_write()
 * Input(s):    t1394_hdl               The target "handle" returned by
 *                                          t1394_attach()
 *              cmd                     Pointer to the command to send
 *
 * Output(s):   DDI_SUCCESS             Target successful sent the command
 *              DDI_FAILURE             Target failed to send command
 *
 * Description: t1394_write() attempts to send an asynchronous write request
 *              onto the 1394 bus.
 */
int
t1394_write(t1394_handle_t t1394_hdl, cmd1394_cmd_t *cmd)

The t1394_write() command issues an asynchronous write quadlet request of write block request to the indicated address (based on cmd_type) and uses the provided cmd->cmd_u.b.data_block message buffer or cmd->cmd_u.q.quadlet_data field

If the CMD1394_OVERRIDE_ADDR option is not set, the 1394 Software Framework fills in the appropriate high-order 16 bits of the cmd_addr field with the 10-bit bus number and 6-bit node ID for the attached device. If the CMD1394_OVERRIDE_ADDR option is set, the Framework uses the complete 64-bit cmd_addr field as provided.

The Framework splits the write request into multiple write requests if any of the following conditions are true.

  1. The specified data length exceeds the prevailing maximum payload size which is either the current maximum payload size or the overridden max_payload value.
  2. The specified data length exceeds the maximum packet size permitted by the current bus speed.

If the CMD1394_DISABLE_ADDR_INCREMENT option is not set, successive write requests are issued with the destination address advancing accordingly. If the CMD1394_DISABLE_ADDR_INCREMENT option is set, the Framework sends each write request to the same destination address.

Context:

Can be called from base kernel context or from interrupt context. If called from interrupt context, cmd_options can not be set to CMD1394_BLOCKING.

Parameters:

Return Values:

If the command succeeds (i.e. if cmd_result equals CMD1394_SUCCESS), bytes_transferred indicates the total number of bytes written. For block writes, the 1394 Framework transmits the data between the data block's b_rptr and b_wptr and leaves both b_rptr and b_wptr unchanged.

Possible immediate errors (see 'Error Codes' in my previous blog entry):

Possible completion statuses (in cmd_result) are below. Note that if the operation fails, bytes_transferred indicates if part of the operation succeeded.


t1394_lock()

/*
 * Function:    t1394_lock()
 * Input(s):    t1394_hdl               The target "handle" returned by
 *                                          t1394_attach()
 *              cmd                     Pointer to the command to send
 *
 * Output(s):   DDI_SUCCESS             Target successful sent the command
 *              DDI_FAILURE             Target failed to send command
 *
 * Description: t1394_lock() attempts to send an asynchronous lock request
 *              onto the 1394 bus.
 */
int
t1394_lock(t1394_handle_t t1394_hdl, cmd1394_cmd_t *cmd)

There are several lock operations provided by the 1394 Software Framework (above and beyond the basic lock operations provided by IEEE 1394). These are described below (with pseudo-code to describe their operation).

For all lock operations, if the CMD1394_OVERRIDE_ADDR option is not set, the 1394 Software Framework fills in the appropriate high-order 16 bits of the cmd_addr field with the 10-bit bus number and 6-bit node ID for the attached device. If the CMD1394_OVERRIDE_ADDR option is set, the Framework uses the complete 64-bit cmd_addr field as provided.

Context:

Can be called from base kernel context or from interrupt context. If called from interrupt context, cmd_options can not be set to CMD1394_BLOCKING.

Parameters:

Return Values:

Possible immediate errors (see 'Error Codes' in my previous blog entry):

Possible completion statuses (in cmd_result) are below.


(2005-07-10 13:15:00.0) Permalink Comments [1]

20050709 Saturday July 09, 2005

The 1394 Software Framework - Outgoing Asynchronous Requests (Part 1 of 2)

Continuing my discussion of the Solaris 1394 Software Framework, in this post I'm going to go into some detail on the methods by which a 1394 target driver can transmit outgoing asynchronous requests and receive the corresponding responses.

Command Allocation/Freeing

Handling of all asynchronous packets is acheived via commands. A command is a data structure which contains fields for the necessary packet components as well as command options and other fields used to specify the desired command processing.

To acquire a command structure for transmitting asynchronous requests and receiving responses, the 1394 target driver calls t1394_alloc_cmd(). This routine allocates and returns a pointer to a cmd1394_cmd_t structure (which I'll say more about below.) Note: This same command structure is also used for handling incoming asynchronous requests, although some fields are used differently.

A 1394 target driver can re-use a command after it has completed (see below), but it should not reissue or alter any fields for any command that is still pending. And when a target driver no longer needs a command, it will call t1394_free_cmd() to release the command structure back to the Framework. Note: 1394 target drivers are responsible for freeing all of their allocated commands before detaching (or the detach will not be allowed - see my previous blog entry.)

Command Action

After initializing the relevant parameters for the outgoing request, the 1394 target driver calls the appropriate asynchronous command routine; either t1394_write(), t1394_read(), or t1394_lock().

The command itself often takes some time to complete, since a packet must be sent over the bus to the destination node and the 1394 Framework must await the response. Therefore, the Framework gives the target driver the option to block or not block pending command completion (more on this below).

Once the command is "handed off" to the 1394 Software Framework, the target driver should not re-use or modify the same allocated command until the target driver can determine that the requested action has fully completed.

A command is "completed" when the bus transaction(s) used to perform the command have finished. The command's cmd_result field indicates either success (CMD1394_CMDSUCCESS) or failure, where a failure is indicated by an error code (See 'Error Codes' below for details).

Command Completion

Target drivers have three options for determining command completion status:

Bus Reset and Command Handling

The IEEE 1394 bus is reset when devices are added to or removed from the bus (or for a variety of other reasons). When this happens, all devices on the bus are re-enumerated and can possibly be assigned different bus addresses (referred to as "Node IDs") from the ones they each had prior to the bus reset.

From the IEEE 1394 protocol perspective, when a bus reset occurs all the pending and in-progress command requests are canceled. Target drivers have two options with respect to processing of any outstanding commands.

Asynch Command Structure

/* cmd1394_cmd: cmd1394 - common command type */
typedef struct cmd1394_cmd
{
        int                     cmd_version;
        volatile int            cmd_result;
        cmd1394_flags_t         cmd_options;
        cmd1394_cmd_type_t      cmd_type;
        void                    (*completion_callback)(struct cmd1394_cmd *);
        opaque_t                cmd_callback_arg;
        uint64_t                cmd_addr;
        uint_t                  cmd_speed;
        uint_t                  bus_generation;
        uint_t                  nodeID;
        uint_t                  broadcast;
        union {
                cmd1394_quadlet_t       q;
                cmd1394_block_t         b;
                cmd1394_lock32_t        l32;
                cmd1394_lock64_t        l64;
        } cmd_u;
} cmd1394_cmd_t;

Command Types

/*
 * cmd1394_cmd.cmd_type
 *    Used to select/indicate the request packet type
 */
typedef enum {
        CMD1394_ASYNCH_RD_QUAD  = 0,
        CMD1394_ASYNCH_WR_QUAD  = 1,
        CMD1394_ASYNCH_RD_BLOCK = 2,
        CMD1394_ASYNCH_WR_BLOCK = 3,
        CMD1394_ASYNCH_LOCK_32  = 4,
        CMD1394_ASYNCH_LOCK_64  = 5
} cmd1394_cmd_type_t;

Command Options

A target driver uses these options to tailor the Framework's command processing behavior. 
/*
 * cmd1394_cmd.flags
 *    Used to select the request's behavior, including
 *    how the destination address is determined, how
 *    a large request will be broken into smaller requests,
 *    whether the command should be resent after a
 *    bus reset has happened, etc.
 */
typedef enum {
        CMD1394_CANCEL_ON_BUS_RESET     = (1 << 0),
        CMD1394_OVERRIDE_ADDR           = (1 << 1),
        CMD1394_OVERRIDE_MAX_PAYLOAD    = (1 << 2),
        CMD1394_DISABLE_ADDR_INCREMENT  = (1 << 3),
        CMD1394_BLOCKING                = (1 << 4),
        CMD1394_OVERRIDE_SPEED          = (1 << 5)
} cmd1394_flags_t;

Packet Data

The asynchronous command structure is used for issuing a variety of requests. And each kind of request requires a different set of parameters. There are four kinds of requests as shown below:

quadlet requests

The structure is used for both write quadlet requests and read quadlet requests. A target driver issues write quadlet requests using t1394_write() and issues read quadlet requests using t1394_read(). 
/* Asynchronous Command (Data Quadlet) */
typedef struct cmd1394_quadlet {
        uint32_t                quadlet_data;
} cmd1394_quadlet_t;
For t1394_write(), the quadlet_data field contains the 4 bytes to be written. For t1394_read(), the quadlet_data contains the 4-byte read data from the requested address.

block requests

The following structure is used for read block and write block requests. A target driver issues write block requests using t1394_write and issues read block requests using t1394_read(). 
/* Asynchronous Command (Data Block) */
typedef struct cmd1394_block {
        mblk_t                  *data_block;
        size_t                  blk_length;
        size_t                  bytes_transferred;
        uint_t                  max_payload;
} cmd1394_block_t;

lock requests

The following structure is used for 32-bit lock requests: 
/* Asynchronous Command (Lock Cmd - 32 bit) */
typedef struct cmd1394_lock32 {
        uint32_t                old_value;
        uint32_t                data_value;
        uint32_t                arg_value;
        uint_t                  num_retries;
        cmd1394_lock_type_t     lock_type;
} cmd1394_lock32_t;
The following structure is used for 64-bit lock requests: 
/* Asynchronous Command (Lock Cmd - 64 bit) */
typedef struct cmd1394_lock64 {
        uint64_t                old_value;
        uint64_t                data_value;
        uint64_t                arg_value;
        uint_t                  num_retries;
        cmd1394_lock_type_t     lock_type;
} cmd1394_lock64_t;
The Framework supports a large number of lock operations, each of which uses the lock values somewhat differently. The value fields are described here in a general way.

Error Codes

There are two contexts under which an error may be reported.

An error might occur when the command is initially issued due to a faulty parameter or another immediately detected error. For example, if t1394_write() is called with a NULL data_block message pointer for a block write, it will return an error status of DDI_FAILURE, and the command's cmd_result is set to CMD1394_ENULL_MBLK. If no error is found, the function return value is DDI_SUCCESS and the command's cmd_result reflects the current status of the transaction(s).

An error might also occur during the ensuing bus transaction(s). In this case, the Framework sets the command's cmd_result appropriately and the command is completed as described above.

Error Codes for Immediately Detected Failures

After a target driver gets DDI_FAILURE from t1394_read(), t1394_write(), or t1394_lock(), the command's cmd_result can be one of the following:

For non-blocking commands, the command cmd_result after a target driver gets DDI_SUCCESS can be CMD1394_NOSTATUS.

Error Codes for Transactions

Outgoing Asynchronous Requests (Part 2 of 2)... upcoming

OK, so that's a ton of detail for one post. And that's just to talk about how the command structure is used for outgoing requests. But I'm gonna stop here for now and finish this discussion of outgoing asynch requests with another post in a couple of days to talk about the interfaces themselves.

[1]It's important to note that it is possible, due to various timing conditions, for the Framework to call the completion_callback() routine before the interface call returns. So target drivers using completion_callback() need to handle this possible sequence of events.
[2]Why polling? Why when we have callbacks and blocking commands already? Well the primary reason is for a purpose not yet fulfilled... being able to do a crash dump to a 1394 disk. When the system is crashed and dumping to disk, interrupts are disabled... which makes both blocking and callbacks impossible. If you're actually reading this, and you want a project, take a look at what it'd take to enable polling throughout the entire 1394 Software Framework and to enable dump to 1394 disk.
[3]It's a really big no-no to issue blocking commands while operating in the interrupt context, e.g. in a callback, as this could cause a deadlock.
[4]These defines can be found in ieee1394.h
[5]A word of caution: A target driver should be careful when sending requests using CMD1394_OVERRIDE_ADDR. Sending requests to devices other that its own could adversely impact the performance or function of the destination node.
(2005-07-09 20:57:00.0) Permalink