venugopal iyer: venu @ sun

Crossbow is now available in OpenSolaris (build 105 upwards)!

The following text provides a high-level overview of some of the data structures that define the major crossbow components and how they tie together. The files mentioned in the text below can be viewed using OpenSolaris Source Browser. Design documents describing details can be found at Crossbow Documents.

At the topmost level is the mac_impl_t, which contains information that is specific to a device. There exists a mac_impl_t for every physical NIC as also for virtual NIC, aggregation and etherstub. The mac_impl_t for a device is created when the device registers with the MAC layer using mac_register(). The main fields in the mac_impl_t are:

• Device info (name, type, media etc.)
• Driver callbacks (start, stop etc.)
• Unicast address
• Notification callback functions.
• Receive and Transmit group information.
• MAC Classification Flow table
• Virtualization Readiness
• List of MAC clients
• Cached Resources (bandwidth limit, CPU bindings etc.)
• Statistics

Refer to usr/src/uts/common/sys/mac_impl.h for the complete definition. (See usr/src/uts/common/sys/mac_provider.h for information about Virtualization readiness - MAC_VIRT_NONE etc.)

The Receive and Transmit ring grouping is provided initially by the driver, and may be dynamically modified subsequently, if supported by the driver. A group - mac_group_t - contains one or more rings - mac_ring_t. See usr/src/uts/common/sys/mac_impl.h for the definition.

The MAC's Classification Flow table is used when classifying inbound packets (using the unicast address) in software. Non-unicast packets are always classified using this table.

A MAC client is associated with each link. The mac_impl_t contains a list of all the MAC clients that are created on the corresponding MAC. A MAC client is represented by mac_client_impl_t and contains the description of the MAC client and its associated resources. The main fields in the mac_client_impl_t include:

• Flow entry associated with the MAC client
• Underlying MAC for this client (if any).
• Receive callback function and args.
• Promiscuous list for this MAC client.
• Hybrid I/O specific information.
• List of VLANs on this MAC client.
• List of subflows configured on this MAC client.
• Statistics

Refer to usr/src/uts/common/sys/mac_client_impl.h for the complete definition.

The flow entry - flow_entry_t - contains information that describes the MAC client and resources associated with it. The main fields include:

• Flow descriptor
• Resources (bandwidth, CPU etc.) configured for this MAC client
• Receive side SRS and related information
• Transmit side SRS and related information
• Receive callback function.
• Statistics

Refer to usr/src/uts/common/sys/mac_flow_impl.h for the complete definition.

The flow descriptor is a generic structure that includes, for a MAC client, the unicast address and the VLAN ID. This is used when classifying incoming packets for a MAC client.

A VLAN id could be associated with a MAC client as a result of creating a VLAN using dladm create-vlan ... or dladm create-vnic -v .... A MAC client may have multiple VLANs associated with it in which case each VLAN will be represented by a flow_entry_t (this means all the VLANs share the same MAC unicast address, but a different VLAN id).

The subflow table contains classification information for any L3 and above flows created on the MAC client (using flowadm(1m)). Subflows are dicussed later in this text.

The resources (in flow_entry_t or mac_impl_t) essentially contains the following information:

• Bandwidth Limit
• Priority
• CPU bindings

These are specified via a mac_resource_props_t structure defined in usr/src/uts/common/sys/mac_flow.h.

SRS (Soft Ring Set) - mac_soft_ring_set_t - forms the heart of packet processing on the receive and transmit side. On the transmit side a MAC client has an SRS, while on the receive side it may have more than one SRS. The main fields for a mac_soft_ring_set_t are:

• SRS type
• SRS state
• Soft Ring list (if any)
• Drain function
• Worker and Poll thread.
• Receive side SRS.
• Transmit side SRS.

See usr/src/uts/common/sys/mac_soft_ring.h for the complete definition.

Transmit side SRS is responsible for accepting packets from the upper layer, applying policies such as bandwidth limit, and sending it to one of the transmit rings to be sent out (packets that are destined to MAC clients on the same MAC are just looped back). The main fields specific to transmit side SRS - mac_srs_tx_t are:

• SRS mode
• Trasmit function and args
• Flow control information
• Statistics

See usr/src/uts/common/sys/mac_soft_ring.h for the complete definition.

Receive side SRS is responsible for polling, if supported, applying policies such as bandwidth limit, and sending the packet to the upper layer. If the MAC client is hardware classified and is assigned a group with multiple receive rings, there will be an SRS for each of the receive ring in that group.The main fields specific to transmit side SRS - mac_srs_rx_t are:

• Receive function and args
• Polling information
• Statistics

See usr/src/uts/common/sys/mac_soft_ring.h for the complete definition.

If there is a bandwidth limit configured on a MAC client, then the SRSs (receive and transmit) use the resource information in the MAC client's flow_entry_t to enforce the limit. If there are multiple receive SRSs for a MAC client, they all share the same bandwidth structure - mac_bw_ctl_t - to enforce the bandwidth limit for that MAC client.

Both Receive and Transmit SRSs use soft rings - mac_soft_ring_t - to fan out to the upper layer or to one of the transmit rings of the underlying driver. The soft rings may be bound to CPUs /9for distributing processing among multiple processors) if there is a CPU binding specified for the MAC client. The main fields of the mac_soft_ring_t are:

• Callback function and args
• Flow control information
• Drain function
• Worker thread

See usr/src/uts/common/sys/mac_soft_ring.h for the complete definition.

Subflows are added using flowadm(1m). These are flows that have classification parameters that include L3 and above information (IP address, protocol, port etc.). When a subflow is added on a MAC client, a flow_entry_t is created and added to the MAC clients subflow table. The flow_entry_t is the same structure that is used to represent a MAC client, except that the flow descriptor field now contains additional information based on the classifying parameters (IP address, ports etc) for that subflow.

Given that the same flow_entry_t is used to represent subflows, SRS etc. apply equally to subflows. Subflows are currently always classified in the software, i.e once the packet is classified for a MAC client (either in the hardware or software), we use that MAC client's subflow table to determine the appropriate subflow, if one exists. Processing for the packet continues in the context of the subflow's SRS.