The 2009.Q3 software update provides a new iSCSI software stack via the COMSTAR framework. COMSTAR is a Solaris framework which centralizes and simplifies the deployment of SCSI targets. COMSTAR supports many different SCSI protocols, including iSCSI, fibre channel, iSER, with support for more protocols under development. In 2009.Q3, the COMSTAR iSCSI port provider has replaced the older iSCSI target daemon. This new COMSTAR iSCSI provider replaces the entire iSCSI stack, from the management of iSCSI targets and initiators down to the interactions with the underlying ZFS volume. COMSTAR provides improved performance, as well as a more flexible management model. In this blog entry, I'll explain the new management model as well as talk about the advantages of using this new framework.

With the iSCSI target daemon in the 2009.Q2 update, every LUN was advertised within its own iSCSI target. As a result, initiators could hit built-in limits which prevent discovery of all necessary targets. Moreover, managing these targets did not scale well as the number of LUNs on the system increased. Access to a LUN was controlled by that LUN's ACL, a list of initiators which were able to access that LUN.

The administrative model used with the COMSTAR framework has some key differences and advantages. This model breaks this one-to-one mapping of LUN to target, and instead allows an administrator to advertise multiple LUNs behind a single target. Each LUN participates in a target group, a set of iSCSI targets which all advertise that LUN. In addition, each LUN has its own initiator group, a list of initiators not unlike the previous software's initiator ACL. This group defines which initiators may access LUNs bound to that initiator group, as well as the CHAP parameters to expect if CHAP authentication is enabled.

Target and initiator groups define sets of targets and initiators which are associated with LUNs. Besides any user-created target and initiator groups, LUNs may be associated with the default target and initiator group. These default groups contain all targets and all initiators, respectively. While using the default target and initiator group can be useful for evaluation purposes, their use is discouraged since their use may expose the LUN to unwanted hosts.

In addition to the flexible mapping of LUNs to targets, the COMSTAR iSCSI stack also allows targets to be bound to network interfaces:

This binding allows fine separation of iSCSI traffic¹ from other data protocols. In addition, iSCSI traffic can be routed over the fastest interfaces, leaving other slow interfaces available management traffic.

Best Practices

When upgrading to 2009.Q3, you'll need to rebuild your iSCSI configuration in this new model. With that reconfiguration in mind, here are some best practices for deploying COMSTAR iSCSI. Keep in mind that IP and SCSI have similar abstractions, and you may find yourself able to solve a problem at the IP level (e.g., creating an LACP aggregation and binding a target to that aggregation's interface) or the SCSI level (e.g., creating a target for each of several interfaces and adding those targets to the same target group).

• Use a single iSCSI target per target group. As the example above indicates, you may also combine targets into a single target group, but I've found it simplest to create a LACP aggregation for a single target.
• Don't use the default initiator group -- that group grants every initiator access to a LUN. A renegade or misconfigured initiator may discover an incorrect LUN; explicitly using an initiator group prevents any damage.
• Avoid using the default target group beyond evaluation.
• If you know a LUN must have LU number 0, create that LUN first. LUNs which are given automatically-assigned LU numbers can be given number zero.
• Don't enable the write cache setting on a LUN unless you completely understand the ramifications of doing so. Consult your application's documentation for information about using this setting.

Upgrading to 2009.Q3

Once you've applied the 2009.Q3 software update, any iSCSI configuration from 2009.Q2 and earlier is set aside and the SAN environment must be rebuilt. All LUNs are still intact and available; however, you'll need to define new iSCSI targets and (optionally) initiators to access those LUNs. Here's a quick and dirty set of steps to get up and running with 2009.Q3:

1. Create one or more iSCSI targets bound to the proper interfaces. This configuration is available in the new Configuration > SAN section.
2. Assign the iSCSI targets you created in the first step to a target group.
3. For each LUN, change its target group to the group created above.
4. (Optional) Create initiator definitions, assign those initiators to initiators groups, then bind LUNs to those initiator groups.
5. (Optional) Configure any CHAP or RADIUS authentication.
6. Once you've validated with the configuration, apply the COMSTAR deferred update to allow complete PGR support.

Additional Information

• Roch explains the performance advantages of the COMSTAR stack.
• COMSTAR project on OpenSolaris.
• Consult the documenation bundled with the 2009.Q3 update for details on deploying iSCSI LUNs with COMSTAR.

---

[1] In some circumstances, traffic to or from and iSCSI target on one interface may travel on a different interface. With certain network configurations (like multiple default routes on the system), the routing configuration will dictate that traffic must use certain interfaces which conflict with the iSCSI configuration.