Solaris iSCSI Initiator and MPxIO

We frequently receive the question about which iSCSI devices are supported with the Solaris iSCSI initiator and MPxIO. Unfortnately, The iSCSI Multipathing BluePrint is out of date with what device support is available in Solaris 10 Update 2. We are working on getting an updated version published. The below information will hopefully give a rough update on MPxIO support with the s10u2. This was clipped from an email response I sent internally.

---

If you can answer the following questions I can help tell you if the Solaris initiator will support multipathing with your iSCSI device.

Q) Do you know if your iSCSI device acts as a symetric (active/active) SCSI device with its iSCSI portals?

Assuming you said yes. The device is most likely supported with the Solaris iSCSI initiator under MPxIO. Only a few non symetric devices are supported by MPxIO.

Q) The next question you have to determine is how the target describes its iSCSI portals. Depending on this you have to configure the initiator differently. Unfornately, The iSCSI specification is a little too open for the initiator to auto discover and automatically configure all array types.

Types:

1) Multiple iSCSI portals are exposed with the same iSCSI node name and each iSCSI portal is in a different iSCSI target portal group. (ex. NetApp, EMC, ... can be configured this way.)

In this case the Solaris iSCSI initiator will automatically plumb all the IO paths. (See MPxIO Setup below.)

2) Multiple iSCSI portals are exposed with different iSCSI node names. (ex. QLogic, HP, ... can be configured this way.)

Again assuming you setup your discovery correctly this will also be automatic. (See MPxIO Setup below.)

3) Multiple iSCSI portals are exposed with the same iSCSI node name and all the portals are accessed via a single virtal IP address (Login Redirection). (ex. EqualLogic, Intransa, Left Hand, Cisco, ... can be configured this way.)

In this cause since the array is virtualizing the hardware the initiator can't guess how many paths to create. So the user has to specify the number of paths with iscsiadm modify initiator-node -c [1-4, # of paths] or iscsiadm modify target-param -c [1-4, # of paths]. The choice of initiator-node or target-param depends on if you want the setting to effect all targets or a specific target. (See MPxIO Setup below)

4) Multiple iSCSI portal are exposed with the same iSCSI node name and all portals are in the same iSCSI target portal group tag. (ex. Netapp, EMC, Sun 5XXX, ... can be configured this way.)

In Solaris 10 Update 3 (and I'm trying to get it out in a patch earlier. You can again use the same setups in #3 to increase the number of paths. This change is already available in solaris nevada and will be in solaris 10 update 3 build 3. (See MPxIO Setup below.)

--- MPxIO Setup ---



Once you have configured the solaris iSCSI initiator you still may have a little work to perform. MPxIO also has to understand it supports the device. This can happen in 3 ways.

1) It is a Sun device and has been hard coded in the drivers internal support tables. (ex. Sun, some EMC and HDS devices (not the ASM 500))

2) The device supports the T-10 TPGS standard. So MPxIO can automatically discovery its supported. (ex. NetApp, EqualLogic, and a few others are starting to implement this at request of Sun.)

--- If all else fails, and it normally does for non-Sun devices ---

3) You can add a /kernel/drv/scsi_vhci.conf entry to claim the device can be supported as a symmetric device. Below is an example.

device-type-scsi-options-list = "SUN SENA", "symmetric-option";

symmetric-option = 0x1000000;

Assuming your device is configured in one of those methods you should see long dev names like the following...

/dev/rdsk/c2t0690A0180071248C2F4AB40332F08F49d0s2

Posted at 07:30AM Jul 11, 2006 by dweibel in General  | 

Two new iSCSI patches for s10+ out...

Sun recently released two new iSCSI patches for the Solaris iSCSI software initiator. One for Sparcv9 and the other for x86/amd64.

119090-20

119091-20

These patches will work on s10 and forward. This patch includes all the Solaris 10 Update 2 features, along with 3 intermittent bug fixes, and one small feature enhancement. That feature enhancement is a way to disable MPxIO for the Solaris iSCSI initiator in the same way possible with fp.conf.

Below is a clip from the iscsi.conf file. While I'm not happy with its double negative wording, disable=no, it does match the exact wording used by fp.conf to be consistent. To disable mpxio with the solaris initiator you will want to change the iscsi.conf entry mpxio-disable="no"; to mpxio-disable="yes";

#
# I/O multipathing feature (MPxIO) can be enabled or disabled using
# mpxio-disable property. Setting mpxio-disable="no" will activate
# I/O multipathing; setting mpxio-disable="yes" disables the feature.
#
# Global mpxio-disable property:
#
# To globally enable MPxIO on all iscsi ports set:
# mpxio-disable="no";
#
# To globally disable MPxIO on all iscsi ports set:
# mpxio-disable="yes";
#
mpxio-disable="no";

Posted at 12:18PM Jul 10, 2006 by dweibel in General  | 

iSCSI device reporting lots of iscsi sessions logging in and out

I just finished up reviewing a customer problem and I thought I would post some information about it to save others some time. In conclusion the problem was due to improperly removing iSCSI devices from the system. When removing an iSCSI device for good from a Solaris system. Its a good idea to issue devfsadm -C to clean up the old /dev paths and also update the MPxIO device cache (/etc/devices/mdi_scsi_vhci_cache).

The customer reported that in their iSCSI device log they were seeing frequent repeating iSCSI sessions logging in and out for no known reason. The Solaris messages log showed no iscsi related information to explain these device messages.

First, We quickly confirmed that the iSCSI initiator was causing sessions to login and out of the device with dtrace. To do this I used the below script.

#!/usr/sbin/dtrace -s
sdt:iscsi:iscsi_conn_state_machine:event
{
  printf("%p %s %s", arg0, (string)arg1, (string)arg2);
}

This script logs connection state changes within the iscsi initiator. If you want to better understand the initiators connection state machine you can either refer to the opensolaris code or the iSCSI RFC. The Solaris iSCSI initiator's connection state machine matches the iSCSI RFCs suggested connection state machine.

The output from the script looked as follows.

CPU     ID                    FUNCTION:NAME
  2   2303   iscsi_conn_state_machine:event 300246927c0 free T1
  2   2303   iscsi_conn_state_machine:event 300246927c0 in_login T5
  0   2303   iscsi_conn_state_machine:event 300246927c0 logged_in T9
  0   2303   iscsi_conn_state_machine:event 300246927c0 in_logout T17
  0   2303   iscsi_conn_state_machine:event 3002467f000 free T1
...

Since we could see activity in the trace this confirms the Solaris initiator is logging in and out of the target iSCSI device. Next, We needed to determine the reason for the login requests. The Solaris iSCSI initiator logs into devices under two conditions. The user wants to do IO to a device or the Solaris iSCSI initiator is trying to determine what devices are accessible to it. The later case can again occur under two conditions; The user has changed discovery information or the Solaris kernel is requesting the iSCSI initiator to update its discovery information, via a HBA tran_bus_config() request. Since the user was not performing management operations and their were already running IO something likely is kicking tran_bus_config(). To isolate the problem I ran the following dtrace.

#!/usr/sbin/dtrace -s
fbt:iscsi:iscsid_poke_discovery:entry
{
        stack();
}

The iSCSI initiators tran_bus_config() entry point is iscsi_tran_bus_config(). It then quickly branches out into two types of config requests. CONFIG_ALL in which Solaris wants the initiator to discover and login to all discovered devices. A CONFIG_ALL occurs when you issue devfsadm. The second type is CONFIG_ONE in which the kernel is requesting a specific device be discovered and logged in. When a CONFIG_ONE occurs the iSCSI initiator looks at all its current iSCSI sessions to see if the requested session is already discovered and logged in. If not discovered then it calls iscsid_poke_discovery(). This cause the initiator to query all enabled discovery methods. In this case SendTargets discovery is enabled to poke discovery would cause a iSCSI login/out along with a SendTargets query of the device. This seems to match the login pattern reported by the device.

The output from the script is show below...

...
  2  33609      iscsid_poke_discovery:entry poke
              iscsi`iscsid_config_one+0x70
              iscsi`iscsi_tran_bus_config+0x144
              genunix`ndi_devi_config_one+0x9c
              genunix`bus_config_one_phci_child+0x38
              genunix`config_client_paths+0xbcI wease
              genunix`mdi_vhci_bus_config+0x114
              scsi_vhci`vhci_scsi_bus_config+0x40
              genunix`ndi_devi_config_one+0x9c
              genunix`resolve_pathname+0x158
              genunix`e_ddi_hold_devi_by_path+0x10
              genunix`e_devid_cache_to_devt_list+0x2a8
              genunix`ddi_lyr_devid_to_devlist+0x10
              md`md_resolve_bydevid+0xe4
              md_raid`raid_check_cols+0x68
              md_raid`raid_iosetup+0x12c
              md_raid`md_raid_strategy+0x208
              md`mdstrategy+0xd4
              ufs`lufs_write_strategy+0x11c
              ufs`ufs_putapage+0x308
              ufs`ufs_putpages+0x2a4
...

Since a stack is reported this confirms that a tran_bus_config() is occuring. We can also see the request was for a CONFIG_ONE and continues on to iscsid_poke_discovery(). This means the kernel has requested iSCSI to discover and login to a specific device. Although that device is not found so we have to update our discovery leading to a iSCSI login/out on the target.

This still leaves the question why the kernel is requesting discovery of a device we don't know about. Looking at the rest of the stack we can see the MPxIO code (vhci_scsi_bus_config()) is involved so maybe this is an old device path still in its cache.

If we add the below trace clip to our last script we can now collect the device name that is not found.

fbt:iscsi:iscsid_config_one:entry
{
        printf("%s %d", (string)arg1, arg2);
}

Once that device name is collected we can run strings on the MPxIO device cache (/etc/devices/mdi_scsi_vhci_cache) to see if this device is in the cache. Which it was explaining why the kernel was asking us about it.

To resolve the problem the customer needed to run devfsadm -C to cleanup the cache of old stale cache data. After that was done the repeating iSCSI login/out messages stopped appearing in the iSCSI target device's logs.

In conclusion its recommended that upon removing iSCSI logical units form a host that the user also issue devfsadm -C to clean up addition Solaris cache information.

Posted at 02:24PM Jun 27, 2006 by dweibel in General  | 

Solaris iSCSI Target released

Last week on OpenSolaris.Org Sun released a Solaris iSCSI target. This is the same target that 550MB/s performance numbers were leaked about half a year ago on the following blog.

Posted at 08:06AM May 15, 2006 by dweibel in General  | 

Sprint SNW 2006

Next week is the Spring 2006 Storage Networking World show. Sun will again be presenting the iSCSI initiator at the show. The difference this time around is I will be attending the Hands-On lab instead of marketing. The goal is to try and collect some direct customer to development feedback on the Solaris iSCSI initiator and Sun's upcoming iSCSI boot solution. If your at the show come check out Sun's iSCSI boot solution or just say hello.

In the IP Hands-On Labs Sun will be presenting the following topics.

• Basic provisioning with the Solaris iSCSI initiator
• Basic provisioning with the Sun StorageTek NAS (iSCSI) Target
• Disaster planning using iSCSI and ZFS
• iSCSI Hybrid Boot

In addition there is an slight chance you might get to see the Solaris iSCSI target mode support that has been discussed over at the OpenSolaris.org storage forums.

Posted at 06:42AM Mar 28, 2006 by dweibel in General  |  Comments[2]

Solaris Express - Updated manpages

The latest Solaris Express contains supports for device logout, iSNS discovery, and multiple sessions per target support. The problem is we didn't get our manpage additions processed in time. Until we get the official additions included you can reference the linked unofficial manpage.

Posted at 05:28PM Feb 20, 2006 by dweibel in General  | 

Solaris Ready : EqualLogic

Clip from [here]

SUN Solaris 10 Native iSCSI Software Initiator



With the January 2006 Solaris 10 Update, Sun is now shipping their long-awaited, native iSCSI software initiator for Solaris X86/X64 and SPARC platforms for production use. With its inclusion in the January 2006 Solaris 10 Update, Sun's iSCSI software initiator is a fully supported feature of Solaris 10. EqualLogic has fully qualified the Sun native initiator with the PS Series storage array family.

System requirements include:

• Solaris X86/X64 and SPARC platforms
  
  
• Solaris 10 January 2006 Update
  
  
• EqualLogic PS Series platforms supporting FW Revision 2.2.3 or higher

Click here for more information on the technology partnership between EqualLogic and Sun Microsystems.

With the qualification of VERITAS Storage Foundation, VERITAS Cluster Server, as well as Sun's native iSCSI software initiator for Solaris 10, EqualLogic provides customers with the most robust offering of iSCSI solutions available for Solaris platforms. These offerings extend EqualLogic's industry leading iSCSI support in Solaris host environments.

Posted at 12:14PM Feb 09, 2006 by dweibel in General  | 

iSCSI officially released with S10U1 (old news)

This is a little bit of old news. Solaris iSCSI support was officially been released for all platforms via Solaris 10 Update 1 (dec2005/jan2006). The one gotcha to watch out for is to make sure you are looking at the correct documentation. There are minor differences in setup / configuration procedures between Solaris Express (Nevada) and Solaris 10.

• Solaris 10 - Solaris Administrators Guide
• Solaris Express - Solaris Administrators Guide

Posted at 08:21AM Jan 13, 2006 by dweibel in General  | 

Connectathon... (bad news)

Bad news is surfacing about the iSCSI connect-a-thon within Sun. The individuals that were going to run the iSCSI plug fest have been moved to a new projects inside of Sun. With this move they have dropped supporting the iSCSI plug fest. I'm still trying to determine what this means inside of Sun. It currently looks like the plug-fest is dead for this year. I will see if I can determine more details on this topic and post them later.

Posted at 08:06AM Jan 13, 2006 by dweibel in General  | 

Connectathon Round the Corner

Another engineer inside of Sun just reminded me that Connect-a-thon is just around the corner. We encourage all those iSCSI / NFS array vendors and other OS platform vendors to attend the 2006 connectathon. Sun will have our latest iSCSI products and features at the plugfest for interop testing. Things I know that will be available are iSNS client support, more methods of multipathing by creating mulitple iSCSI sessions, enhanced logout support, IMA, and more. All those features have already been putback to Solaris Nevada and will be available via Solaris Express in December. Count on more being ready by connectathon 2006!

For details go to http://www.connectathon.org/

Posted at 05:00PM Nov 29, 2005 by dweibel in General  |  Comments[0]

iSCSI and persistent tape binding?

There has been an outstanding complaint from customers around Solaris and Fiber Channel devices for a number of years. It looks like we have has created a short term workaround for this problem.

http://blogs.sun.com/roller/page/hbainsights?entry=persistent_bind_solution_creating_consistent

This same solution will work for iSCSI tape devices.

Posted at 08:15AM Nov 22, 2005 by dweibel in General  |  Comments[0]

Target ISID Problems

We have recently found a large number of the targets in the market space are incorrectly handling the iSCSI ISID. The ISID is the initiator part of the session identifier. An I_T_Nexus between an iSCSI initiator and target is created by the combination of the following parts.

iSCSI Initiator Name + ',i,'+ ISID, iSCSI Target Name + ',t,'+ Portal Group Tag (iSCSI RFC Section 2.1)

The problem we are seeing with many targets is they are ignoring the ISID in this I_T_Nexus. When a second session is connecting to the target with the same initiator-name and a different ISID some targets are treating this second session the same as the first. This means the first session is implicitly disconnected and the second session is treated as the first reconnecting (Connection Reinstatement iSCSI RFC Section 5.3.4).

Users with these misbehaving targets can encounter this problem with the Microsoft iSCSI initiator or with Solaris 10 Update 2 initiator (Solaris Express 12/05). As part of Solaris 10 Update 2 we have added some new abilities for multipathing that will allow the creation of multiple Sessions to the same iSCSI Target Name/Portal Group Tag. I will present more details on this feature in the future.

In the mean time I recommend target vendors correct this problem with one of two approaches.

• If your array doesn't have the resources to handle more that one session from a specific initiator-name. Then on login if the ISID is different than the original ISID return the following login failure. Status Class == 0x03 and Status Detail == 0x02, "Target has incufficient session, connection, or other resources."
• The other option is to properly record and support the ISID field so multiple sessions for the same initiator-name are supported.

Fixing this issue now will help your customers avoid problems in the future. If your an array vendor and have more questions on this problem email them to iscsi-support-ext at sun dot com.

Posted at 08:12AM Nov 22, 2005 by dweibel in General  |  Comments[0]

Solaris 10 Hardware 2

We keep getting questions from vendors and customers when the Solaris iSCSI initiator will be released and supported for Solaris 10. It used to be alot more complicated to answer these questions. Although the answers are getting alot simplier.

The Solaris iSCSI initiator is already supported on Solaris 10 via Solaris 10 Hardware Release 2. Get the details from the following links.

http://www.sun.com/software/solaris/get_hardware_2.jsp

http://www.sun.com/software/solaris/releases.xml

Posted at 07:47AM Nov 22, 2005 by dweibel in General  |  Comments[1]

Many Thanks!

I just wanted to say thanks to those who attended Sun's SNW iSCSI demo and provided feedback. Much of that feedback made it directly to our development team. If you were unable to attend the SNW iSCSI demo and would like to provide feedback please send your comments to iscsi-support-ext at sun dot com.

Posted at 12:00AM Nov 02, 2005 by dweibel in General  |  Comments[1]

Storage Networking World - Fall 2005

If your in Orlando at SNW this week. Come take a peek at Sun's
Solaris 10 iSCSI initiator at the 2005 Fall Storage Networking World!

Storage Networking World
October 24 - 27, 2005
JW Marriott Grande Lakes Resort
Orlando, FL
http://www.snwusa.com/

Posted at 04:09PM Oct 21, 2005 by dweibel in General  |  Comments[0]