Request: iSCSI Target Vendors - Solaris iSNS client support

Sun is working hard to implement iSNS client support for the Solaris iSCSI initiator in the near future. In preparation for this support our team has spent allot of time testing with different vendors hardware. Our development team has an interop lab containing hardware from most major iSCSI vendors. So this request is more for some of the smaller iSCSI vendors in the market space.

We have seen a couple issues with vendor implementations around iSNS. Those vendors have been notified and are working to resolve any issues with their devices. Again this request is really targeted to the vendors that we don't have hardware or time to test. For you we have a couple easy checks you can perform to verify your devices are going to work with the Solaris initiator's iSNS support.

Step 1.

Ensure your target can properly register with Microsoft's 3.0 iSNS server. (Yes, I said Microsoft. They have a good reference implementation of an iSNS server that is available for free, Today.)

Step 2.

When registering with the iSNS server make sure you are properly representing your portal groups. Specifically the PG Tag. This iSNS PG Tag should represent the same value received during an iSCSI full feature login for the TargetPortalGroupTag. If not our initiator will not trust the data you are returning and will fail the login.

We have seen two gotchas in this area with some vendor implementations

Gotcha 1.

If you don't register a PG Tag you are assigned a default PG Tag of 1 (RFC 4171 Section 6.5.4). In this case you must return a TargetPortalGroupTag of 1 during the iSCSI full feature login.

Gotcha 2.

Be careful in the ordering of your keys in your iSNS registration. If ordered wrong your PG Tag may be ignored by the iSNS server. As an example if you think you register PG Tag 5. I recommend you actually snoop/ethereal a trace of the value the iSNS server is returning to the initiator. It may have ignored the incorrect ordering of your keys and you also will gain the default PGTag of 1 causing the initiator to fail the login.

If you verify these two checks with you implementation you should be safe. In addition we hope to make our iSNS client support available via Solaris Express in the next couple months and recommend you test with our initiator.

Posted at 11:20AM Oct 14, 2005 by dweibel in General  |  Comments[0]

Solaris Express packages on S10

A number of people in the field have been applying Solaris Express (nevada) packages against Solaris 10 to get iSCSI. I don't think this is officially supported by Sun? I'm still trying to figure that out. Most people doing this are hitting the following problem on sparc.

WARNING: iscsi driver unable to online iqn.1986-03.com.sun:01:00093d12170c.434c5250.vol0 lun 0

What this really means is the associated target driver is not configured or is failing its attach(). Those sneaky customers out there applying express packages against S10 will see this 100% of the time on sparc. The problem is not with the iSCSI software. Your missing a /etc/driver_aliases entry. You need to ensure you have the following driver_alias entry to cause the iSCSI logical unit to bind to the target driver. This entry states that a class scsi device with INQUIRY Device Type of 00 (Disk) should bind to the target driver sd (Disk Driver).

sd "scsiclass,00"

You can find more details on driver_aliases in its manpage.

If you apply a full Solaris Nevada install or use official S10 patches to get iSCSI this will not occur. We add the entry for you. As a final note, Beware stability levels are different between Solaris Express and official S10 patches. Solaris Express is the bleeding edge, similar to downloading code from opensolaris.org building it and installing it on your machine.

Posted at 08:38AM Oct 14, 2005 by dweibel in General  |  Comments[1]

iSCSI Kernel Visibilty

My presentation last week at the SNIA conference consisted of listing the features of the Solaris iSCSI Software Initiator. Some of those features were the standard marketing / protocol checkboxes and others included kernel statistics and dtrace probe points. These have been a tremendous time saver in development of the initiator and understanding performance characteristics of iSCSI targets.

Prior to my presentation I was doing some performance testing and I had a number of chances to show off this kernel visibilty. One of the most common performance characteristics I check on arrays is the maximum command sequeunce window size. iSCSI has a window in which the target can throttle the initiator so that it doesn't send to much data at any point in time. This is done by a simple exchange of sequeunce numbers. The target first tells the initiator the maximum number of commands that can be sent at any point in time. Assuming that the command the initiator wants to send is lower than this maximum the initiator sends the command and increaments the counter. Then on each response from the target it can open the window as more resources become available.

We have encountered performance problems in the past where we are running up against the maximum command window so the initiator can't send more IO and it sits idle. This can quickly be confirmed by both kstats(1M) or dtrace(1M).

The first approach is with kstats. To start out you need to issue a 'kstat -m iscsi'. This will report all the initiator hba, session, and connection structures. This report is long so pipe it to more or redirect the output to a file. Once you have this complete list of data you can focus on a specific instance by issuing a 'kstat -n '. More specifically you can use -n and grep for specific fields, in this case the cmd_sn. The results of this are show below.

        bash-2.05b# kstat -n iscsi_sess_1_42 | grep cmd_sn
        _cmd_sn                         1
        _cmd_sn_exp                     1
        _cmd_sn_max                     1

In this output the next command the initiator wants to send is 1. Also the maximum it can send is 1. So at this point in time it can send one IO at a time. The array can open the window greater as it has resources. Generally if you see the command sequeunce number to be one more than the maximum, the initiator is sitting idle waiting for the target to free up resources.

As a final note, related to kstats, if you add an interval value to the kstat command you can monitor these values in real time.

kstat(1M)'s are pasee these days at Sun so I will also give you a quick way to do the same monitoring with a dtrace(1M). If you paste the below chunk into some file, maybe /trace.d. Then chmod the permissions on this file to executable.

	#!/usr/sbin/dtrace -s
	fbt:iscsi:iscsi_timeout_checks:entry
	{
		printf("%s,%d - cmdsn:%d maxcmdsn:%d expcmdsn:%d",
		    (string)((iscsi_sess_t *)arg0)->sess_name,
		    ((iscsi_sess_t *)arg0)->sess_tpgt,
		    ((iscsi_sess_t *)arg0)->sess_cmdsn,
		    ((iscsi_sess_t *)arg0)->sess_maxcmdsn,
		    ((iscsi_sess_t *)arg0)->sess_expcmdsn);
	}

Once you execute the script file you will get results like the those below. If you put a little more work into it you can have dtrace give you deltas, filter specific targets and other interesting modifications.

        dtrace: script '/trace.d' matched 1 probe
        CPU     ID                    FUNCTION:NAME
          0  30136       iscsi_timeout_checks:entry ,  - \
            cmdsn:1 maxcmdsn:1 expcmdsn:1
...

In summary seeing larger windows like the below window, is normally better for performance. The assumption is the target has to be able to handle the window it allows in a reasonable period of time. If it can't, the SCSI packet timeout is going to be exceeded and the initiator will start to send task management aborts.

	bash-2.05b# kstat -n iscsi_sess_1_42 | grep cmd_sn
        _cmd_sn                         138
        _cmd_sn_exp                     138
        _cmd_sn_max                     1162

And the below example of a window size of one is poor, unless it a tape drive. Then its pretty normal.

	bash-2.05b# kstat -n iscsi_sess_1_46 | grep cmd_sn
        _cmd_sn                         439
        _cmd_sn_exp                     439
        _cmd_sn_max                     439

Advice to array vendors is to be very careful about your window handling. Do allow us to send IO the target can't handle or your customer is going to have a bad experience.

I will try and publish more clips later

Posted at 02:29PM Aug 12, 2005 by dweibel in General  |  Comments[2]

SNIA Dev. Conf. / Plug Fest

I'm slowly getting caught up from the DevConf/PlugFest last week. Once I'm back to normal I will post some highlights. I just wanted to send out a quick note of thanks to all those iSCSI vendors that attended the PlugFest and the individuals that attended my presentation on the Solaris iSCSI Initiator. You can access the slides from my presentation by one of the following to links. (staroffice) (pdf)

The number one comment I heard from people related to the Solaris iSCSI Initiator was "When is it going to ship?" It was nice to be able to say you can download it today as part of Solaris Express. I wish I could have said, Download Solaris 10 Update 1 today. Although the feature list for Solaris 10 Update 1 is still growing so it will be well worth the wait.

This years Plug Fest attendance was pretty light so Sun is going to start drumming up support early for next years 2006 Connectathon. Sun has had a long history with Connectathon promoting interop testing with NIF/CIFS/etc. So we are going to continue promoting Connectathon for iSCSI. We encourage all the iSCSI vendors out there to attend. We are also already working on a couple other key vendors to attend so it will be well worth everyones time.

Posted at 05:21PM Aug 10, 2005 by dweibel in General  |  Comments[1]

SNIA Developers Conference - Aug 1- 5 (iSCSI/MPAPI/SMI-S)

  Sun will be shipping me and a couple peers to San Jose for the SNIA Developers Conference the week of Aug. 1-5. Paul Von Behren who sits across from me will be presenting "How to Read and use the SMI-S Specification" Aug. 1. Aug. 3 he will also present "Multipath Management API". At the same time on Aug. 3 I will be presenting in another room, "iSCSI Initiator Support - Solaris". The on Aug. 4 Mark Carlson will be presenting "SMI-S Backwards Compatability & Future Directions"

  Stop by and say hello. I all be at the convention all week.

  http://www.developersolutions.org/events/developersolutions/Agenda/

Posted at 02:21PM Jul 17, 2005 by dweibel in General  |  Comments[1]

iSCSI Documentation Available

I just noticed some of the official iSCSI documentation is available on docs.sun.com.

http://docs.sun.com/app/docs/doc/819-2723/6n50a1n01?a=view

Posted at 06:49PM Jun 20, 2005 by dweibel in General  |  Comments[0]

Important iSCSI Links

A couple important iSCSI links(shortcuts) for those digging into iSCSI internals.

STANDARDS

RFC 3720 - Internet Small Computer Systems Interface (iSCSI)

RFC 3980 - T11 Network Address Authority (NAA) Naming Format for iSCSI Node Names

RFC 3721 - Internet Small Computer Systems Interface (iSCSI) Naming and Discovery

RFC 3722 - String Profile for Internet Small Computer Systems Interface (iSCSI) Names

RFC 3385 - Internet Protocol Small Computer System Interface (iSCSI) Cyclic Redundancy Check (CRC)/Checksum Considerations

RFC 3783 - Small Computer Systems Interface (SCSI) Command Ordering Considerations with iSCSI

RFC 3347 - Small Computer Systems Interface protocol over the Internet (iSCSI) Requirements and Design Considerations

RFC 3723 - Securing Block Storage Protocols over IP

draft-ietf-ips-isns-22 - Internet Storage Name Service (iSNS)

draft-ietf-ips-iscsi-boot-12 - Bootstrapping Clients using the iSCSI Protocol

draft-ietf-ips-iscsi-slp-09 - Finding Internet Small Computer Systems Interface (iSCSI) Targets and Name Servers using Service Location Protocol version 2 (SLPv2)

draft-ietf-ips-iscsi-mib-10 - Definitions of Managed Objects for iSCSI

draft-ietf-ips-iser-02 - iSCSI Extensions for RDMA Specification

draft-ietf-ips-iwarp-da-01 - Datamover Architecture for iSCSI (DA)

draft-ietf-ips-scsi-mib-07 - Definition of Managed Objects for SCSI Entities

Posted at 09:00AM May 13, 2005 by dweibel in General  |  Comments[0]

Introduction

My name is David Weibel, and I'm a Development Engineer in Network Storage. I've been working at Sun for the past 4 1/2 years on host side storage device drivers/utilities. The first product I helped write and release at Sun was StorEdge Traffic Manager for Windows NT/2000/2003. I worked on that product through its 2.0 -> 4.0 releases.

Since then I have transitioned from working on Traffic Manager. I'm now working with a small team to release the Solaris iSCSI Software Initiator. The iSCSI Initiator is currently available via the Solaris Express program and will officially release as part of Solaris 10 Update 1. To help Solaris Express customers I have worked with Torrey get some basic documentation posted on his blog. I also troll Sun Forums for iSCSI related questions. If your using Solaris Express the Sun Forums are a great place to post your questions. If you don't receive a quick response on the forums feel free to cross post a link into the comments of my blog.

Posted at 09:00AM May 10, 2005 by dweibel in General  |  Comments[1]