iSCSI Kernel Visibilty

My presentation last week at the SNIA conference consisted of listing the features of the Solaris iSCSI Software Initiator. Some of those features were the standard marketing / protocol checkboxes and others included kernel statistics and dtrace probe points. These have been a tremendous time saver in development of the initiator and understanding performance characteristics of iSCSI targets.

Prior to my presentation I was doing some performance testing and I had a number of chances to show off this kernel visibilty. One of the most common performance characteristics I check on arrays is the maximum command sequeunce window size. iSCSI has a window in which the target can throttle the initiator so that it doesn't send to much data at any point in time. This is done by a simple exchange of sequeunce numbers. The target first tells the initiator the maximum number of commands that can be sent at any point in time. Assuming that the command the initiator wants to send is lower than this maximum the initiator sends the command and increaments the counter. Then on each response from the target it can open the window as more resources become available.

We have encountered performance problems in the past where we are running up against the maximum command window so the initiator can't send more IO and it sits idle. This can quickly be confirmed by both kstats(1M) or dtrace(1M).

The first approach is with kstats. To start out you need to issue a 'kstat -m iscsi'. This will report all the initiator hba, session, and connection structures. This report is long so pipe it to more or redirect the output to a file. Once you have this complete list of data you can focus on a specific instance by issuing a 'kstat -n '. More specifically you can use -n and grep for specific fields, in this case the cmd_sn. The results of this are show below.

        bash-2.05b# kstat -n iscsi_sess_1_42 | grep cmd_sn
        _cmd_sn                         1
        _cmd_sn_exp                     1
        _cmd_sn_max                     1

In this output the next command the initiator wants to send is 1. Also the maximum it can send is 1. So at this point in time it can send one IO at a time. The array can open the window greater as it has resources. Generally if you see the command sequeunce number to be one more than the maximum, the initiator is sitting idle waiting for the target to free up resources.

As a final note, related to kstats, if you add an interval value to the kstat command you can monitor these values in real time.

kstat(1M)'s are pasee these days at Sun so I will also give you a quick way to do the same monitoring with a dtrace(1M). If you paste the below chunk into some file, maybe /trace.d. Then chmod the permissions on this file to executable.

	#!/usr/sbin/dtrace -s
	fbt:iscsi:iscsi_timeout_checks:entry
	{
		printf("%s,%d - cmdsn:%d maxcmdsn:%d expcmdsn:%d",
		    (string)((iscsi_sess_t *)arg0)->sess_name,
		    ((iscsi_sess_t *)arg0)->sess_tpgt,
		    ((iscsi_sess_t *)arg0)->sess_cmdsn,
		    ((iscsi_sess_t *)arg0)->sess_maxcmdsn,
		    ((iscsi_sess_t *)arg0)->sess_expcmdsn);
	}

Once you execute the script file you will get results like the those below. If you put a little more work into it you can have dtrace give you deltas, filter specific targets and other interesting modifications.

        dtrace: script '/trace.d' matched 1 probe
        CPU     ID                    FUNCTION:NAME
          0  30136       iscsi_timeout_checks:entry ,  - \
            cmdsn:1 maxcmdsn:1 expcmdsn:1
...

In summary seeing larger windows like the below window, is normally better for performance. The assumption is the target has to be able to handle the window it allows in a reasonable period of time. If it can't, the SCSI packet timeout is going to be exceeded and the initiator will start to send task management aborts.

	bash-2.05b# kstat -n iscsi_sess_1_42 | grep cmd_sn
        _cmd_sn                         138
        _cmd_sn_exp                     138
        _cmd_sn_max                     1162

And the below example of a window size of one is poor, unless it a tape drive. Then its pretty normal.

	bash-2.05b# kstat -n iscsi_sess_1_46 | grep cmd_sn
        _cmd_sn                         439
        _cmd_sn_exp                     439
        _cmd_sn_max                     439

Advice to array vendors is to be very careful about your window handling. Do allow us to send IO the target can't handle or your customer is going to have a bad experience.

I will try and publish more clips later

Posted at 02:29PM Aug 12, 2005 by dweibel in General  |  Comments[2]