Hasham Pathan's blog

Using ZFS iSCSI LUN

ZFS - A New Approach to Data Management - Works on Pooled Storage Model and eliminates the need of volume manager. If you are new to ZFS, you may want to take look at ZFS resources here One of the important feature of ZFS that people don't talk as much is "iSCSI support". Steps below should help you quickly bring up the iSCSI configuration using ZFS. The setup here includes OpenSolaris 2008.05 running on dual core AMD system. This system will be configured to provide ZFS iSCSI volume(iSCSI target). A system running Solaris 10 is used as iSCSI client(initiator). In this case the "iSCSI client" is a virtual machine running Solaris 10 05/08 on the same physical system. Note that these steps should work seamlessly on any other system with Solaris 10.

High Level Steps:

1. Create a ZFS Volume (In existing ZFS pool OR Create a New Pool if you have spare disks )
2. Make Volume as it iSCSI target
3. Configure Solaris as iSCSI Client and access the iSCSI LUN
4. Create & Mount the File System on iSCSI LUN

Step 1: Create Volume: In existing ZFS pool OR Create a New Pool if you have spare disk(s)
Since I am running OpenSolaris 2008.05, it by default comes with ZFS as root file system which means we already have a default ZFS pool configured. We will create a new volume on this default pool. You can list the currently configured ZFS Pools & ZFS volumes using the commands below.
-bash-3.2# zpool list
NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
rpool   230G  19.8G   210G     8%  ONLINE  -
-bash-3.2#

-bash-3.2# zfs list -t volume
No volumes Listed

Create a new volume on default pool name "rpool".  Note the use of  "-V" switch and 2gb as size.
-bash-3.2#zfs create -V 2gb rpool/testvol
-bash-3.2# zfs list -t volume
NAME            USED  AVAIL  REFER  MOUNTPOINT
rpool/testvol     2G   207G  62.7M  -
-bash-3.2# 

Listing the properties of the newly created volume. Note the "shareiscsi" value. It's set to "off" by default

-bash-3.2# zfs get all rpool/testvol
NAME           PROPERTY         VALUE                  SOURCE
rpool/testvol  type             volume                 -
rpool/testvol  creation         Mon Jul 28 11:31 2008  -
rpool/testvol  used             2G                     -
rpool/testvol  available        207G                   -
rpool/testvol  referenced       16K                    -
rpool/testvol  compressratio    1.00x                  -
rpool/testvol  reservation      none                   default
rpool/testvol  volsize          2G                     -
rpool/testvol  volblocksize     8K                     -
rpool/testvol  checksum         on                     default
rpool/testvol  compression      off                    default
rpool/testvol  readonly         off                    default
rpool/testvol  shareiscsi       off                     default
rpool/testvol  copies           1                      default
rpool/testvol  refreservation   2G                     local

Step 2: Make Volume as it iSCSI target
Check if you already have any iSCSI target configured on the system using the below command.
-bash-3.2# iscsitadm list target
-bash-3.2#
No targets Reported. Now let's make the "new ZFS volume" as iSCSI target using the command below. Verify the settings for the volume.
-bash-3.2# zfs set shareiscsi=on rpool/testvol
-bash-3.2# zfs get shareiscsi rpool/testvol
NAME           PROPERTY    VALUE          SOURCE
rpool/testvol  shareiscsi  on             local

Verify that the ZFS iSCSI target is listed using the command below:
-bash-3.2# iscsitadm list target
Target: rpool/testvol
    iSCSI Name: iqn.1986-03.com.sun:02:cad2198d-1048-4657-cf4f-aed99a4726fa
    Connections: 0
Note Down the iSCSI name, we will need it at the time of the iSCSI Client configuration.

3. Configure Solaris10 system as iSCSI Client and Access the iSCSI LUN
Login As Root user. Verify the number of Disks Seen by the OS - You can use "format" command for this. Note the number of disk reported before the "iSCSI client" setup.
bash-3.00# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c0d0 <DEFAULT cyl 2085 alt 2 hd 255 sec 63>
          /pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
Specify disk (enter its number):

As you can see, this system is reporting ONE Disk in "format" command output. Now let's proceed with the iSCSI client configuration. Note that you can configure iSCSI client to be in "discover mode" or "Static Mode". The steps below are for static mode. We are also assuming that this system wasn't configured to use iSCSI target. You may have to tweak these steps if you are already using iSCSI on your system.
bash-3.00# iscsiadm list static-config
No staitic configuration is reported. Add a static configuration Using the below command. Note that "iSCSI Name" provided and the IP address is the address of the iSCSI Target system.
bash-3.00# iscsiadm add static-config iqn.1986-03.com.sun:02:cad2198d-1048-4657-cf4f-aed99a4726fa,129.158.226.83

bash-3.00# iscsiadm list static-config
Static Configuration Target: iqn.1986-03.com.sun:02:cad2198d-1048-4657-cf4f-aed99a4726fa,129.158.226.83:3260
bash-3.00#

Enable the iSCSI client(initiator) in Static Mode using the below command:
bash-3.00# iscsiadm modify discovery --static enable

Note the detection of new device in system log messages.
bash-3.00# dmesg|tail -5
Jul 28 12:11:47 dhcp-cblr03-226-127 genunix: [ID 936769 kern.info] sd1 is /scsi_vhci/disk@g010000e0815ba8fc00002a00488d6a28
Jul 28 12:11:47 dhcp-cblr03-226-127 genunix: [ID 265984 kern.warning] WARNING: Page83 data not standards compliant SUN      SOLARIS          1
Jul 28 12:11:48 dhcp-cblr03-226-127 genunix: [ID 408114 kern.info] /scsi_vhci/disk@g010000e0815ba8fc00002a00488d6a28 (sd1) online
Jul 28 12:11:48 dhcp-cblr03-226-127 genunix: [ID 834635 kern.info] /scsi_vhci/disk@g010000e0815ba8fc00002a00488d6a28 (sd1) multipath status: degraded, path /iscsi (iscsi0) to target address: 0000iqn.1986-03.com.sun:02:cad2198d-1048-4657-cf4f-aed99a4726faFFFF,0 is online Load balancing: round-robin
Jul 28 12:11:54 dhcp-cblr03-226-127 /sbin/dhcpagent[52]: [ID 732317 daemon.warning] accept_v4_acknak: ACK packet on e1000g0 missing mandatory lease option, ignored
bash-3.00#

Now let's run "format" command again to see if report the newly detected device. You may choose to use it as raw device or use it under volume manager control. The steps here partition the device and create a UFS file system.
bash-3.00# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
       0. c0d0 <DEFAULT cyl 2085 alt 2 hd 255 sec 63>
          /pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
       1. c2t010000E0815BA8FC00002A00488D6A28d0 <DEFAULT cyl 1021 alt 2 hd 128 sec 32>
          /scsi_vhci/disk@g010000e0815ba8fc00002a00488d6a28
Specify disk (enter its number): 1
selecting c2t010000E0815BA8FC00002A00488D6A28d0
[disk formatted]...
truncated output...
format> p
Please run fdisk first.
format> fdisk
No fdisk table exists. The default partition for the disk is:
  a 100% "SOLARIS System" partition
Type "y" to accept the default partition,  otherwise type "n" to edit the
 partition table.
y
format> p
Truncted output.....
partition> p

Truncated output.......
partition> mod
Select partitioning base:
        0. Current partition table (original)
        1. All Free Hog
Choose base (enter number) [0]? 1
Truncated output.......

Do you wish to continue creating a new partition
table based on above table[yes]?
Free Hog partition[6]?
Enter size of partition '0' [0b, 0c, 0.00mb, 0.00gb]: 1gb
Enter size of partition '1' [0b, 0c, 0.00mb, 0.00gb]:
Enter size of partition '3' [0b, 0c, 0.00mb, 0.00gb]:
Enter size of partition '4' [0b, 0c, 0.00mb, 0.00gb]:
Enter size of partition '5' [0b, 0c, 0.00mb, 0.00gb]:
Enter size of partition '7' [0b, 0c, 0.00mb, 0.00gb]:

Okay to make this the current partition table[yes]? yes
Enter table name (remember quotes): "new"
Ready to label disk, continue? yes

Verify the parition table information:
partition> p
Current partition table (new):
Total disk cylinders available: 1020 + 2 (reserved cylinders)
Part      Tag    Flag     Cylinders        Size            Blocks
  0 unassigned    wm       1 -  512     1024.00MB    (512/0/0)  2097152
  1 unassigned    wm       0               0         (0/0/0)          0
Truncated output....
.....
partition> quit
format> quit
bash-3.00#

4. Create & Mount the File System on iSCSI LUN

Next step is to create a UFS file system on the iSCSI LUN. Note the device name passed to format command is iSCSI LUN slice 0 (s0) - The slice we created using the "partition menu" of the format command.
bash-3.00# newfs /dev/rdsk/c2t010000E0815BA8FC00002A00488D6A28d0s0
newfs: construct a new file system /dev/rdsk/c2t010000E0815BA8FC00002A00488D6A28d0s0: (y/n)? y
/dev/rdsk/c2t010000E0815BA8FC00002A00488D6A28d0s0:      2097152 sectors in 512 cylinders of 128 tracks, 32 sectors
        1024.0MB in 26 cyl groups (20 c/g, 40.00MB/g, 19456 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 81984, 163936, 245888, 327840, 409792, 491744, 573696, 655648, 737600,
 1311264, 1393216, 1475168, 1557120, 1639072, 1721024, 1802976, 1884928,
 1966880, 2048832
bash-3.00#

Mount the file system using the command below:
bash-3.00# mount /dev/dsk/c2t010000E0815BA8FC00002A00488D6A28d0s0 /mnt
bash-3.00# df -k /mnt
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c2t010000E0815BA8FC00002A00488D6A28d0s0
                      984911    1041  924776     1%    /mnt

We are now READY to use file system created on ZFS iSCSI volume. You may want to update "/etc/vfstab" file for file systems to be mounted automatically across system reboots.

Posted at 12:43PM Aug 04, 2008 by Hashamkha Pathan in Sun  |  Comments[1]