Storage virtualization with COMSTAR and ZFS
COMSTAR is a software framework that enables you to turn any
OpenSolaris host into a SCSI target that can be accessed over the
network by initiator hosts. COMSTAR breaks down the huge task of
handling a SCSI target subsystem into independent functional modules.
These modules are then glued together by the SCSI Target Mode Framework (STMF).
COMSTAR features include:
* Extensive LUN Masking and mapping functions
* Multipathing across different transport protocols
* Multiple parallel transfers per SCSI command
* Scalable design
* Compatible with generic HBAs
COMSTAR is integrated into the latest OpenSolaris
In this entry I will demonstrate the integration between COMSTAR and ZFS
Architecture layout :
You can install all the appropriate COMSTAR packages
# pkg install storage-server
On a newly installed OpenSolaris system, the STMF service is disabled by default.
You must complete this task to enable the STMF service.
View the existing state of the service
# svcs stmf
disabled 15:58:17 svc:/system/stmf:default
Enable the stmf service
# svcadm enable stmf
Verify that the service is active.
# svcs stmf
online 15:59:53 svc:/system/stmf:default
Create a RAID-Z storage pool.
The server has six controllers each with eight disks and I have
built the storage pool to spread I/O evenly and to enable me to build 8
RAID-Z stripes of equal length.
# zpool create -f tank \
raidz c0t0d0 c1t0d0 c4t0d0 c6t0d0 c7t0d0 \
raidz c1t1d0 c4t1d0 c5t1d0 c6t1d0 c7t1d0
raidz c0t2d0 c4t2d0 c5t2d0 c6t2d0 c7t2d0 \
raidz c0t3d0 c1t3d0 c5t3d0 c6t3d0 c7t3d0 \
raidz c0t4d0 c1t4d0 c4t4d0 c6t4d0 c7t4d0 \
raidz c0t5d0 c1t5d0 c4t5d0 c5t5d0 c7t5d0 \
raidz c0t6d0 c1t6d0 c4t6d0 c5t6d0 c6t6d0 \
raidz c0t7d0 c1t7d0 c4t7d0 c6t7d0 c7t7d0 \
spare c0t1d0 c1t2d0 c4t3d0 c6t5d0 c7t6d0 c5t7d0
After the pool is created, the zfs utility can be used to create 50GB ZFS volume.
# zfs create -V 50g tank/comstar-vol1
Create a logical unit using the volume.
# sbdadm create-lu /dev/zvol/rdsk/tank/comstar-vol1
Created the following logical unit :
GUID DATA SIZE SOURCE
--------------------------- ------------------- ----------------
600144f07bb2ca0000004a4c5eda0001 53687025664 /dev/zvol/rdsk/tank/comstar-vol1
Verify the creation of the logical unit and obtain the Global Unique Identification (GUID) number for the logical unit.
# sbdadm list-lu
Found 1 LU(s)
GUID DATA SIZE SOURCE
-------------------------------- ------------------- ----------------
600144f07bb2ca0000004a4c5eda0001 53687025664 /dev/zvol/rdsk/tank/comstar-vol1
This procedure makes a logical unit available to all initiator hosts on a storage network.
Add a view for the logical unit.
# stmfadm add-view GUID_number
Identify the host identifier of the initiator host you want to add to your view.
Follow the instructions for each port provider to identify the initiators associated with each
port provider.
You can see that the port mode is Initiator
# fcinfo hba-port
HBA Port WWN: 210000e08b91facd
Port Mode: Initiator
Port ID: 2
OS Device Name: /dev/cfg/c16
Manufacturer: QLogic Corp.
Model: 375-3294-01
Firmware Version: 04.04.01
FCode/BIOS Version: BIOS: 1.4; fcode: 1.11; EFI: 1.0;
Serial Number: 0402R00-0634175788
Driver Name: qlc
Driver Version: 20080617-2.30
Type: L-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 200000e08b91facd
Max NPIV Ports: 63
NPIV port list:
Before making changes to the HBA ports, first check the existing port
bindings.
View what is currently bound to the port drivers.
In this example, the current binding is pci1077,2422.
# mdb -k
Loading modules: [ unix genunix specfs dtrace mac cpu.generic
cpu_ms.AuthenticAMD.15 uppc pcplusmp scsi_vhci zfs sd sockfs ip hook
neti sctp arp usba qlc fctl
fcp cpc random crypto stmf nfs lofs logindmux ptm ufs sppp nsctl ipc ]
> ::devbindings -q qlc
ffffff04e058f560 pci1077,2422, instance #0 (driver name: qlc)
ffffff04e058f2e0 pci1077,2422, instance #1 (driver name: qlc)
Quit mdb.
> $q
Remove the current binding, which in this example is qlc.
In this example, the qlc driver is actively bound to pci1077,2422.
You must remove the existing binding for qlc before you can add that
binding to a new driver.
Single quotes are required in this syntax.
# update_drv -d -i 'pci1077,2422' qlc
Cannot unload module: qlc
Will be unloaded upon reboot.
This message does not indicate an error.
The configuration files have been updated but the qlc driver remains
bound to the port until reboot.
Establish the new binding to qlt.
Single quotes are required in this syntax.
# update_drv -a -i 'pci1077,2422' qlt
Warning: Driver (qlt) successfully added to system but failed to
attach
This message does not indicate an error. The qlc driver remains bound
to the port, until reboot.
The qlt driver attaches when the system is rebooted.
Reboot the system to attach the new driver, and then recheck the
bindings.
# reboot
# mdb -k
Loading modules: [ unix genunix specfs dtrace mac cpu.generic
cpu_ms.AuthenticAMD.15 uppc pcplusmp scsi_vhci zfs sd sockfs ip hook
neti sctp arp usba fctl stmf lofs fcip cpc random crypto nfs logindmux
ptm ufs sppp nsctl ipc ]
> ::devbindings -q qlt
ffffff04e058f560 pci1077,2422, instance #0 (driver name: qlt)
ffffff04e058f2e0 pci1077,2422, instance #1 (driver name: qlt)
Quit mdb.
> $q
You can see that the port mode is Target
# fcinfo hba-port
HBA Port WWN: 210000e08b91facd
Port Mode: Target
Port ID: 2
OS Device Name: /dev/cfg/c16
Manufacturer: QLogic Corp.
Model: 375-3294-01
Firmware Version: 04.04.01
FCode/BIOS Version: BIOS: 1.4; fcode: 1.11; EFI: 1.0;
Serial Number: 0402R00-0634175788
Driver Name: qlc
Driver Version: 20080617-2.30
Type: L-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 200000e08b91facd
Max NPIV Ports: 63
NPIV port list:
Verify that the target mode framework has access to the HBA ports.
# stmfadm list-target -v
Target: wwn.210100E08BB1FACD
Operational Status: Online
Provider Name : qlt
Alias : qlt1,0
Sessions : 0
Target: wwn.210000E08B91FACD
Operational Status: Online
Provider Name : qlt
Alias : qlt0,0
Sessions : 1
Initiator: wwn.210000E08B89F077
Alias: -
Logged in since: Thu Jul 2 12:02:59 2009
Now for the client setup :
On the client machine verify that you can see the new logical unit
# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c0 scsi-bus connected configured unknown
c0::dsk/c0t0d0 CD-ROM connected configured unknown
c1 scsi-bus connected configured unknown
c1::dsk/c1t0d0 disk connected configured unknown
c1::dsk/c1t2d0 disk connected configured unknown
c1::dsk/c1t3d0 disk connected configured unknown
c2 fc-private connected configured unknown
c2::210000e08b91facd disk connected configured unknown
c3 fc connected unconfigured unknown
usb0/1 unknown empty unconfigured ok
usb0/2 unknown empty unconfigured ok
You might need to rescan the SAN BUS in order to discover the new logical unit
# luxadm -e forcelip /dev/cfg/c2
# format
Searching for disks...
AVAILABLE DISK SELECTIONS:
0. c1t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@0,0
1. c1t2d0 <FUJITSU-MAV2073RCSUN72G-0301-68.37GB>
/pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@2,0
2. c1t3d0 <FUJITSU-MAV2073RCSUN72G-0301-68.37GB>
/pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@3,0
3. c2t210000E08B91FACDd0 <SUN-COMSTAR-1.0 cyl 65533 alt 2 hd 16 sec 100>
/pci@7c0/pci@0/pci@9/SUNW,qlc@0/fp@0,0/ssd@w210000e08b91facd,0
Specify disk (enter its number):
You can see the SUN-COMSTAR-1.0 in the disk properties
Now you can build storage pool on top of it
# zpool create comstar-pool c2t210000E08B91FACDd0
Verify the pool creation
# zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT
comstar-pool 49.8G 114K 49.7G 0% ONLINE -
After the pool is created, the zfs utility can be used to create a ZFS volume.
# zfs create -V 48g comstar-pool/comstar-vol1
For more information about COMSTAR please check the COMSTAR project on OpenSolaris
