An Early Look - COMSTAR and VMware ESX 3.5
As I mentioned a post or two ago, I also test our various storage products with VMware ESX. One of the things I wanted to tinker with was Project COMSTAR and how, even at this early stage of development, it looked with VMware ESX.
COMSTAR stands for Common Multiprotocol SCSI Target and while eventually it will encompass iSCSI, SAS and FibreChannel target modes, for this testing I was going to focus on the FC target.
I first grabbed a SunBlade x8420 server running build 87 of OpenSolaris. This server had a dual port FC Express Module (our Qlogic version, current COMSTAR bits need a Qlogic chipset) and I was going to use one port on the card in target mode and the other port in traditional initiator mode to connect to a FC array in my lab. This is not exactly how you would do things in the real world as if you had a FC array, why not just connect directly to that? But, this is just a test. I would expect a normal installation to take something like a cheap JBOD connected in some manner and then serve that out via COMSTAR. I just needed some disk space and the FC array was handy. I re-zoned the switch I had to put one HBA port into a zone with my FC array to grab a 500GB lun, and put the other HBA port into a zone with my ESX server, to act as the target port.
So, once the hardware was cabled up right, I set about following the instructions on the COMSTAR project page on how to Install. That's pretty straightforward, so I'll just point to that page. When I was finished, I ended up with my desired HBA port bound to the qlt driver and in target mode as show below:
# mdb -k
Loading modules: [ unix genunix specfs dtrace cpu.generic cpu_ms.AuthenticAMD.15 uppc pcplusmp scsi_vhci ufs mpt ip hook neti sctp arp usba qlc fctl nca stmf lofs emlxs fcp md cpc random crypto zfs smbsrv nfs fcip logindmux ptm nsctl sdbc sv ii sppp nsmb rdc ]
> ::devbindings -q qlt
ffffff02d47726e0 /pci@1,0/pci10de,5d@d/pci111d,8018@0/pci111d,8018@2/pci1077,14b@0, instance #0 (driver name: qlt)
# fcinfo hba-port
...
HBA Port WWN: 2100001b320a61b4
Port Mode: Target
Port ID: 10100
OS Device Name: Not Applicable
Manufacturer: QLogic Corp.
Model: d59e69e0
Firmware Version: 4.3.1
FCode/BIOS Version: N/A
Type: F-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 2000001b320a61b4
Now that I had the driver set up, it was time to actually create the target lun. First I had to create my backing store. I decided to use a ZVOL from a ZFS file system residing on my FC array lun. Just to get it quickly going, I took a single 500GB FC lun and built a single zpool from it. Again, not optimal, but this was just supposed to be a quick test.
Now to build the ZVOL.
# zfs create -V 50g p1/fs/t1
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
p1 50.0G 438G 18K /p1
p1/fs 50.0G 438G 18K /comstar
p1/fs/t1 50G 488G 24K -
Now to create the COMSTAR lun with my ZVOL as the backing store
# sbadm create-lu -s 50g /dev/zvol/rdsk/p1/fs/t1
Created the following LU:
GUID DATA SIZE SOURCE
-------------------------------- ------------------- ----------------
6000ae40030000000000482ddd0b0001 53687091200 /dev/zvol/rdsk/p1/fs/t1
Next I wanted to add the appropriate mapping/masking to present this lun to my ESX server's HBA ports. I wanted to connect it to two HBA ports, to see if ESX would handle any server side multi-pathing. So I had to set up a HostGroup entry with the WWNs from my ESX server HBAs.
# stmfadm create-host esx2
# stmfadm add-hg-member -g esx2 wwn.210100E08BB0C5C2 wwn.210000E08B90C5C2
Finally, I added the lun to the host I created.
# stmfadm add-view -h esx2 6000ae40030000000000482ddd0b0001
I thought I was done at that stage, so I performed a rescan of the HBA in ESX. But nothing appeared. Strange. I checked the FC switch and noticed that my target port wasn't even logged into the switch.
A quick check of svcadm showed that the stmf service was disabled. Oops.
# svcadm enable stmf
# svcs -a | grep stmf
online 13:38:39 svc:/system/device/stmf:default
Now, a rescan picked up the lun. Here's a brief line from the ESX vmkernel log to show what it looks like.
May 16 13:38:17 esx2 vmkernel: 10:21:46:00.382 cpu1:1034)ScsiScan: 395: Path 'vmhba3:C0:T2:L0': Vendor: 'SUN ' Model: 'COMSTAR ' Rev: '1.0 '
May 16 13:38:17 esx2 vmkernel: 10:21:46:00.382 cpu1:1034)ScsiScan: 396: Type: 0x0, ANSI rev: 5
May 16 13:38:17 esx2 vmkernel: VMWARE SCSI Id: Supported VPD pages for vmhba3:C0:T2:L0 : 0x0 0x83 0x86
ESX gave me a multi-path, in Fixed mode, across the HBA ports. ESX doesn't really load-balance though, I/O will only go down the 'active' path.
Disk vmhba3:2:0 /dev/sdh (51200MB) has 2 paths and policy of Fixed
FC 6:1.0 210000e08b90c5c2<->2100001b320a61b4 vmhba3:2:0 On active preferred
FC 6:1.1 210100e08bb0c5c2<->2100001b320a61b4 vmhba4:2:0 On
From there I was able to create a vmfs3 file system on the lun and clone a VM into the new file system. I fired up Iozone in my Win2003 Virtual Machine and let it run. I noticed that the I/O was a bit choppy as measured at the Solaris server side, but my guess is that this is a result of the many layers of kernel and caching that I/Os are going down. (VM -> ESX file system -> ESX Kernel -> Solaris ZFS -> Solaris Kernel -> Array Cache). Like I said, this wouldn't be considered an optimal setup. I would probably swap the FC array for a cheap JBOD connected via SAS. Also, you could present the COMSTAR lun as a VMware RDM lun to further cut down on caching. Further testing and investigation will be needed to really hash out the best deployment strategies. Stay tuned for that.
So there you have it. VMware ESX connected via the COMSTAR FiberChannel target.
COMSTAR stands for Common Multiprotocol SCSI Target and while eventually it will encompass iSCSI, SAS and FibreChannel target modes, for this testing I was going to focus on the FC target.
I first grabbed a SunBlade x8420 server running build 87 of OpenSolaris. This server had a dual port FC Express Module (our Qlogic version, current COMSTAR bits need a Qlogic chipset) and I was going to use one port on the card in target mode and the other port in traditional initiator mode to connect to a FC array in my lab. This is not exactly how you would do things in the real world as if you had a FC array, why not just connect directly to that? But, this is just a test. I would expect a normal installation to take something like a cheap JBOD connected in some manner and then serve that out via COMSTAR. I just needed some disk space and the FC array was handy. I re-zoned the switch I had to put one HBA port into a zone with my FC array to grab a 500GB lun, and put the other HBA port into a zone with my ESX server, to act as the target port.
So, once the hardware was cabled up right, I set about following the instructions on the COMSTAR project page on how to Install. That's pretty straightforward, so I'll just point to that page. When I was finished, I ended up with my desired HBA port bound to the qlt driver and in target mode as show below:
# mdb -k
Loading modules: [ unix genunix specfs dtrace cpu.generic cpu_ms.AuthenticAMD.15 uppc pcplusmp scsi_vhci ufs mpt ip hook neti sctp arp usba qlc fctl nca stmf lofs emlxs fcp md cpc random crypto zfs smbsrv nfs fcip logindmux ptm nsctl sdbc sv ii sppp nsmb rdc ]
> ::devbindings -q qlt
ffffff02d47726e0 /pci@1,0/pci10de,5d@d/pci111d,8018@0/pci111d,8018@2/pci1077,14b@0, instance #0 (driver name: qlt)
# fcinfo hba-port
...
HBA Port WWN: 2100001b320a61b4
Port Mode: Target
Port ID: 10100
OS Device Name: Not Applicable
Manufacturer: QLogic Corp.
Model: d59e69e0
Firmware Version: 4.3.1
FCode/BIOS Version: N/A
Type: F-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: 2000001b320a61b4
Now that I had the driver set up, it was time to actually create the target lun. First I had to create my backing store. I decided to use a ZVOL from a ZFS file system residing on my FC array lun. Just to get it quickly going, I took a single 500GB FC lun and built a single zpool from it. Again, not optimal, but this was just supposed to be a quick test.
Now to build the ZVOL.
# zfs create -V 50g p1/fs/t1
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
p1 50.0G 438G 18K /p1
p1/fs 50.0G 438G 18K /comstar
p1/fs/t1 50G 488G 24K -
Now to create the COMSTAR lun with my ZVOL as the backing store
# sbadm create-lu -s 50g /dev/zvol/rdsk/p1/fs/t1
Created the following LU:
GUID DATA SIZE SOURCE
-------------------------------- ------------------- ----------------
6000ae40030000000000482ddd0b0001 53687091200 /dev/zvol/rdsk/p1/fs/t1
Next I wanted to add the appropriate mapping/masking to present this lun to my ESX server's HBA ports. I wanted to connect it to two HBA ports, to see if ESX would handle any server side multi-pathing. So I had to set up a HostGroup entry with the WWNs from my ESX server HBAs.
# stmfadm create-host esx2
# stmfadm add-hg-member -g esx2 wwn.210100E08BB0C5C2 wwn.210000E08B90C5C2
Finally, I added the lun to the host I created.
# stmfadm add-view -h esx2 6000ae40030000000000482ddd0b0001
I thought I was done at that stage, so I performed a rescan of the HBA in ESX. But nothing appeared. Strange. I checked the FC switch and noticed that my target port wasn't even logged into the switch.
A quick check of svcadm showed that the stmf service was disabled. Oops.
# svcadm enable stmf
# svcs -a | grep stmf
online 13:38:39 svc:/system/device/stmf:default
Now, a rescan picked up the lun. Here's a brief line from the ESX vmkernel log to show what it looks like.
May 16 13:38:17 esx2 vmkernel: 10:21:46:00.382 cpu1:1034)ScsiScan: 395: Path 'vmhba3:C0:T2:L0': Vendor: 'SUN ' Model: 'COMSTAR ' Rev: '1.0 '
May 16 13:38:17 esx2 vmkernel: 10:21:46:00.382 cpu1:1034)ScsiScan: 396: Type: 0x0, ANSI rev: 5
May 16 13:38:17 esx2 vmkernel: VMWARE SCSI Id: Supported VPD pages for vmhba3:C0:T2:L0 : 0x0 0x83 0x86
ESX gave me a multi-path, in Fixed mode, across the HBA ports. ESX doesn't really load-balance though, I/O will only go down the 'active' path.
Disk vmhba3:2:0 /dev/sdh (51200MB) has 2 paths and policy of Fixed
FC 6:1.0 210000e08b90c5c2<->2100001b320a61b4 vmhba3:2:0 On active preferred
FC 6:1.1 210100e08bb0c5c2<->2100001b320a61b4 vmhba4:2:0 On
From there I was able to create a vmfs3 file system on the lun and clone a VM into the new file system. I fired up Iozone in my Win2003 Virtual Machine and let it run. I noticed that the I/O was a bit choppy as measured at the Solaris server side, but my guess is that this is a result of the many layers of kernel and caching that I/Os are going down. (VM -> ESX file system -> ESX Kernel -> Solaris ZFS -> Solaris Kernel -> Array Cache). Like I said, this wouldn't be considered an optimal setup. I would probably swap the FC array for a cheap JBOD connected via SAS. Also, you could present the COMSTAR lun as a VMware RDM lun to further cut down on caching. Further testing and investigation will be needed to really hash out the best deployment strategies. Stay tuned for that.
So there you have it. VMware ESX connected via the COMSTAR FiberChannel target.
Hi, I've done the same ZFS/COMSTAR/FC om commodity hardware and using it for ESX Server backup. Will be interesting to follow your performance findings as you go along.
Posted by Per on May 17, 2008 at 12:29 PM MDT #