ZFS Clones, iSCSI and VMware ESX
I came across Mike La Spina's Ubiquitous Talk blog a while back and was stoked that we have people taking advantage of the recent changes in OpenSolaris to support the Software iSCSI Initiator in Vmware ESX. For the last while, we had compatibility issues but those have been addressed and code changes put back so there are some cool things we can do now. I've seen Mike post in the VMware VMTN forums about this as well so that is very cool.
I wanted to take things a bit further in this blog post and show how you can use the snapshot/cloning features in ZFS to easily provision virtual machines in ESX. I'm using svn_98 for this, running on a X4500. I'm not going to create a large iSCSI lun and use vmfs3 on it to store multiple VMs, instead I'm going to create a small lun to use as a RDM lun. This will allow me to take snapshots and clones at the virtual machine level, rather than at some coarser granularity.
So let's start by creating a simple 10G ZVOL (using -s for sparse) and turn it into an iSCSI target.
# zfs create -s -V 10g pool0/vmware/iscsi/lun0
Let's just take a look at how much space that uses. 34k! The power of the sparse!
# zfs list pool0/vmware/iscsi/lun0
NAME USED AVAIL REFER MOUNTPOINT
pool0/vmware/iscsi/lun0 34.1K 13.0T 34.1K -
As Mike also showed in his blog, I'm going to deviate from the quicker "shareiscsi=" parameter in zfs and set up my iscsi target manually. When I get to the clones, I'll be setting the lun numbers using the same method.
# iscsitadm create target -b /dev/zvol/rdsk/pool0/vmware/iscsi/lun0 zvolt
I also want to create an ACL mapping to my software initiator.
# iscsitadm list initiator
Initiator: isv6220c
iSCSI Name: iqn.1998-01.com.vmware:isv-6220c-5e4d4229
CHAP Name: Not set
# iscsitadm modify target -l isv6220c zvolt
# iscsitadm list target -v
Target: zvolt
iSCSI Name: iqn.1986-03.com.sun:02:941a05d7-e46d-4986-df97-f6b371e94281.zvolt
Connections: 0
ACL list:
Initiator: isv6220c
TPGT list:
LUN information:
LUN: 0
GUID: 0
VID: SUN
PID: SOLARIS
Type: disk
Size: 10G
Backing store: /dev/zvol/rdsk/pool0/vmware/iscsi/lun0
Status: online
After rescanning my ESX server, here's what I see. I have two paths because I have 2 Network ports on my X4500. ESX sets up Session Multipathing.
Disk vmhba32:3:0 /dev/sdb (10239MB) has 2 paths and policy of Fixed
iScsi sw iqn.1998-01.com.vmware:isv-6220c-5e4d4229<->iqn.1986-03.com.sun:02:941a05d7-e46d-4986-df97-f6b371e94281.zvolt vmhba32:3:0 On active preferred
iScsi sw iqn.1998-01.com.vmware:isv-6220c-5e4d4229<->iqn.1986-03.com.sun:02:941a05d7-e46d-4986-df97-f6b371e94281.zvolt vmhba32:4:0 On
Now it's time to create the VM in the VI3 client.
Right-Click on your ESX server -> New Virtual Machine. Choose "Custom"
Walk through the rest of the steps as you normally would and, because you chose "Custom", you will be able to choose your Virtual Disk method. Choose "Raw Device Mapping" and select the lun that is being presented from your OpenSolaris iSCSI target.
Follow your normal OS installation methods from here on out. I used Windows 2003 Enterprise x64 for a quick installation.
Taking a look at the filesystem usage
# zfs list pool0/vmware/iscsi/lun0
NAME USED AVAIL REFER MOUNTPOINT
pool0/vmware/iscsi/lun0 2.09G 13.0T 2.09G -
So, my 10G zvol is still only using 2G of space. I'll let this be my "golden" image. This is where I'd install VMTools, any patches needed, any custom apps..whatever. Once I get the image "Just Right", I can move on to snapping and cloning.
Shut down the Virtual Machine in VI3.
Make a snapshot
# zfs snapshot pool0/vmware/iscsi/lun0@w2k3-golden
Initially, the snapshot takes no additional space.
pool0/vmware/iscsi/lun0 2.09G 13.0T 2.09G -
pool0/vmware/iscsi/lun0@w2k3-golden 0 - 2.09G -
Next we clone it to make a new read/write copy.
# zfs clone pool0/vmware/iscsi/lun0@w2k3-golden pool0/vmware/iscsi/clone
Now add it as a new target.
Note the -u flag in my command below, here I'm creating multiple luns under the same target (zvolt). The "shareiscsi" parameter only lets you create new targets, all with lun0 only. This isn't a problem for a few targets, but VMware ESX has a 64 target limit and it also considers each path to a multi-pathed target as a brand new target. You could potentially run out of targets. That's why I like creating multiple luns under a single target.
# iscsitadm create target -u 1 -b /dev/zvol/rdsk/pool0/vmware/iscsi/clone zvolt
# iscsitadm list target -v
Target: zvolt
iSCSI Name: iqn.1986-03.com.sun:02:941a05d7-e46d-4986-df97-f6b371e94281.zvolt
Connections: 2
Initiator:
iSCSI Name: iqn.1998-01.com.vmware:isv-6220c-5e4d4229
Alias: isv-6220c.central.sun.com
Initiator:
iSCSI Name: iqn.1998-01.com.vmware:isv-6220c-5e4d4229
Alias: isv-6220c.central.sun.com
ACL list:
Initiator: isv6220c
TPGT list:
LUN information:
LUN: 0
GUID: 600144f048dd3e090000144f2103c800
VID: SUN
PID: SOLARIS
Type: disk
Size: 10G
Backing store: /dev/zvol/rdsk/pool0/vmware/iscsi/lun0
Status: online
LUN: 1
GUID: 0
VID: SUN
PID: SOLARIS
Type: disk
Size: 10G
Backing store: /dev/zvol/rdsk/pool0/vmware/iscsi/clone
Status: online
And here we can see we have 1 orginal lun, 1 snapshot and 1 clone..all taking just around 2.1G of space.
pool0/vmware/iscsi/clone 55.5K 13.0T 2.09G -
pool0/vmware/iscsi/lun0 2.09G 13.0T 2.09G -
pool0/vmware/iscsi/lun0@w2k3-golden 0 - 2.09G -
Rescan for the new lun in ESX. It shows up as vmhba32:3:1/vmhba32:4:1, indicating it is seen as a lun under the same target as before.
Disk vmhba32:3:0 /dev/sdb (10239MB) has 2 paths and policy of Fixed
iScsi sw iqn.1998-01.com.vmware:isv-6220c-5e4d4229<->iqn.1986-03.com.sun:02:941a05d7-e46d-4986-df97-f6b371e94281.zvolt vmhba32:3:0 On active preferred
iScsi sw iqn.1998-01.com.vmware:isv-6220c-5e4d4229<->iqn.1986-03.com.sun:02:941a05d7-e46d-4986-df97-f6b371e94281.zvolt vmhba32:4:0 On
Disk vmhba32:3:1 /dev/sdc (10239MB) has 2 paths and policy of Fixed
iScsi sw iqn.1998-01.com.vmware:isv-6220c-5e4d4229<->iqn.1986-03.com.sun:02:941a05d7-e46d-4986-df97-f6b371e94281.zvolt vmhba32:3:1 On active preferred
iScsi sw iqn.1998-01.com.vmware:isv-6220c-5e4d4229<->iqn.1986-03.com.sun:02:941a05d7-e46d-4986-df97-f6b371e94281.zvolt vmhba32:4:1 On
Now, we can follow the same method for Virtual Machine creation as above, only this time, use the new lun for the Raw Device. This will boot up as a true clone, so you probably will need to adjust hostname and network settings to avoid conflicts.
I started playing around with the clone to see how it affected space. I booted it up and added the recommended patches from Windows Update Center. You can see that it started using more space as it needed to track its own changed blocks. The snapshot still hasn't deviated from the master "lun0" yet.
pool0/vmware/iscsi/clone 849M 13.0T 2.78G -
pool0/vmware/iscsi/lun0 2.09G 13.0T 2.09G -
pool0/vmware/iscsi/lun0@w2k3-golden 0 - 2.09G -
Next I booted the VM on lun0 and patched it. Here we can see the snapshot adding some space as it copies in the original blocks from lun0 before they get modified. lun0 itself grows as we patch the VM on top of it.
pool0/vmware/iscsi/clone 850M 13.0T 2.78G -
pool0/vmware/iscsi/lun0 2.93G 13.0T 2.79G -
pool0/vmware/iscsi/lun0@w2k3-golden 144M - 2.09G -
Just for fun, I created one more clone, added it to the iscsi target and created a 3rd VM using the methods above.
After booting it up you can see that 3 Virtual Machines, each thinking they have a 10G boot disk and with a fully patched OS, are only using 3.79G of space in the pool0/vmware/iscsi filesystem. That's quite a bit of space savings over having each VM be its own discrete block of storage.
pool0/vmware/iscsi 3.79G 13.0T 38.4K /pool0/vmware/iscsi
pool0/vmware/iscsi/clone 851M 13.0T 2.78G -
pool0/vmware/iscsi/clone2 20.6M 13.0T 2.09G -
pool0/vmware/iscsi/lun0 2.94G 13.0T 2.79G -
pool0/vmware/iscsi/lun0@w2k3-golden 152M - 2.09G
