Ryan Arneson's Weblog

I came across Mike La Spina's Ubiquitous Talk blog a while back and was stoked that we have people taking advantage of the recent changes in OpenSolaris to support the Software iSCSI Initiator in Vmware ESX. For the last while, we had compatibility issues but those have been addressed and code changes put back so there are some cool things we can do now. I've seen Mike post in the VMware VMTN forums about this as well so that is very cool.

I wanted to take things a bit further in this blog post and show how you can use the snapshot/cloning features in ZFS to easily provision virtual machines in ESX. I'm using svn_98 for this, running on a X4500. I'm not going to create a large iSCSI lun and use vmfs3 on it to store multiple VMs, instead I'm going to create a small lun to use as a RDM lun. This will allow me to take snapshots and clones at the virtual machine level, rather than at some coarser granularity.


So let's start by creating a simple 10G ZVOL (using -s for sparse) and turn it into an iSCSI target.

# zfs create -s -V 10g pool0/vmware/iscsi/lun0

Let's just take a look at how much space that uses. 34k! The power of the sparse!

# zfs list pool0/vmware/iscsi/lun0
NAME                      USED  AVAIL  REFER  MOUNTPOINT
pool0/vmware/iscsi/lun0  34.1K  13.0T  34.1K  -

As Mike also showed in his blog, I'm going to deviate from the quicker "shareiscsi=" parameter in zfs and set up my iscsi target manually. When I get to the clones, I'll be setting the lun numbers using the same method.

# iscsitadm create target -b /dev/zvol/rdsk/pool0/vmware/iscsi/lun0 zvolt

I also want to create an ACL mapping to my software initiator.

# iscsitadm list initiator
Initiator: isv6220c
    iSCSI Name: iqn.1998-01.com.vmware:isv-6220c-5e4d4229
    CHAP Name: Not set

# iscsitadm modify target -l isv6220c zvolt

# iscsitadm list target -v
Target: zvolt
    iSCSI Name: iqn.1986-03.com.sun:02:941a05d7-e46d-4986-df97-f6b371e94281.zvolt
    Connections: 0
    ACL list:
        Initiator: isv6220c
    TPGT list:
    LUN information:
        LUN: 0
            GUID: 0
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size:   10G
            Backing store: /dev/zvol/rdsk/pool0/vmware/iscsi/lun0
            Status: online

After rescanning my ESX server, here's what I see. I have two paths because I have 2 Network ports on my X4500. ESX sets up Session Multipathing.

Disk vmhba32:3:0 /dev/sdb (10239MB) has 2 paths and policy of Fixed
 iScsi sw iqn.1998-01.com.vmware:isv-6220c-5e4d4229<->iqn.1986-03.com.sun:02:941a05d7-e46d-4986-df97-f6b371e94281.zvolt vmhba32:3:0 On active preferred
 iScsi sw iqn.1998-01.com.vmware:isv-6220c-5e4d4229<->iqn.1986-03.com.sun:02:941a05d7-e46d-4986-df97-f6b371e94281.zvolt vmhba32:4:0 On

Now it's time to create the VM in the VI3 client.

Right-Click on your ESX server -> New Virtual Machine. Choose "Custom"

Walk through the rest of the steps as you normally would and, because you chose "Custom", you will be able to choose your Virtual Disk method. Choose "Raw Device Mapping" and select the lun that is being presented from your OpenSolaris iSCSI target.

Follow your normal OS installation methods from here on out. I used Windows 2003 Enterprise x64 for a quick installation.

Taking a look at the filesystem usage

# zfs list pool0/vmware/iscsi/lun0
NAME                      USED  AVAIL  REFER  MOUNTPOINT
pool0/vmware/iscsi/lun0  2.09G  13.0T  2.09G  -

So, my 10G zvol is still only using 2G of space.  I'll let this be my "golden" image. This is where I'd install VMTools, any patches needed, any custom apps..whatever. Once I get the image "Just Right", I can move on to snapping and cloning.

Shut down the Virtual Machine in VI3.

Make a snapshot

# zfs snapshot pool0/vmware/iscsi/lun0@w2k3-golden

Initially, the snapshot takes no additional space.

pool0/vmware/iscsi/lun0              2.09G  13.0T  2.09G  -
pool0/vmware/iscsi/lun0@w2k3-golden      0      -  2.09G  -

Next we clone it to make a new read/write copy.

# zfs clone pool0/vmware/iscsi/lun0@w2k3-golden pool0/vmware/iscsi/clone


Now add it as a new target.

Note the -u flag in my command below, here I'm creating multiple luns under the same target (zvolt). The "shareiscsi" parameter only lets you create new targets, all with lun0 only. This isn't a problem for a few targets, but VMware ESX has a 64 target limit and it also considers each path to a multi-pathed target as a brand new target. You could potentially run out of targets. That's why I like creating multiple luns under a single target.

# iscsitadm create target -u 1 -b /dev/zvol/rdsk/pool0/vmware/iscsi/clone zvolt

# iscsitadm list target -v
Target: zvolt
    iSCSI Name: iqn.1986-03.com.sun:02:941a05d7-e46d-4986-df97-f6b371e94281.zvolt
    Connections: 2
        Initiator:
            iSCSI Name: iqn.1998-01.com.vmware:isv-6220c-5e4d4229
            Alias: isv-6220c.central.sun.com
        Initiator:
            iSCSI Name: iqn.1998-01.com.vmware:isv-6220c-5e4d4229
            Alias: isv-6220c.central.sun.com
    ACL list:
        Initiator: isv6220c
    TPGT list:
    LUN information:
        LUN: 0
            GUID: 600144f048dd3e090000144f2103c800
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size:   10G
            Backing store: /dev/zvol/rdsk/pool0/vmware/iscsi/lun0
            Status: online
        LUN: 1
            GUID: 0
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size:   10G
            Backing store: /dev/zvol/rdsk/pool0/vmware/iscsi/clone
            Status: online


And here we can see we have 1 orginal lun, 1 snapshot and 1 clone..all taking just around 2.1G of space.

pool0/vmware/iscsi/clone             55.5K  13.0T  2.09G  -
pool0/vmware/iscsi/lun0              2.09G  13.0T  2.09G  -
pool0/vmware/iscsi/lun0@w2k3-golden      0      -  2.09G  -

Rescan for the new lun in ESX. It shows up as vmhba32:3:1/vmhba32:4:1, indicating it is seen as a lun under the same target as before.


Disk vmhba32:3:0 /dev/sdb (10239MB) has 2 paths and policy of Fixed
 iScsi sw iqn.1998-01.com.vmware:isv-6220c-5e4d4229<->iqn.1986-03.com.sun:02:941a05d7-e46d-4986-df97-f6b371e94281.zvolt vmhba32:3:0 On active preferred
 iScsi sw iqn.1998-01.com.vmware:isv-6220c-5e4d4229<->iqn.1986-03.com.sun:02:941a05d7-e46d-4986-df97-f6b371e94281.zvolt vmhba32:4:0 On

Disk vmhba32:3:1 /dev/sdc (10239MB) has 2 paths and policy of Fixed
 iScsi sw iqn.1998-01.com.vmware:isv-6220c-5e4d4229<->iqn.1986-03.com.sun:02:941a05d7-e46d-4986-df97-f6b371e94281.zvolt vmhba32:3:1 On active preferred
 iScsi sw iqn.1998-01.com.vmware:isv-6220c-5e4d4229<->iqn.1986-03.com.sun:02:941a05d7-e46d-4986-df97-f6b371e94281.zvolt vmhba32:4:1 On

Now, we can follow the same method for Virtual Machine creation as above, only this time, use the new lun for the Raw Device. This will boot up as a true clone, so you probably will need to adjust hostname and network settings to avoid conflicts.

I started playing around with the clone to see how it affected space. I booted it up and added the recommended patches from Windows Update Center. You can see that it started using more space as it needed to track its own changed blocks. The snapshot still hasn't deviated from the master "lun0" yet.

pool0/vmware/iscsi/clone              849M  13.0T  2.78G  -
pool0/vmware/iscsi/lun0              2.09G  13.0T  2.09G  -
pool0/vmware/iscsi/lun0@w2k3-golden   0      -  2.09G  -

Next I booted the VM on lun0 and patched it. Here we can see the snapshot adding some space as it copies in the original blocks from lun0 before they get modified. lun0 itself grows as we patch the VM on top of it.

pool0/vmware/iscsi/clone              850M  13.0T  2.78G  -
pool0/vmware/iscsi/lun0              2.93G  13.0T  2.79G  -
pool0/vmware/iscsi/lun0@w2k3-golden   144M      -  2.09G  -

Just for fun, I created one more clone, added it to the iscsi target and created a 3rd VM using the methods above.

After booting it up you can see that 3 Virtual Machines, each thinking they have a 10G boot disk and with a fully patched OS, are only using 3.79G of space in the pool0/vmware/iscsi filesystem. That's quite a bit of space savings over having each VM be its own discrete block of storage.

pool0/vmware/iscsi                   3.79G  13.0T  38.4K  /pool0/vmware/iscsi
pool0/vmware/iscsi/clone              851M  13.0T  2.78G  -
pool0/vmware/iscsi/clone2            20.6M  13.0T  2.09G  -
pool0/vmware/iscsi/lun0              2.94G  13.0T  2.79G  -
pool0/vmware/iscsi/lun0@w2k3-golden   152M      -  2.09G

I've published two additional papers to BigAdmin. Similar to the X4500 + EBS/NBU papers posted earlier, these papers show how to use Sun's new JBODs with Solaris/ZFS and Disk Backup.

Check them out

Sun Storage J4400 Array as Disk Storage for Symantec Veritas NetBackup 6.5 Software

Sun Storage J4400 Array as Disk Storage for Sun StorageTek Enterprise Backup Software 7.4

The papers' examples show the 24-disk J4400, but all the theory and operation would apply to the 12-disk J4200 as well. You'd just have half the storage.

Enjoy.

One of my jobs here at Sun is to get various devices, arrays, HBAs on the VMware Hardware Compatibility List. This gives everyone the warm fuzzies that our products are tested and compatible with the ever popular VMware ESX.

In the early days, I did the FibreChannel arrays (Sun's 6x00, 2x00 & 3x00 lines) as well as the NAS arrays (our 5x00 products). After a re-org or two, I mainly focus on the NAS line as well as what we could call our Storage Servers..which brings me to the X4500.

A post or two ago, I gave some pointers on how to setup the X4500 as a big disk cache for backups (Netbackup, EBS/Legato). This would be somewhat of a traditional use of the X4500. It's a Solaris server, it's got a lot of disk (48 SATA drives from 500GB to 1TB in size). ZFS makes volume management a snap.

Another way to think of the X4500 is as a storage target. It has network ports, it can talk NFS, iSCSI, Samba. So why not make it look like a BIG NAS array and see what happens!

Now, let's bring VMware ESX into the mix. Everyone knows that you can connect up FibreChannel storage to ESX and create your vmfs3 filesystem/datastores. Most people also know that you can use iSCSI luns as well. What some people aren't aware of is that NFSv3 connected devices are also possible as datastores.

VMware Configuration

In order to connect NFSv3 mountpoints to the vmkernel, the vmkernel needs it's own IP. This is accomplished in the Configuration -> Network screen of your VI3 client application. It would be a best practice to put this "data network" vmkernel on a separate interface than your main Service Console and Virtual Machine Network.

Another tweak you might want to do is bump the maximum number of NFS mountpoints up from the default of 8 to 32. This is done in the Advanced Settings -> NFS section.

X4500 Configuration

To start with, I've been testing the X4500 with Solaris 10u4 and Solaris 10u5 installed. I usually jumpstart with the SUNWCall group (everything except OEM). Nothing special otherwise. Most people find it prudent to mirror the boot drives using Solaris Volume Manager. Check out the installation guide for those procedures. This will also work using any of the OpenSolaris flavors.

Configuring the X4500 is a matter of setting up the zpool and zfs filesystems and sharing them out via NFS. You can choose a mirrored config or RAIDZ/RAIDZ2 depending on your specific needs around capacity, performance and RAS. For a whitepaper I'm working on I chose a mirrored configuration consisting of 22 mirrored pairs and 2 hotspares, but your needs and configuration may vary. I chose mirror because I thought the Random nature of Vmware I/O patterns would make a mirror config the obvious choice, however, in further testing I'm seeing that there really isn't much difference between the IOPs and ResponseTime in any of the configs. I'm guessing that the I/Os, having to go through the various layers and NFS protocol, are getting cached and abstracted out enough to negate any perf advantage once it actually hits the spinning disk. Given that, I think I'm leaning towards a RAIDZ2 recommendation to maximize capacity and RAS. I hope to show some numbers in my whitepaper coming soon.

Once you've settled on a zpool configuration, you can create a few ZFS mountpoints. That's fairly straight forward.

# zfs create -o sharenfs=anon=0 nfspool/nfs1
# zfs set mountpoint=/nfs1 nfspool/nfs1

I use anon=0 because the vmkernel wants to write to the NFS filesystem as root. You could use an experimental ESX "Virtual Machine Delegate" feature under the Security Profile screen in the GUI that allows changing the user that writes/reads from the VMs to something else if you have security issues around anon=0.

Once that's done, it's a simple matter of adding the mountpoint to ESX via Configuration -> Storage -> Add Storage -> Network Filesystem. Give ESX the X4500 name or IP, the filesystem being shared (in this case /nfs1) and give your datastore a name.

That's it. You are now using the X4500 as a big NAS server for VMware ESX. There are some very interesting uses for this configuration. VMware's LabManger for one. LabManager uses NFS and really is looking for something big and low cost. That fits the X4500 to a tee. Or maybe you need a large system to store test VMs. Perhaps you need a lab environment that maybe doesn't need screaming performance, but really wants to keep space and costs low.

Since you are using the ZFS filesystem on the X4500, you get all of the fancy features like snapshots and cloning as well as checksums. Snapshots create the .zfs hidden filesystem under your mountpoint so it is a simple matter of logging onto the X4500, cd'ing to the directory you want and copying any files you want from your snapshot right back over. Cloned filesystem can be added by cloning a snapshot, sharing the newly promoted clone out via the "sharenfs" parameter and mounting it up on ESX just like a normal filesystem. Add the cloned VM's in that new filesystem to the Inventory and you are good to go.

There are a couple of things you need to keep in mind however. One, like all NFS implementations with ESX, you cannot do Raw Device Mapping. Another is that the X4500 itself could represent a Single-Point-of-Failure in that the motherboard would not be a redundant component and, due to its internal storage nature, it is not clusterable. There are multiple network ports and redundant power supplies, but if the MB dies, the box dies. Again, depending on your usage, this may or may not be a problem.

Hopefully, this information will be enough to get the curious started and stay tuned for my upcoming Whitepaper.

In a future post, I hope to talk about using the X4500 as a iSCSI target array. Yes...we are fixing the iSCSI compatibility problem we have with ESX's initiator and our Solaris iSCSI target.

*edit* Added a clarifying paragraph on the Solaris versions used.

With this announcement we can see that Oracle is realizing what a powerful development environment Solaris 10 is. A world-class enterprise OS, coupled with the flexibility and security of Zones and the observability that you get with Dtrace? Simply unbeatable.

As someone who has been working with Solaris 10 and Oracle for the past year in a test environment, I can appreciate how indespensible things like Dtrace are when you start having to look under the covers.

This is great news. Having Oracle select Solaris 10 as the preferred 64-bit development and deployment environment is certainly a feather in Sun's cap.

My wife and I are planning to take a trip to Napa/Sonoma sometime this fall and are interested in maximizing our time touring wineries. Anyone out there have any suggestions? Must Sees? (Must avoids?) Any tips on getting around? Any and all suggestions welcome.

On a meta note, I do hope to post to this blog more than every 3-4 months.

To the person who commented in the "Sometimes I dream.." post below, the quote is from the MJ's Gatorade Commercial.

Sometimes I dream, that he is me You've got to see that's how I dream to be I dream I move, I dream I groove Like Mike If I could Be Like Mike Like Mike Oh, if I could Be Like Mike Be Like Mike, Be Like Mike Again I try Just need to fly For just one day if I could Be that way I dream I move I dream I groove Like Mike If I could Be Like Mike I wanna be, I wanna be Like Mike Oh, if I could Be Like Mike That's poetry man, poetry.

• Come Fly with Me (1989)
• Michael Jordan's Playground (1993)
• Air Time (1993)
• Above and Beyond (1996)
• His Airness (1999)

But what really makes this DVD set stand out is what is on discs 2 & 3 for this re-release version. You get 5 complete NBA games showcasing MJ at his best. His 63-point effort against the Boston Celtics in 1986. His 69-point game against the Cleveland Cavaliers (who else?) in 1990. The 55-points in Game 4 of the NBA Finals against Phoenix in 1993. The 38-point "flu" game against Utah in the 1997 NBA Finals and finally Game 6 of the 1998 NBA Finals, his last game as a Chicago Bull. These are complete games, not highlight reels, the only cuts are the commercials.

I've finished watching the Celtics game where he dropped in 63 in Double OT (only to lose) and am about halfway through the Cavs game and I'm loving every minute of it. Confession time, I have every Chicago Bull NBA Final game on VHS tape. 1991 versus the Lakers, 1992 versus the Trailblazers, 1993 versus the Suns. I have the ECF games versus the Knicks in 1994. I'm missing Jordon's comeback game in 1995 (but I specificallly bought the group Live's album Throwing Copper for the "Lightning Crashes" track, it was the theme music NBC played during the game intro), but I'm still hunting. I've got the great 1996 season cap against the Sonics and the 1997 and 1998 battles against the Jazz. Scattered through there I have various other playoff games as well. The 4 game sweep against the Shaq and the Magic, the battles against the Knicks and some against the Heat. As you can see, it is something of a sickness..as my wife reminds me everytime I scream when she suggests we throw out some of those old tapes. Humph! What does she know?

Anyway, if you're a fan of Number 23, I suggest you run, don't walk mind you, run, and get this DVD set.

What's amazing is the discussion that this one photograph (and others of a similar theme, Global Uses of Technology) has generated on a few internal email aliases I'm on. Personally I like the thought of showing global citizens benefiting from Sun Technology. It says a lot about what Sun is about. Technology, innovation, not necessarily Solaris 10 and Opteron servers, although those are key pieces. Keep in mind this is a branding campaign, not a sales pitch. You're not supposed to see some IT drone using Widget X.

I'm being completely blown away by some of my fellow employees assertion that this particular photo of a Sumo Wrestler wearing zubon (lightweight cotton pants) lying on a tatami using a PDA was somehow offensive. What!!? How can that be offensive? Admittedly, I'm a tough person to offend. I think this country (and the world for that matter) is waaay to easily offended these days. It's not enough to disagree intelligently anymore, no no no, that's not EXTREME enough. Doesn't demand a response. If you merely dislike something, I can safely chalk that up to subjective opinion and move on. No, one must be _offended_ to get any attention these days. Being offended implicitly demands some form of redress. The offense must be stopped, covered up, removed. Think of the children! Usually the next words you hear from the offended person are: Why, there ought to be a law!

In most cases though, if I squint just right and stand on one foot, I can sort of, vaguely, make out why someone would be offended. In this case, I am completely at a loss. I cannot see one offensive thing. Has America become so inundated by bland, Gap-attired IT Drones in our technology marketing that the shock of someone from another culture is just too much? Would a blond, blue-eyed, perky co-ed be acceptable? How Global is that?

What's truly ironic to me is I'm having these discussions on our internal Mac-users alias and the Bloggers alias, the two places, if any, I would think would embrace a global and stimulating perspective. In any case, if these pictures alone drum up the same level of discussion in the big wide Real World as they have on two little ol' email lists here at Sun, the movers and shakers behind the new branding campaign have already achieved something cool. In my mind at least.

I recalled wrong. It was wet sticky clay. About 200 yards of it down a steep hill to the railroad tracks and then back up the gully to the pavement. So here I sit in my office, covered in mud, my bike, sitting in the bike lockers outside, is weighed down by about 10 pounds of clay. Good thing it's Saturday. Should be a lovely ride home

On to the obligatory introductions! I'm a Test drone from sector 7G in the Network Storage division here in Broomfield, Colorado. I mostly dream up ways of acting like a customer and then break stuff accordingly, in the hopes of getting problems noticed before they hit customers. My focus is more on Customer Emulation of workloads than your normal Functional or System Test.

Right now, I'm mainly focused on incorporating Oracle RAC with various bits and pieces of Network Storage products, the QFS filesystem software in particular. I'll post various tidbits of interest as I go along.

A quick caveat though, I have no responsibility for any official qualifications at the moment, QFS, Oracle, Cluster or otherwise, so don't take any particular configuration you may see here as being officially being supported by Sun or Oracle. Check with the "Official Docs"(tm) at all times.