Virtualization and ST25XX/6000/99XX Said Syed's Weblog

Matt,

Currently, you cant really reserve a PCI card for a Virtual Machine. However, one of the ways to get around this is to not map any luns from that JBOD to any other Virtual Machine.

To answer your question regarding ZFS. Yes, you can create ZFS pools from LUNs on that JBOD without VMware interfering with your ZFS communication. You can do that by mapping the LUNs to the Virtual machine as Raw Device Mappings instead of VMFS volumes. An RDMs give direct access to storage LUNs from Virtual Machines.

RDM (Raw Device Mapping)
=======================
In cases where a virtual machine requires direct access to storage and without VMFS-3 file system, a Raw Device Mapping can be used. This allows for the virtual machine to have raw access to the lun(s) just like a physical server would with the exception of data location, permission and locking which is performed by a mapping file resident in a VMFS volume.

RDM with Physical Compatibility Mode
====================================
An RDM with Physical Compatibility Mode provides direct access to the lun or physical disk for applications requiring lower level control. The virtualization layer in the ESX server storage I/O stack is by-passed and the mapping file is directly accessed.

RDM with Virtual Compatibility Mode
===================================
An RDM with Virtual Compatibility Mode provide for complete virtualization of the mapped device. The guest operating system thinks of the Lun as a virtual disk file in a VMFS-3 volume. This mode allows users to take advantage of snapshots as well as remote replication as it is more portable across storage hardware.

For ZFS, I would suggest RDM in Physical Compatibility Mode but you would lose the capability to create snapshots in VMware. Not sure if that is important to you or not.

Regarding the performance questions, as long as your ESX server is sized properly and not over committed too much on memory etc, I dont think you will see much performance degradation in terms zfs pools.

Here is a good VMware document that talks about RDMs and VMFS volumes and the differences:
http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_san_cfg.pdf

Give it a try. Let me know how it works out.

Said-

Posted by Said Syed on January 30, 2009 at 09:03 AM CST #

Matt,

I got your response in my email but it looks like for some reason it did not post to the blog. So I am incorporating my answer to your questions here:

Matt:
=====
Said,

Thank you for your quick and helpful response - I did more reading and still dont completely understand about physical mapping a RDM for my external attached SCSI JBOD. Do I need to set up a Physical Compatibility Mode RDM for each hard drive in the JBOD, or can I just make an RDM for the PCI card and get all the devices attached to it?

SAID:
=====
First off, you need to ensure that the JBOD you are using is supported on ESX servers. To do that, you will need refer to the SAN HCL guide:

http://www.vmware.com/resources/compatibility/pdf/vi_san_guide.pdf

Now, once thats done, the way it works is:

- Each hard drive would be recognized by the ESX server as a lun (depending on the kind of array/JBOD you have)
- You chose to either create VMFS volumes from on each of these LUNs or you simply go to your VM's "Edit Settings", go to "Hardware", Select Hard drive and click on "Add". Go through the menu and select Raw Device Mapping with Physical Compatibility Mode.
- Then you go to your OS and perform a devfsadm

Again, refer to the SAN administration guide I referred to and VMware Basic administration guide for further detailed assistance on this.

MATT:
=====
Your comment about the VM Snapshot is interesting - I assume that I would be able to snapshot the Solaris VM's boot disk images that are hosted on the ESX server's VMFS-3 filesystem, but it would simply ignore the RDM-mode ZFS scsi array. If so, that is fine, as I can snapshot in ZFS within the VM if I want to do that. Just have to keep all the meta layers straight.

SAID:
=====
You chose to create snapshots for your VM data (OS and data luns if they are in VMFS or RDM with Virtual Compatibility Mode). You just wont be able to snapshot RDMs in Physical Compatibility Mode. Yes, you can use ZFS snapshot within the VM's OS to do this.

MATT:
=====
Another question - If I run a "zpool xport tank" in the VM, can I detach this array then from the ESX server and plug it in to physical machine and import it?

SAID:
=====
You should be able to do that. ZFS pools dont know they are part of a virtual machine.

MATT:
=====
I export a ZFS pool on Solaris 10 x86 VM and then import it on a physical SF v240 Solaris 10 SPARC box?

SAID:
=====
You know, I dont know.. HMMM. I have to try it. Never thought about that. I will post my answer soon on this.

Posted by Said Syed on January 31, 2009 at 03:18 PM CST #

Said,

Im getting closer to trying this out.... I have a VMWare ESX 3.5 U3 server with an
LSI PCI-e scsi card in it. I attached an external SCSI jbod - a Dell PV220S w/6 U320
SCSI drives in it. I could not get the ESX server to add a disk as a RAW device and posted a Q to VMWare about it... this is what they sent me... a hack to make a raw device:

http://itknowledgeexchange.techtarget.com/virtualization-pro/vmware-esx-storage-how-to-get-local-storage-to-act-as-a-raw-disk-for-vms/

next move is to try this and add all 6 drives to ESX as raw devices - then use
opensolaris to try to make a ZFS FS with the set. I feel a little like Ive gone off the rails here-- im out in the weeds somewhere -- but have to give it a go anyway. :)

Matt

Posted by Matt on February 16, 2009 at 12:17 PM CST #

Hi; curretly we have several performance issues with a Brocade 48k Intermix FICON/FC/FCip environment. Under Ficon run VSM, FC for EMC DMX and FCip for replica to other Brocade 48k.

Some of you know the best practices for Intermix configuration or limitants that have this environment.

Which could be the best architecture for connections / blades / ports in this particular configuration?

Regards

Posted by Israel Erick Pacheco Caballero on February 26, 2009 at 02:09 AM CST #

Israel,

I am attaching your response here instead of email. I prefer we have our discussion over the Blog if that is ok with you so everyone can benefit:

====================
Hi;

Yes, our blades are FR4-18i.

This is a history.

Our customer has a FCIP extension solution that he’s using for FICON traffic. It’s VSM on one side, and 9940B tape drives on the other side, at a distance of 1000 Km. The FCIP tunnel has 150 Mbps of uncommited bandwidth. When the customer starts a remote tape backup through VSM, he says that the bandwidth used through the tunnel to write to two tape drives is only of 10 Megabytes per second (or 5 MB/s per tape drive). He would expect to see the full 150 Mbps in use, and he doesn’t know if he should blame the FCIP solution, or if the problem lies somewhere else. What do you think?

The other problem is about FC (EMC DMX). When the compression is disabled in brocade, we are able to transfer at 85% BW available (145Mbps), but with compression enabled, the performance goes down to 60% BW. FC have 4 process incoming from EMC DMX. When 1 process is running, the BW utilization is good, but when 4 processes are delivery, the BW utilization goes down, like a process saturation problem into the blades.

Do you know the best practices and limitations for INTERMIX environments.What could we expect as a maximum performance? FC has some recommendations that can not be enabled due to the FICON mixon architecture, like DLS or IOD.

I have a SAN healt if you want to know in deep our confguration. This problem actually is under revision by Sun Support USA and Brocade Support USA, but this question are not answered. I want to have your point of view and know cases like in your experience maybe have.

Thanks
Israel
============

Posted by Said Syed on February 26, 2009 at 12:28 PM CST #