Building on the Bare Minimum

    When the need arises to upgrade from a bare minimum storage configuration or to start off with a more scalable storage configuration for VMware, Sun Storage arrays include all of the configuration combinations and choices needed. Virtualized Data Centers rely on on-demand highly scalable storage infrastructure in order to perform optimally. The need for more storage can increase exponentially over time and the storage infrastructure including storage arrays and the SAN need to be highly flexible in terms of scalability. The table below illustrates the scalability of Sun Storage arrays:

For latest and detailed Sun Storage architectural information, please refer to www.sun.com/storage.

Given the extensive possibilities on scalability for multiple architectural scenarios, Sun Storage arrays provide a very robust set of products and capabilities specifically for virtualization technologies such as VMware. In order to achieve optimal storage availability, performance and other architectural considerations, simple additions to the bare minimum designs may be sufficient.

Segregating Different Data Types

 For an optimal VMware infrastructure, segregating different types of data is essential. VMware makes use of storage for a number of reasons such as:

- Virtual Machine home directory which includes the VM Operating System, virtual memory and other essential objects
- Critical VM home directories should be segregated from Non-Critical VMs
   = Software ISO images for easy sharing and access of OS and application install files
- User and Application Data which may include critical and non-critical data such as User home directories and Employee Pay Roll

The need for Data Type Segregation arises due to the unique data access requirements for each. VM home directories have unique data access needs because the data includes the operating system and virtual memory of the VM which could lead to slow VM performance if placed along with random data type such as an Oracle OLTP  high performance random I/O database. Here are recommendations and strategies to address the issue of data segregation. :

1) Segregate Virtual Machine Home Directories based on the type of application and I/O. For example:
    - Test VMs and Production VMs
    - Database VMs which have higher memory requirements and may have higher swap file usage should not be on the same array volumes
2) Random I/O and Sequential I/O data should reside in different RAID Groups or Virtual Disks.
3) Use Luns or Volumes from different RAID Groups or Virtual Disks for different data types.
    - Luns for Virtual Machine Home Directories should not share the same RAID Group or Virtual Disk as an Oracle OLTP Database Lun
    - Non-Critical data such as ISOs and Non-Critical VM home directories may share the same RAID Group.
4) Use Low-Cost Modular Sun Storage arrays such as the Sun Storage 2510 for ISOs and non-critical VMs.
- VMs can boot from iSCSI targets on the ESX Server
    - Sun Storage 2510 can be shared across multiple ESX servers in order to take advantage of all VMware features
5) Mid-range modular for mid-range performance
    - Based on user requirements
6) High performance I/O intensive data such as Oracle OLTP, Data Warehousing should reside on high-end 6580/6780 and 9000
7) Use of Storage Domains on Sun Storage 2500 and 6000 series products is highly recommended in order to isolate ESX servers from other servers. Using Storage Domains allows creation of virtual partitions on the storage array and enables lun mapping per host initiator, host or host group. The following table shows Storage Domain feature scalability of the arrays.For more details on Storage Domains, please refer to Sun Storage Common Array Manager Installation Guide.

8) SAN switch zoning should also be used to segregate traffic for each initiator or HBA along with lun mapping and storage domains. Zoning is primarily used to isolate Initiator – to – Target traffic and disallows SAN error propagation across zones, the most important one being Registered State Change Notification events. An RSCN is a required notification sent out to all devices of a particular SAN, switch or a particular zone or set of zones every time due to a change in the SAN, causing the I/O traffic to halt momentarily. The change could be a host reboot, a cable plug/un-plug, a hardware error, a new storage device  and similar events. This I/O traffic halt, although brief, has been known to cause I/O exchange failures, path failures and in some very extreme situations data corruption See Figure 2 below. With zoning implemented, effects of RSCN are minimized to the specific zone which experienced the change event. Best practice for SAN zoning is to use one initiator (Host Bus Adapter) per zone and if possible, one target (Array port) per zone. Zoning also helps isolate Non-ESX server traffic from ESX server traffic.

In the figure above, there are three zones illustrated. Zone A and Zone C both have one initiator and one target, while Zone B has two initiators and targets. Both switches in the SAN are connected via an Inter-Switch Link. If Host X rebooted and it’s HBA in Zone B logs out of the SAN, an RSCN will be sent to Host Y’s initiator in Zone B and cause all I/O going to that initiator to halt momentarily and recover within seconds. However, another RSCN will be sent out to Host Y’s initiator in Zone B when Host X’s HBA logs back in to the SAN and cause another momentary halt in I\O. Initiators in Zone A and Zone C are protected from these events because there are no other initiators in these zones. Most latest SAN switches provide RSCN suppression methods, however, suppressing RSCNs is not recommended since RSCNs are the primary way for initiators to determine an event has occurred and to act on the specified event such as lost of access to targets. It is important to follow established SAN best practices such as single initiator zones in order to avoid situations described and others not listed.

Enhancing Availability

    VMware simply requires shared storage to enable all advanced features addressing Availability at the Virtual Machine level as long as there are more than one ESX servers sharing the same volume in a storage array. There is, however, a need to address Data Availability at the storage array level as well. Although Sun Storage arrays have built-in redundancy for data access, cache mirroring across controllers and power, and the likely hood of a complete failure is very small, it is recommended, if possible, to spread the data across multiple disk trays via mirror and/or striping on the low cost arrays specifically. Sun Storage 2500 series, 6000 series and StorageTek 9900V series offer modular add-on capabilities to attach disk trays/chassis/frame to an existing set of array controllers with no downtime in most cases. Striping data across multiple trays using RAID 5,6 or 1+0 enhances data availability by avoiding loss of access due to complete disk tray failure. Consider three virtual-disks, 4 disk RAID-5, a 5 disk RAID-6 and a 4 disk RAID-1+0, stripped/mirrored across a Sun Storage 2540 with four trays as shown in the figure below:

In this example, all virtual-disks will continue to function properly even with the loss of one entire disk tray. The RAID-6 virtual-disk will continue to to function even with the loss of two complete disk trays as long as the two trays did not include the bottom one. Similarly, The RAID-1+0 virtual-disk can sustain a combination of entire disk tray failures. This is just an example of how striping or mirroring across disk trays enhances availability and eliminates array single points of failures. For VMware environments where multiple servers are consolidated onto just a few physical machines and the data, including operating systems and virtual machine home directories, resident on shared storage, it is strongly recommended that RAID configurations similar to the example above be used to stripe and/or mirror data across multiple disk trays to avoid storage array single points of failures.

Enhancing Performance

    Storage I/O performance for VMware  can be enhanced using different strategies. Since VMware VI3  does not officially support Load-Balancing for Storage Multi-Pathing and only one path for each Lun is active on the ESX server at any given time regardless of the total number of paths available for use. Some important strategies include:

1) Lun Mapping should be balanced across each available Target Host Ports.
     - Spreading luns across all available target host ports is a best practice. This avoids overloading one target host port while other available ports are idle.
2) Avoid mapping more heavily used Luns to the same Target Host Ports unless the luns are mapped to a group of ports to further enhance availability.
- Preferred Owning Controller on the array and/or Preferred Path on VMware should be configured for each heavily used Lun mapped to multiple target ports such that one heavily used lun will use a different preferred target port over the other. It is recommended to use VMware to specify Preferred Path for Active/Active arrays such as the Sun StorageTek 9900 series and to set Preferred Owning Controller at the array for Active/Passive or Asymmetric arrays such as the Sun Storage 2500 and 6000 series arrays.

- In Figure-4 above, four Fibre Channel Host Bus Adapter ports on a Sun fire X4600-M2 are mapped to a Sun Storage 6780 array with one CSM200 disk tray through a fibre channel SAN. All four volumes are mapped to four Target Host Ports while one controller is the Preferred Owning Controller for each of the volumes as indicated above. Sun Common Array Manager can be used to configure Preferred Owning Controller for Sun Storage 2500 and 6000 series asymmetric arrays as show in Figure-5 below.

- VMware Preferred Path selection can be used for Active/Active storage arrays where all controllers on the subsystem present all paths to the ESX server as active paths for I/O and the server uses the specified path to be the Preferred Path when available. This can be done using VMware vCenter client as shown in Figure-6 below.
- It is not recommended to use both VMware and the array to specify preferred path. This can lead to Path Thrashing in case of a path failure or similar event. VMware will try to use any next available path while the array may want to force the Lun to use a different target port on the preferred controller if the lun was mapped to more than one target ports on the preferred controller. It is also not recommended to apply VMware Fixed Path Policy to an Asymmetric or Active/Passive controller array as it can lead to performance related and path thrashing problems as well. For further details on Path Thrashing, refer to VMware's SAN Configuration Guide.

3) Striping and mirroring across disk trays also enhances performance since I/O is separated across different back-end loops instead of the same loop being used to access disks on the same disk tray
4) RDM luns in VMware may offer slight performance improvement over VMFS volumes. If performance is a priority over some VMware features such as Storage VMotion, RDM can be the disk choice in order to slightly improve storage I/O performance. RDM volumes may be required for clustering such as Microsoft Cluster Service (MSCS) or data services on the storage arrays. More information on VMware RDM luns can be found in VMware's SAN Configuration Guide.

Tiered Storage Architecture

    Sun Storage Tiered Storage Architecture offer a multitude of possibilities, specifically for Virtualization technologies such as VMware. Tiered Storage can simplify Storage Infrastructure management by consolidating many storage sub-systems behind one which allows management and provisioning of the entire storage infrastructure from one storage array. Some uses of Tiered Storage are:
1) Consolidating Storage Sprawl behind one Storage Sub-System
    - Allows single point storage infrastructure management and administration
    - Allows single point lun storage provisioning
2) Protecting investment by attaching older storage sub-systems or disk trays behind newer and improved sub-system controllers. Luns resident on slower, older sub-systems or disk trays can be used for:
- Non-critical data such as ISO images, Test/Development Virtual machine home directories
    - Data Archiving and backup using in-system replication of production Luns to tiered sub-system and then provisioning the replicated luns to backup servers
    - Data not requiring high performance I/O
    - Data migration
    - Virtual Machines on older arrays may not have to be migrated to newer sub-systems. Tiered Storage can be used to provision the same luns to the ESX servers. A simple rescan of the luns from the ESX servers is sufficient to recognize the volumes and virtual machines on the luns. See Figures-7 below.

Sun Storage 6000 Series Tiered Storage Capabilities

    Sun Storage 6000 series series arrays offer unique Tiered Storage Architecture capabilities. A Sun Storage 6540 can be converted into the next generation Sun Storage 6780/6580 simply by replacing the 6450 array controller tray with a 6580/6780 controller tray. An existing 6140 can be moved under a 6580/6780 control by converting the 6140 controller tray into a Sun Storage CSM200 disk tray and then attaching to the 6780/6580 controller tray along with any existing CSM200 disk trays in the 6580/6780. This procedure allows migration of existing data and storage arrays, with data intact protection, under newer next generation 6580/6780 arrays which offer enhanced performance, scalability and reliability. See Tables 1 and 3 above. Users are able to re-use and protect capitol investment as a result of this capability.

NOTE: This procedure requires Sun Support Services on-site assistance. Failure to do so may result in complete data loss.

See Figure-8 below for an illustration of how a Sun Storage 6140 and 6540 is moved under Next-Generation a 6780 control.

Sun StorageTek 9900V Series Tiered Storage Capabilities

With Sun StorageTek 9900V series arrays' External Storage support, a complete Tiered Storage architecture can be created. Sun StorageTek 9900V offer enterprise class Tier-1 storage scalability, availability and performance (See Tables 1 and 3). Tier-2 and Tier-3 storage arrays can be virtualized behind Sun StorageTek 9900V arrays as external arrays. Luns or volumes from external arrays are provisioned through the front-end Sun StorageTek 9900V to the SAN for access by servers. A wide range of Sun Storage and Non-Sun Storage arrays are supported as External Storage for the Sun StorageTek 9900V. Additional capacity licenses may be needed to enable External Storage support and to take advantage of premium features such as Shadow Image (In-system replication), TrueCopy (Remote Replication), Dynamic Provisioning (Thin Provisioning) and others. Table-5 provides information on total external storage capacity currently supported with Sun StorageTek 9900V arrays.

Figure-9 is an illustration of a Sun StorageTek 9990V virtualizing an EMC Clariion and a Sun StorageTek 9970 array using the External Storage Feature for a Tiered Storage Architecture spreading older and non-critical data to older arrays while keeping critical Guest OS data, non-virtualized server data and high-performance data on the high performance newer array.

More details on Sun StorageTek 9900V External Storage Feature can be found on www.sun.com/storage

References

Sun Storage 6580 and 6780 Release Notes
- http://dlc.sun.com/pdf/820-5776-11/820-5776-11.pdf
Sun Storage 6580 and 6780 Hardware Installation Guide
- http://dlc.sun.com/pdf/820-5773-10/820-5773-10.pdf
Sun Storage 2500 Series Array Hardware Installation Guide
- http://dlc.sun.com/pdf/820-0015-12/820-0015-12.pdf
Sun Storage 6140 Array Hardware Installation Guide
- http://dlc.sun.com/pdf/819-7497-11/819-7497-11.pdf
Sun Storage Common Array Manager Software Installation Guide
- http://dlc.sun.com/pdf/820-5747-10/820-5747-10.pdf
Using RAID 6 for Increased Reliability Sun Blueprint
- http://wikis.sun.com/display/BluePrints/Using+RAID+6+for+Increased+Reliability+and+Performance
VMware Storage VMotion Best Practices for Sun Storage 2500 and 6000 Series Arrays
- http://www.sun.com/storage/white-papers/wmware-storage-VMotion.pdf
VMware VI3 and Virtual Infrastructure Basic System Administration Guide
- http://www.VMware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_admin_guide.pdf
VMware Storage/SAN Compatibility Guide
- http://www.VMware.com/resources/compatibility/pdf/vi_san_guide.pdf
VMware Remote Command-Line Interface Installation and Reference Guide
- http://www.VMware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_rcli.pdf
VMware vCenter Site Recovery Manager Compatibility Guidelines
- http://www.VMware.com/pdf/srm_storage_partners.pdf
VMware Fibre Channel SAN Configuration Guide
- http://www.VMware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_san_cfg.pdf

Building Blocks for Optimal Storage Design

    Storage architecture and design for VMware can become complex if not well planned and well thought out. To determine optimal storage design for VMware, following considerations need to be taken into account. All VMware features have a direct impact on how a storage infrastructure needs to be designed and architected. Cost, Capacity and Performance are also important points to consider while creating a storage layout for VMware. A “Cost Conscious”, peak performing scalable design is possible as long as all aspects of the design are understood. Some aspects of an Optimal Storage Design include the bare minimum design possible to create a functioning VMware infrastructure based on minimum configuration requirements and how scalable the design is.

The Bare Minimum Storage Design

    The “Bare Minimum Storage Design” is the minimum configuration essential for a functioning storage array. Bare minimum storage configurations are obviously very limited in terms of capacity, bandwidth and performance. If an application requires fully redundant, higher performance and higher capacity storage I/O, bare minimum storage configuration may not be an ideal choice. Sun Storage bare minimum configuration will support all VMware features. Following is the absolute bare minimum Sun Storage configurations supported with VMware and VMware features:

If there is a need to use Sun Storage Bare Minimum configurations for VMware environments, following best practices are recommended for optimal integration.

Disk and RAID layout

    With the limited number of disk drives available in the bare minimum Sun Storage array, choosing the right RAID type is essential. In order to take maximum advantage of a 5 disk configuration, all five disk drives should be of the same type and capacity. One drive needs to be designated as a Spare in case of a drive failure. With four remaining drives, there are limited options on choosing a RAID Group based on Cost, Capacity and Performance. For example, RAID-5 will provide available capacity of 3 out of the 4 disks with space equal to one disk used for parity stripes. RAID-1 will allow for usable space of upto two drives with the remaining two drives used for mirroring and decreased performance. RAID-0 will offer the most available space with usable capacity of all four drives, however, with no redundancy built in.Table-2 below provides general comparison of different RAID types for cost, capacity, performance and availability.

Designing for Maximum Performance

    Performance can be enhanced at multiple levels of Sun Storage arrays. However, with bare minimum configurations, there are two most common components which can have a higher impact. Controller Cache and disk drives. Depending on the type of I/O, one or the other or both may need to be optimized. For heavy sequential Reads, larger cache is recommended with cache pre-fetch enabled for optimal performance. The cache pre-fetching feature allows intelligent data block placement into cache prior to an I/O request based on past read requests, hence enhancing read performance. The higher the amount of cache, the more data can be pre-fetched into cache at a given time. In addition to larger cache, if high performance storage I/O is a key requirement for a given configuration, Sun Storage arrays can be configured with high performance SAS or Fibre Channel drives. For capacity and other SAS and Fibre Channel drive specifications, refer to www.sun.com/storage.

Volume Layout for VMware

    For optimal VMware volume layout configuration on a Sun Storage bare minimum configuration, it is recommended that only VMFS volumes be used. VMFS takes complete advantage of all VMware features including VMware level snapshots and Storage VMotion which would be an ideal tool for data migration if and when additional storage arrays are added to the bare minimum layout. Following recommendations should be followed if a bare minimum Sun Storage array is being configured for VMware:

• Follow Virtual Machine sizing recommendations outlined in this document


• Create a volume or a lun on the array equal to, or larger than, the sizing calculation for Virtual Machine home directories and use VMware to create VMFS filesystem on this volume or lun. This space will be used to create Virtual Machines and their home directories


• If needed and necessary, create a volume or lun for ISO images which can be shared with ESX servers to install operating systems on Virtual Machines, applications and updates without the need for a cdrom drive on the servers and having to physically replace CDs as needed.


• Create subsequent volumes or luns for user/application data as necessary for each Virtual Machine, ideally two to three volumes or luns based on application/data type.

STAY TUNED FOR PART III....!!!!!!!!!!!!!!!

Introduction

This document discusses consideration for accurately sizing the Storage Infrastructure for VMware virtualized environments and takes a more pragmatic approach to the need as opposed to common generalities. VMware is a multifaceted product with options and add-ons which have a direct impact on the amount of raw storage space required for a successful implementation. Without proper planning, there is room for errors and potential lack of scalability and support for some of these features. The purpose of the document is to provide architectural guidelines to implement a VMware configuration from the very basic storage array, such as the iSCSI Sun Storage 2510^TM, to modular midrange and high Sun Storage arrays. Guidelines will be based on general VMware storage capabilities along with advanced features such as VMware VMotion^TM, VMware HA^TM, VMware DRS^TM, VMware Fault Tolerance^TM and Disaster recovery. All of these features have an impact on how a storage array or a SAN, both IP based and FC based, needs to be implemented in order to achieve optimal availability, reliability and performance. Detailed best practices for such implementations will also be discussed. Application storage considerations are not included in this white paper.

General Storage Sizing Considerations

There are four main considerations to keep in mind for storage sizing:
1. Basic Space Requirement
2. Shared Storage
3. Storage VMotion
4. Disaster Recovery

Basic Space Requirement

VMFS Space Overhead

An additional amount of disk space is used up by VMware to lay down the VMFS filesystem on any given lun. It is not a set percentage of available space to be formatted, however, an algorithm based on the space required to save metadata. Currently, there isn't a specific calculation to determine what the exact overhead for VMFS is, however, as a general rule about 500MB of overhead should be considered as space used for metadata when creating a VMFS volume.

Virtual Machine Home Directory

VMware has specific requirements to calculate basic Virtual Machine Home Directory space. A Virtual Machine Home Directory includes

• Virtual Machine Operating System


• Virtual Memory for the Virtual Machine; In the form of a flat file


• Log files


• Virtual Machine Config file (vmx)


• Swap file (vswp)


• Snapshots (If any)

To determine the basic VM home directory space requirement, following questions need to be answered

• # of VM: Number of virtual machines being consolidated to


• Size of OS: OS + Swap + Patches


• “Size of OS” x 2: Account for snapshot space VMware creates when a VM is set to “Suspend” mode


• Memory: Virtual Memory for each VM


• Log Size: VM Status, error logs

Once all of the required information has been gathered, the following calculation can be used to determine space required:

Example:

This calculation lays the foundation for a very basic VMware storage sizing. Additionally, for configurations which may include snapshots and virtual machine templates it is recommended to reserve 15% of space for snapshots, another 15% for possible Virtual Machine Templates and 10% to 15% of free space. Example continuing from above: