It's been a while I haven't added a new entry to this blog, but there was a good reason: we have been working very hard on a new release of the LDoms product. This new release is now available as LDoms 1.1:

• LDoms Official Web Page

• LDoms 1.1 Documentation

• LDoms 1.1 Software Download

LDoms 1.1 Features:

• Warm and Cold Migration

  Migration is the action of moving a domain from a physical platform to another physical platform. Cold migration is moving a domain which is not active (the domain is either stopped or unbound). Warm migration is moving a domain which is active. For warm migration, the domain to migrate is suspended on the source platform, then the entire state of the domain is transfered over the network to the target platform. Finally the domain is resumed on the target platform.

• Virtual I/O Dynamic Reconfiguration (VIO DR)

  Virtual I/O Dynamic Reconfiguration allows virtual services (vsw, vds) and virtual devices (vnet, vdisk) to be dynamically added to a domain without having to reboot that domain.

• Network NIU Hybrid I/O

  Network NIU Hybrid I/O is a feature available on UltraSPARC-T2 based system which enables a virtual network interface from a guest domain to directly access a physical network interface owned by an I/O domain. That way a guest domain can have the performance of a physical network interface while using a virtual network interface.

• Network VLAN Support for Virtual Network Interface and Virtual Switch

  Network VLAN support provides VLAN 802.1Q support to virtual network interfaces and virtual switches. This enables to assign port-vlan-id and vlan-id to virtual network interfaces and virtual switches.

• Network Virtual I/O Performance Enhancement

  Performance of the virtual network interface has been improved by about 30%.

• Virtual Switch Support for Link Aggregated Interface

  A virtual switch can now be associated with an interface which is a link aggregation of some network interfaces.

• Virtual Disk Multipathing and Failover

  Virtual Disk Multipathing enables a virtual disk to be connected to its backend through different service domains, and ensures the availability of the virtual disk in case one of the service domain becomes unavailable.

• Single-Slice Disk OS Installation Support

  Single-slice disk support has been improved and Solaris can now be installed on such a disk. Also single-slice disks are now visible with the format(1m) command.

• Public and Supported XML Interface with the LDoms Manager

  The XML interface with Domain Manager is now public, fully supported and documented. The Domain Manager can now be connected using the XMPP protocol, and the XML support provides a convenient way to interface an application with the Domain Manager.

• Other Improvements and Bug Fixes

  This release also includes a lot of other improvements and bug fixes. Among other:

  • WANboot Support
  • Support for Setting Persistent WANboot keys from OBP
  • VDisk iostat Support in Guest Domains
  • VDisk EFI Label Support for Disk Image
  • VDisk Support for Disk Managed by Multipathing Software (Veritas DMP, EMC Powerpath)
  • Improved Interrupt Distribution
  • Improved Error Messages
  • Numerous Improvements and Extension to the Domain Services Infrastructure
  • Improved Console Behavior when not using Virtual Console
  • Listing of MAC Addresses Assigned to Guest Domains
  • Fine-Grained Control of Configuration Listing
  • Improved Utilization Percentage Reporting
  • Option to Explicitly Set a Domain's Hostid
  • Improved Persistence of VIO Services and VDS Volume Names
  • Enhancement Algorithm when Dynamically Removing CPUs from a Domain
  • Improved Annotations in "ldm ls-spconfig" Output
  • Improved Support for Large Fragmented Memory Configuration
  • Lot of Bug Fixes (over 100)

LDoms 1.1 Usage Examples

With this release, LDoms are now more flexible, dynamic and have enhanced virtual I/O performances. This is particularly useful when you have a system like the Sun SPARC Entreprise T5440 server where you can have up to 256 cpu threads and 512GB of memory. Here are some examples of the benefit of using some of the new features of LDoms 1.1 on such a large and powerful system.

Sun SPARC Entreprise T5440 Server Overview

On the Sun SPARC Entreprise T5440 server you can create up to 128 domains; but the configuration you need will mostly depend on the number of cpus and memory you want to use for each domain. Here are some configuration examples:

Num. of Domains	CPUs/Domain	Memory/Domain	Comments
128	2	4GB	each has two cpu threads
32	8	16GB	each domain can have an entire cpu core (1 core has 8 threads)
4	64	128GB	each domain can have an entire cpu chip (1 chip has 8 cores)
1	256	512GB	the domain has all 4 cpu chips (4 chips x 8 cores x 8 threads = 256)

Of the course the number of domains and the associated number of cpus and memory can be adjusted depending on your need.

The T5440 also has 4 PCI buses, that means that you can create up to 4 I/O domains. Such domains can have direct access to physical I/O resources, like a network card or a physical disk. So you can effectively split the T5440 into 4 fully independent systems (for example 4 domains, each with 64 cpu threads and 128GB of memory) which have their own physical disks and network interfaces and do not depend on any other domain because they don't need to use virtual I/O.

Domain Migration

With such a large number of domains (up to 128), it is important to be able to easily move a domain from one system to another system in case you need to shutdown the entire platform (for example during a maintenance), or if you have some new hardware available and you want to free some resources on an existing system (for example to allocate more cpus or memory to the existing domains). You can easily do this with the Domain Migration feature of LDoms 1.1. Migration is done with a single command:

        primary# ldm migrate domain_to_migrate system_to_migrate_to

Then the system will automatically select the appropriate type of migration depending of the state of the domain to migrate:

• if the domain is inactive (either stopped or unbound) then a cold migration is done;

• if the domain is active then a warm migration is done: the domain is first suspended, then its entire state (cpu, memory, configuration) is transfered over the network to the remote system, and finally the domain is resumed on that remote system.

Network NIU Hybrid I/O

To use network NIU hybrid I/O, you need an UltraSPARC-T2 based system, like the T5120 or the T5220, and some 10-Gigabit Ethernet XAUI Adapters. An XAUI adapter is managed by the nxge driver and is assigned to an I/O domain. It can also be associated with a virtual switch (vsw). Then the virtual network interface (vnet) of a guest domain connected to a virtual switch associated with an XAUI adapter can operate in "hybrid" mode. In such a mode, the guest will be able to receive and send network packets to the XAUI adapter without having to go through the virtual switch or the I/O domain. That way a guest domain is able to get the performance a physical network interface while using a virtual network interface.

The figure below illustrates the difference between virtual I/O and hybrid I/O by showing the path of the network packets in the different modes:

• The guest domain on the left of the figure uses virtual I/O, and the path of network packets is indicated by the black arrow.
• The guest domain on the right of the figure uses hybrid I/O, and the path of network packes is indicated by the red arrow.

Click on the image to enlarge

Configuring a virtual network interface in hybrid mode is very simple: just specify "mode=hybrid" when adding or setting a virtual network interface. For example:

        primary# ldm add-vnet mode=hybrid vnet0 primary-vsw0 ldg1

or

        primary# ldm set-vnet mode=hybrid vnet0 ldg1

Note the setting "mode=hybrid" is just an hint to the system so that it tries to use hybrid I/O. If the system is unable to use the hybrid mode (for example because the virtual switch is not associated with an XAUI adapter) then the system will automatically fallback to the legacy mode and use virtual I/O.

The T5120 and T5220 can have up to 2 XAUI adapters, and each XAUI adapter can be shared 3 times in hybrid mode. That means that you can have up 6 domains having a virtual network interface using hybrid I/O.

See Raghuram's blog for more details about Network Hybrid I/O.

Virtual Disk Failover and Multipathing

When you have a domain using virtual I/O, one concern is to keep that domain up and running even if the service domain providing virtual I/O services happens to be down. LDoms already provide a solution to that problem for virtual network I/O, by using IP Multipathing (IPMP) in the guest domain on top of two virtual network interface associated with a virtual switches from different service domains.

In addition, LDoms 1.1 provides a solution to that problem for virtual disk I/O. If multiple service domains have access to the same virtual disk backend (for example a file on a NFS server, or a shared LUN on a SAN) then a virtual disk can be associated with all these service domains and the path to access the virtual disks backend will change depending on the availability of the service domains.

Virtual disk multipathing is configured by putting the vdsdev representing the same virtual disk backend into the same multipathing group (mpgroup). This is done when using the add-vdsdev command. For example, if we have two service domains (primary and alternate), each with a vds service (primary-vds0 and alternate-vds0), and each service domain is able to access the same NFS file /home/domain/ldg1/vdisk0, then we can put that backend file into the same mpgroup "foo".

        primary# ldm add-vdsdev mpgroup=foo /home/domain/ldg1/vdisk0 vdisk0@primary-vds0
        primary# ldm add-vdsdev mpgroup=foo /home/domain/ldg1/vdisk0 vdisk0@alternate-vds0

Finally the backend file can be exported as a virtual disk to the domain ldg1:

        primary# ldm add-vdisk vdisk0 vdisk0@primary-vds0 ldg1

That way the virtual disk will be accessible in domain ldg1, primarily through the primary domain. But if the primary domain goes down then the virtual disk will remain accessible through the alternate domain. This is illustrated in the following figures:

• Initially all service domains (primary and alternate) are up, and the guest domain access the virtual disk backend through the primary domain:

  

  Click on the image to enlarge

• If the primary domain is down then the path to the virtual disk backend fails over and the backend becomes accessible through the alternate domain:

  

  Click on the image to enlarge

On the T5440, you can have up to 4 I/O domains. So you can easily make a configuration where guest domains can be resilient to the failure of the service domain by creating at least 2 I/O domains. Then you can setup guest domains so that they use IPMP on top of their virtual network interfaces, and virtual disk multipathing through the service domains you have. That way, guest domains will remain fully functional and preserve disk and network access even if a service domain is down.