waterfalls

« Server Virtualizatio... | Main | Server Virtualizatio... »

Server Virtualization - Using LDOMs on T5440

Monday Oct 13, 2008

The Sun Fire T5440 can have at most 4 UltraSPARC T2 processor. Each UltraSPARC-T2 Procesor is directly connected to ¼th of the entire system memory with 1Gigabyte memory interleaving and owns a PCIe Root-Complex. When fully populated with Processor and memory, Solaris can see 256 CPUs and 512GB of memory. That is a lot for many applications except for some large databases. With this class of system, it is not usually possible to consume the entire system with a singe instance of most applications. But that is in fact a very good opportunity to consolidate a bunch of such applications in this system using LDOMs, there-by reducing Power consumption and rack space. An example is the SugarCRM application. It is a web based application written using PHP and has a MySQL database backend. Yun Chew has written a nice blog demonstrating how to consolidate SugarCRM application on this system using LDOMs. I can think of many such applications that can be consolidated on this and T5140 and T5240 based systems.

The work done by Yun referred to above, there was no need to create any IO domains, but because T5440 has 4 PCIe Root-Complex, it is possible to create up to 4 IO domains for applications sensitive to IO performance. Such applications, like database can be run in the IO domain so that the application can have direct access to the physical disks. The other domains - like application server domains can access the database over virtual NIC. Each of the application server domains can have another virtual NIC to communicate with the external world.

The good thing about LDOMs based virtualization is that, even if the Primary Domain goes down, other domains continue to be functional. Many other virtualization technology does not have this advantage, which is why Live Migration is very critical for such virtualization technology.

To get the best performance out of a LDOMs based application deployment, it is important to understand the system topology a bit so that it becomes easier to determine what to place where. I have tried to create a sketch of the system topology below for reference.

t5440

When creating domains, IO and CPU requirement for the applications that would run in the virtualized environment should be estimated. The IO-performance of virtualized 1Gig network and virtualized disk is same as native. But compared to native-IO, virtualized-IO consumes more CPU cycles, often in the range of 5%-25%, depending on the size and frequency of the IO. Hence, when doing resource planning for LDOMs environment, couple of points should be considered to get the best performance from the T5440 LDOMs environment.

Is the application CPU intensive?

Does it scale up with additional CPUs?

Is the application Disk or Network IO intensive?

Moderately IO intensive applications would consume less than 50% of maximum IO capacity of the device

Is the application both CPU and IO intensive?
How many interrupt sources the domain would need to manage?

PCIe based Fiber Channel HBAs normally have 2 interrupt source.
PCIe based 1G network devices have either 1 or 2 interrupt sources, while 10G network devices have 8 interrupt sources
Each virtualized IO device created out of vsw or vds have 1 interrupt sources

The number of VCPUs that need to be allocated to a Domain depends largely on the ability of the application to make good use of the VCPUs. In addition to the VCPUs needed by the application, extra VCPUs should be allocated to handle interrupts. For optimal performance, when VCPUs are allocated to a domain, then they should be allocated in multiples of 4 at least, preferably in multiples of 8 where possible.

In the next section I will describe how to create IO domain with Inter-IO Domain Dependency

Like this post?