Server Virtualization - LDOMs
Monday Oct 13, 2008
With the introduction of Chip Multi-Threading (CMT) in the SPARC Processor Family, a new sun4v based architecture was also introduced.
This
sun4v interface allows the Operating System to communicate with the
hardware via a layer called the Hypervisor. The Hypervisor provides a
Hardware Abstraction to the Operating System. The Hypervisor itself is
not an Operating System and is delivered with the platform bundled with
the Firmware. Now it should be possible to carve out different groups
of actual Hardware components and present it to the Operating System.
This
combination of the Hypervisor and sun4v based Operating System are the
key enablers for LDOMs. LDOMs is supported on all UltraSPARC T1 and
UltraSPARC T2 based system. There are some nice documents
about LDOMs including discussion forums that you can join or post your questions.
LDOMs Concept
A UltraSPARC T1 processor is equipped with up to 8 cores, with 4 Hardware Threads (Strands) per core. Each Hardware Thread is seen as a CPU by the Operating System. A UltraSPARC T2 Processor is also equipped with up to 8 cores per chip with 8 Hardware Threads per core.
When creating domains, CPUs are allocated to a domain. A CPU allocated to one domain cannot be shared with another domain. Similarly when memory is allocated to a domain, the same memory cannot be allocated to another domain. Hence CPU and memory are partitioned across domains. However, the IO devices like network cards or disks can be shared. When sharing disks, a single slice of a disk cannot be shared with multiple domains, however different slices of a disk can be allocated to different domains. It is also possible to create large files on a mounted filesystem and make a file available to a domain as disk.
UltraSPARC
T1 based T2000 have 2 PCI-e Root-Complex, UltraSPARC T2 based T5120
and T5220 have 1 PCI-e Root-Complex along with 2xOn-Chip 10Gigabit
Ethernet, UltraSPARC T2 Plus based T5140 and T5240 also have 2 PCI-e
Root-Complex, and T5440 has 4 PCI-e Root Complex. It is possible to
allocate a Root-Complex to a Guest Domain so that the Guest has direct
access to the devices connecting to the Root-Complex.
LDOMs Components
- Primary
Domain - This is default or the first domain that is available with a
new system. Initially all system resources remain allocated to this
domain. This is the only domain that can be used to configure other
domains. This Domain is sometimes referred as Control Domain.
- Service
Domain - A domain that provides disk and network services to other
domains. For example, if a Guest Domain makes a Disk Image stored in
its filesystem available for booting another domain, then it can be
called a Service Domain
- IO Domain - A domain that owns physical IO devices. When such domain shares its devices with another domain , it can also be terms as Service Domain
- Guest Domain - A domain that depends on any of the above three domains for its IO services.
- Virtual Disk Client (vdc) - A device driver component active in Guest Domain to provide disk view to the domain
- Virtual Disk Server (vds) - A device driver component active in Service Domain, that is responsible for the physical IO after receiving requests from the vdc.
- Virtual Network Client (vnet) - Similar to vdc above, but provide Virtual NIC service to the Guest
- Virtual Network Switch (vsw) - A switch implementation that communicates with vnet on one side and and with the NIC device-driver on the other side.
- Virtual Console Concentrator (vcc) - Provide Console access to a Guest Domain
- MAU - These are the On-Chip Cryptographic Co-Processors. There is 1 MAU per core.
Steps for Creating a Domain
- Some CPU and Memory resources from the Primary Domain must be removed so that it can be allocated to other domains
- A vcc instance need to be created in the Primary Domain
- A vsw and vds (Virtual Disk Server Device) instance need to be created
- At this time a Guest Domain can be created
- It should be assigned a Console Port (vcc)
- Its vdc should be associated with a Virtual Disk Service
- Its vnet should be associated with a vsw
Tony Shoumack wrote a nice blueprint to provide detailed help with domain creation using LDOMs.
The
per core FPU of UltraSPARC T2 and UltraSPARC T2 Plus are just
functional units of the core. When a Domain need to execute Floating
Point instruction, the core associated with the Domain takes care of it.
If
the Domain need to accelerate Cryptographic Operations by offloading it
to the On-Chip Cryptographic Co-Processor, then, MAUs need to be
assigned to the domain.
In the next section, I will cover how to allocate devices and CPU to get the best performance.
|
|
|
|
