Martin's Blog

With the second incarnation of the true CMT systems just being announced I would like to take the chance to propose a deployment of Niagara 2 based systems that adresses security as well as high performance. The T5x20 systems both use the Niagara 2, this CPU can be viewed as a true system-on-chip:

• Eight cores, each providing eight hardware threads (and a lot more), these translate to the CPUs in a conventional SMP system. Each core carries a floating point unit and crypto unit that is able to do symmetrical block and asymmetrical (public key) cipher algorithms. All cores are connected to a crossbar to communicate to each other and the other components.
• Eight banks of level two cache, which translate to main memory in a SMP. These banks of cache on the one end are connected to the same crossbar that was mentioned above and on the other end to (chip-)external main memory.
• a x8 PCI-E root complex and
• a "network interface unit" (NIU) providing two 10Gbit ports.

One powerful use case for this Architecture is high performance Webservers that require integrated security. Potential attacks on the server can be reduced such as

• Exploiting (undiscovered) security flaws
• Denial of service attacks by willingly or accidently overloading it

The whole scenario to be described below needs a T5120 or T5220, with any number of cores and a decent amount of memory. The memory amount is of course governed by the applications to be run, but we will at least deploy three logical domains, so one should have 16GB of RAM in the system.
Logical domains (LDOMs) are a partitioning or hardware virtualization technology of sun4v based systems. Up to now these are systems based on Niagara and Niagara2 CPU (their official name being Sun UltraSPARC-T1 and -T2 CPU). The LDoms are implemented by an hypervisor running on the CPU governing the access to the physical hardware. The partitioning is realized by grouping physical resources into guests, these guests are the above mentioned LDoms. The resources that can be distributed among the LDoms are the 64 hardware threads, the main memory, the PCIe root complex, and the NIU. One distinguishes three main kinds of LDoms:

• Control Domain, the only domain that can change the hyepervisor configuration
• Service or I/O Domains have access to physical I/O devices, and provide I/O services to other guests
• "usual" Logical Domains have only CPU resources and memory physically assigned, all I/O is virtual via the already mentioned I/O domains

The picture (click it to enlarge) gives an overview of the configuration, from left to right one has

• The control domain which is also a service domain delivering the boot devices as virtual disks to all other LDoms in the system. It runs a virtual switch private to the LDoms inside the system with access to the outside world. The control domain runs the virtual switch but does not have access to the virtual network the switch provides. All administration is done through this domain.
• A regular logical domain in the middle, which is meant to host the application. There may be more logical domains of that kind, i.e. a multi-part application or test environments.
• A frontend domain that is the central idea of the whole proposal: The on-chip NIU is assigned to this LDom, and all external traffic is handled by the 10Gbit interfaces. The frontend domain routes or "firewalls" the external traffic to the application domains from above.

The frontend domain shields the system from the traffic from the outside, the on-chip, per-core crypto units can be used for a simple webserver terminating SSL connections there and serving static content. The physical assignment of the NIU protects the hypervisor from denial-of-attacks which could severely impact the hypervisor if one chooses a "classical" LDom deployment:
In a classical deployment a service domain transports incoming traffic through the hypervisor to the LDom the traffic is meant for. If the incoming interface is a 10Gbit interface that is hit by a denial-of-service attack the hypervisor could end up in only handling the malicious traffic and the traffic would impose a severe load on the service domain running the virtual switch infrastructure driving the incoming interface.
The frontend domain will need quite a few cores, although that of course depends on the load on the external interfaces.