SPARC Enterprise M-class Servers The Secrets of Olympus

I see on Sun's Software Download Center that XCP 1070 firmware is now available for download. XCP is the firmware that runs on the Sun SPARC Enterprise M-class service processor. According to the Sun SPARC Enterprise M8000/M9000 Servers Product Notes, the one major new feature in XCP 1070 is:

In XCP Version 1070, the following new feature is introduced:
• Support for SPARC64® VII processors

The product notes refer to Solaris 5/08 (also available for download on the Software Download Center) for SPARC64-VII support.

One interesting limitation noted in the product notes is:

For Solaris domains that include SPARC64 VII processors, a single
domain of 256 threads or more might hang for an extended period of
time under certain unusual situations. Upon recovery, the uptime
command will show extremely high load averages.

The above limitation references Change Request CR6619224 Tick accounting needs to be made scalable, which describes the problem in detail:

Solaris performs some accounting and bookkeeping activities every
clock tick. To do this, a cyclic timer is created to go off every
clock tick and call a clock handler (clock()). This handler performs,
among other things, tick accounting for active threads.

Every tick, the tick accounting code in clock() goes around all the
active CPUs in the system, determines if any user thread is running
on a CPU and charges it with one tick. This is used to measure the
number of ticks a user thread is using of CPU time. This also goes
towards the time quantum used by a thread. Dispatching decisions are
made using this. Finally, the LWP interval timers (virtual and profiling
timers) are processed every tick, if they have been set.

As the number of CPUs increases, the tick accounting loop gets larger.
Since only one CPU is engaged in doing this, this is also single-threaded.
This makes tick accounting not scalable. On a busy system with many CPUs,
the tick accounting loop alone can often take more than a tick to process
if the locks it needs to acquire are busy. This causes the invocations of
the clock() handler to drift in time. Consequently, the lbolt drifts. So,
any timing based on the lbolt becomes inaccurate. Any computations based
on the lbolt (such as load averages) also get skewed.

The issue of tick scalability has been around for a while. Eric Saxe mentions the issue in his May 21, 2007 blog entry tick, tick, tick.... The change request does say that the problem is already fixed in Solaris Nevada (OpenSolaris) build 81, so hopefully this limitation will be removed with an upcoming patch or release of Solaris 10.

In my last post OpenSolaris support for Ikkaku: 2U Single-CPU SPARC Enterprise Server some people have noticed the line:

        Ikkaku is a 2U, single CPU version of SPARC Enterprise
        M-series (sun4u) server utilizing the SPARC64-VII
        (Jupiter) processor.

I've been asked by a few people about the new Jupiter processor for the SPARC Enterprise M-Class servers. Of course, I can't divulge any Sun or Fujitsu proprietary information. But anyone can go to OpenSolaris.org and learn quite a bit...

Cores and Strands

The Jupiter CPU was announced even before the the SPARC-Enterprise servers began shipping in April 2006. The Jupiter processor's product name is going to be SPARC64-VII. I did a search of "sparc64-vii" and got a hit for FWARC 2007/411 Jupiter Device binding update, which tells us:

        Jupiter CPU is a 4 core variant of the current shipping
        Olympus-C CPU, which has 2 cores.  Both Olympus-C and
        Jupiter has 2 CPU strands per CPU core.

According to the Sun SPARC Enterprise Server Family Architecture white paper (published by Sun in April 2007), current SPARC Enterprise M-Class servers will be upgradable to the new Jupiter CPU modules. Among other things, this means that a fully-loaded, SPARC-Enterprise M9000-64 with 64 CPU chips would have 256 cores, capable of running 512 concurrent threads in a single Solaris image. With 512 DIMMs, and assuming 4GB DIMMs are available soon, the system would max-out at 2TB of RAM, enough to keep 512 threads pretty happy.

Shared Contexts

Another search uncovered this email notification that OpenSolaris now supports Shared Contexts for SPARC64-VII. Steve Sistare's Blog, describes Shared Context for the UltraSPARC T2 processor:

In previous SPARC implementations, even when processes share physical memory, they still have private translations from process virtual addresses to shared physical addresses, so the processes compete for space in the TLB. Using the shared context feature, processes can use each other's translations that are cached in the TLB, as long as the shared memory is mapped at the same virtual address in each process. This is done safely - the Solaris VM system manages private and shared context identifiers, assigns them to processes and process sharing groups, and programs hardware context registers at thread context switch time. The hardware allows sharing only amongst processes that have the same shared context identifier. In addition, the Solaris VM system arranges that shared translations are backed by a shared TSB, which is accessed via HWTW, further boosting efficiency. Processes that map the same ISM/DISM segments and have the same executable image share translations in this manner, for both the shared memory and for the main text segment.

Presumably, Shared Context on Jupiter is similar if not identical.

Integer Multiply Add Instruction

If you go into the OpenSolaris source browser and search for "Jupiter" in the kernel source, you get about a half dozen C file hits. One of the files, opl_olympus.c, includes the following code, and more importantly, comments:

        101 /*
        102  * Set to 1 if booted with all Jupiter cpus (all-Jupiter features enabled).
        103  */
        104 int cpu_alljupiter = 0;
        ...
        284 /*
        285  * Enable features for Jupiter-only domains.
        286  */
        287 void
        288 cpu_fix_alljupiter(void)
        289 {
        290     if (!prom_SPARC64VII_support_enabled()) {
        291         /*
        292          * Do not enable all-Jupiter features and do not turn on
        293          * the cpu_alljupiter flag.
        294          */
        295         return;
        296     }
        297 
        298     cpu_alljupiter = 1;
        299 
        300     /*
        301      * Enable ima hwcap for Jupiter-only domains.  DR will prevent
        302      * addition of Olympus-C to all-Jupiter domains to preserve ima
        303      * hwcap semantics.
        304      */
        305     cpu_hwcap_flags |= AV_SPARC_IMA;
        306 }

The comments in the above snippets imply that a kernel can either be in "all-Jupiter" mode, or "not all-Jupiter" mode. From the Sun SPARC Enterprise M4000/M5000/M8000/M9000 Servers Administration Guide we get the following description of the two modes:

A SPARC Enterprise M4000/M5000/M8000/M9000 server domain runs in one of the following CPU operational modes:
• SPARC64 VI Compatible Mode - All processors in the domain - which can be SPARC64 VI processors, SPARC64 VII processors, or any combination of them - behave like and are treated by the OS as SPARC64 VI processors. The new capabilities of SPARC64 VII processors are not available in this mode.
• SPARC64 VII Enhanced Mode - All boards in the domain must contain only SPARC64 VII processors. In this mode, the server utilizes the new features of these processors.

Based on the source code, the main difference with SPARC64 VII Enhanced Mode appears to be the addition of the AV_SPARC_IMA hardware capability, which is an Integer Multiply-Add (IMA) instruction (see also CR6591339).

Integer Multiple-Add is important to cryptographic algorithms, and presumably Jupiter's ima instruction is similar to the xma instruction of the Itanium and other processors. The xma instruction takes three operands A, B and C, and produces the result A*B+C in a single instruction. Integer Multiply-Add instructions have a significant impact on RSA key generation and other cryptographic algorithms.

Summary

In summary, what I've learned by searching Sun's own web page:

• Jupiter has four cores, two threads per core.
• It supports Shared Context, for improved performance on applications that use a lot of shared memory.
• It supports a new Integer Multiple Add instruction to improve crptographic algorithms.
• It can be installed on SPARC Enterprise M-class servers.

I haven't seen anything about improved clock speeds, but all in all, a pretty significant improvement to what is already a great product.

That's about all I can say. We'll all have to keep waiting, and watching, for more Jupiter info as Sun decides to release it.