Brian Doherty's Weblog

Tuesday December 06, 2005

Don't judge the UltraSPARC T1 book by its cover

At first glance, the new Sun Fire T1000 and Sun Fire T2000 systems look like your typical one, two, or four way, rack and stack, 1U/2U servers. Knowing that there's only one UltraSPARC T1 chip in these systems somewhat reinforces the notion that this is a small system. But don't let the physical size of this system or the chip-count fool you! From the perspective of your Java application, this is a pretty big system! The UltraSPARC T1 chip has up to 8 cores, with 4 hardware threads per core, for a total of 32 hardware threads. The Java Virtual Machine and your Java application see this as a system with 32 CPUs.

There are many applications that will dynamically create threads and other objects based on the number of CPUs available in the system. The JVM itself creates a pool of parallel garbage collection threads based on the number of available CPUs. This can result in some surprises, particularly for Java applications running in a 32-bit JVM that are already pushing the limits of the 32-bit virtual address space on systems with smaller numbers of CPUs.

When moving from a system with 4 hardware threads to 32 hardware threads, each additional application thread will result in additional reserved virtual address space for the thread's stack. With JDK 5.0 and the HotSpot JVM, the default options will result in a net increase of 28 parallel garbage collection threads, each with 512KB of reserved virtual memory for their thread stack, for a total of 14MB of additional reserved virtual memory. If the Java application is also sizing thread pools based on available CPUs, 512KB of virtual memory memory will be reserved for each of those thread stacks. If you are well below the 4GB virtual address space limit of a 32-bit process, then this additional virtual memory probably won't matter. However, if you are pushing the limits of the 32-bit virtual memory address space, the additional memory reservations may push you over the top, or get you uncomfortably close to it, and could result in out of memory errors.

Fortunately, there are some techniques to address this issue.

• Reduce the size of the Java heap.
  Reducing the maximum size of the Java heap (-Xmx) will allow some of the freed address space to be used for other purposes. This is typically not a viable solution, though, as the additional CPU resources on the UltraSPARC T1 may actually require a larger Java heap in order to handle the increased application throughput.
• Reduce the default thread stack size.
  The default thread stack size can be changed using the -Xss option. Selecting a value such as 256k or 128k will reduce the amount of virtual memory reserved for each Java thread's stack. For applications that are highly recursive or otherwise require deep thread stacks, this too may not be a viable solution.
• Limit the number of dynamically created threads.
  The number of parallel garbage collection threads used by the HotSpot JVM can be set explicitly with the -XX:+ParallelGCThreads=N option. Java applications that dynamically create threads typically have options to override the default size of their thread pools. Reducing the sizes of thread pools might also reduce the throughput of the operations that these thread pools handle. Reducing the size of the parallel gc thread pool is typically a good idea on UltraSPARC T1 systems. Two parallel gc threads per core is a good starting point.
• Partition work into multiple processes.
  Many applications can partition their workload across multiple processes. When applying this approach, it's generally wise to also limit the number of threads created in each process as mentioned above. It might also be advantageous to run these processes in their own processor set or under the control of the Solaris Resource Manager.
• Partition work into multiple zones.
  This approach is similar to the 'partition work into multiple processes' approach. However, some applications can only partition a workload across processes running on different operating system instances. The Solaris Zones feature will allow this approach to work by providing different operating systems instances on one physical system.
• Use a 64-bit JVM.
  All these process address space issues can be completely avoided by adding the -d64 option to the command line. If the application has any native libraries, those libraries will need to be compiled for the 64-bit data model to make use of this option. Some applications will see increased latencies with a 64-bit JVM (due to larger data sizes and cache effects). However, by increasing the size of the Java heap, many applications will also see increased throughput. Many system administrators will find the single 64-bit JVM easier to manage than the partitioned alternatives.

If your application has never run on a 32 CPU machine, it may have it's own scalability problems. Fortunately, the HotSpot JVM has been scaling to larger numbers of CPUs and large Java heaps for years. If you find that your application doesn't scale well, try running it with JDK 1.5.0_06 with the -XX:+UseBiasedLocking option, which can help improve the performance of applications with significant amounts of uncontended locking. If you are having a specific problem with your Java technology application on the UltraSPARC T1 based platforms, please contact us at the Java Performance Community Website

The new UltraSPARC T1 systems are exciting products for Java applications that need throughput. Many existing Java technology server applications will scale up nicely on these systems with minimal tuning when running with JDK 1.5.0_06 and the HotSpot JVM. With a little tuning, you can get high levels of throughput with these systems and you can do so with significant savings in power consumption, cooling, and floor space.

Benchmark and power consumption results are available here.

[ T: http://technorati.com/tag/NiagaraCMT ]