Thursday June 28, 2007

Thomas Nau, self-described Solaris Geek and head of the Infrastructure Department at the University of Ulm, gave a talk this week on DTrace at the HPC Consortium meeting in Dresden. In his view, DTrace is the tool of choice for understanding system and application performance issues.

Thomas described briefly how DTrace works to support dynamic, lightweight instrumentation of both kernel and user code with some 40,000 probe points available for use within Solaris and the D scripting language to perform custom processing. He particularly likes DTrace's aggregation facilities that support gathering and condensing data for easier interpretation (for example, creating simple, text-based histograms of data value collected during a run.)

He also pointed out that, contrary to what some say, DTrace does not need to be run as root. Instead one can use Solaris RBAC (role-based access control) facility to grant particular, DTrace-specific privileges to users. These privileges are dtrace_proc, dtrace_user, and dtrace_kernel. See here for more details.

For those wanting to go beyond the simple ASCI text output and graphics created by DTrace, Thomas recommended a utility called the Chime Visualization Tool, which is available on the OpenSolaris community website.

Chime sample output.

Thomas mentioned that /usr/demo/dtrace contains lots of D scripts that can help the new user learn DTrace. But the Solaris Dynamic Tracing (DTrace) Guide available on docs.sun.com remains his favorite reference.

As they say, things are just bigger in Texas.

On Monday, several members of the staff from the Texas Advanced Computing Center (TACC) joined the HPC Consortium meeting in Dresden by remote link. The audio was unfortunately not very good, but we managed to hear most of what was said. Speakers were Jay Boisseau (TACC Director) and (I believe) Tommy Minyard (Assistant Director). If there was a third speaker, I apologize--as I said, the audio was not good.

Ranger will be a 504 TFLOPs system, built using the Sun Constellation System architecture with two ultra-dense switches and almost 4000 Sun four-socket, quad-core, nodes using close to 16000 AMD Barcelona processors. With 2GB of memory per core, there will be over a hundred Terabytes of memory total in the system along with 72 Sun "Thumper" storage systems with a total of 1.7 PBytes of raw storage. The InfiniBand interconnect is a 7-stage, non-blocking Clos network with latencies and bandwidths of approximately, 2.3us and 950 Mbytes/sec, respectively.

Ranger will run Linux and the OpenFabrics InfiniBand stack. It will use Lustre as its cluster file system and will run two MPI libraries: MVAPICH and Open MPI (the code base on which Sun's MPI for Solaris is based.) TACC will use multiple compiler suites, including Sun Studio. Sun Grid Engine will be used as Ranger's distributed resource management system.