Sunday November 11, 2007

Greg Newby, Chief Scientist at the Arctic Region Supercomputing Center (ARSC)at the University of Alaska Fairbanks spoke today about Midnight, their new Sun-based HPC supercomputer. For the curious, the name comes from "Land of the Midnight Sun."

Midnight contains 2312 computational cores or 413 nodes, configured in 19 racks. The aggregate peak performance of its dual core AMD processors is about 12.1 TFLOPs. It was interesting to see that ARSC had opted for a mix of node sizes for Midnight. There are 55 X4600 nodes, each with 16 cores and 64 GB memory. In addition, there are 358 4-core X2200 m2 nodes, each with 16GB memory. It's also noteworthy that ARSC opted for a generous memory configuration of 4 GB per core rather than the more standard 1-2 GB/core.

The system uses a 4X SDR InfiniBand interconnect built around two Voltaire 288-port switches. The interconnect is used for both MPI communication and for connection to the Lustre cluster file system. One switch is configured with 256 attached X2200 m2 nodes while the other hosts the X4600 nodes and the balance of the X2200m2 systems.

The software stack used on Midnight includes the following components:

• PBS Pro is used to manage the X4600s and the two groups of X2200 m2 systems as three separate resource pools for scheduling purposes
• SUSE 9 is run on the computational nodes while Red Hat is used on the X4500 (Thumper) storage servers used by Lustre
• A Voltaire-provided MPI library
• A variety of compiler suites, including Pathscale, PGI, and Sun Studio

In closing, Greg shared some of his experiences with testing and accepting such a large system and ended by noting that Sun must improve its support of Linux-based systems in order to be successful with these engagements. As it happened, Peter Braam (founder of Cluster File Systems, maker of Lustre, and now a VP at Sun) was the next speaker at the Consortium. More later on what Peter said in response to Greg's comments regarding Sun's support for Linux.