Tuesday June 24, 2008

I received a phone call yesterday morning from a firm claiming to be preparing a plaque recognizing Sun's selection as one of the 20 best large companies to work for in Massachusetts by the Boston Business Journal. They wanted to send a mock-up of the plaque to me for my approval. A weird request to make of a random engineer in a 30K+ person company. Figuring this was some sort of headhunter scam to extract additional information from me, I asked how they had found my name and phone number. Whereupon she asked, "Aren't you the CEO?" I responded by giving her Sun's main phone number in Burlington and suggested she call there for help.

I had a good laugh about this with Eric, the engineer in the office next to mine. As it happens, Eric's office is across from our mailstop. While talking with him, I was idly sorting through my mail when I came across a piece with this mailing label:

The sender of this letter has nothing whatsoever to do with the firm that had just called me about the plaque. Oh boy. I have no idea how this happened and I can't imagine what kinds of mailings, phone calls, and invitations I'll now receive as a result. Jonathan, see you in Davos? :-)

Monday June 23, 2008

The latest early access version of Sun HPC ClusterTools -- Sun's MPI library -- has just been made available for download here. As an active member of the Open MPI community, we continue to build our MPI offering on the Open MPI code base, making pre-compiled libraries freely available and offering a paid support option for interested customers. Wondering why we would base our MPI implementation on Open MPI? Read this.

What is particularly cool about CT 8 is that in addition to supporting Solaris, we've added Sun support for Linux (RHEL 4 & 5 and SLES 9 & 10), including use of both the Sun Studio compilers and tools and GNU C. We've also included a DTrace provider for enhanced MPI observability under Solaris as well as additional performance analysis capabilities and a number of other enhancements that are all detailed on the Early Access webpage.

Los Alamos National Laboratory and IBM recently announced they had broken the PetaFLOP barrier with a LINPACK run on the Roadrunner supercomputer. The Open MPI community, including Sun Microsystems, was proud to have played a role in this HPC milestone. As described by Brad Benton, member of the Roadrunner team, the 1.026 PetaFLOP/s LINPACK run was achieved using an early, unmodified snapshot of Open MPI v1.3 as the messaging layer that tied together Roadrunner's 3000+ AMD-powered nodes. For more details on specific MPI tunables used, read this subsequent message from Brad and this follow-up message from Jeff Squyres, Open MPI contributor from Cisco.

About two years ago, we decided to change Sun's MPI strategy from one of continuing to develop our own proprietary implementation of MPI to instead joining a community-based effort to create a scalable, high-performance, and portable implementation of MPI. We joined the Open MPI community because we felt (and still feel) strongly that combining forces with other vendors and other organizations is the most effective path to creating the middleware infrastructure needed to support the needs of the HPC community into the future.

Sun was the 2nd commercial member to join the Open MPI effort, which at the time consisted of a small handful of research and academic organizations. Two years later, the community looks like this:

This mix of academic/research members and commercial members brings together into one community a focus on quality, stability and customer requirements on the one hand, with a passion for research and innovation on the other. Of course, it does also create some challenges as the community works to achieve an appropriate balance between these sometimes opposing forces, but the results to date have been impressive, as witnessed by the use of Open MPI to set a new LINPACK world record on the biggest supercomputer in the world.

Sunday June 22, 2008

[WARNING: This blog entry is primarily about vomit.]

I flew home from Dresden via Frankfurt on Friday, boarding LH 422 for the eight-hour flight to Boston. Sitting just forward of me was a coed group of boisterous college-aged kids who pretty much quieted down once they had stowed their gear and found their seats. With the exception of Oscar, who was sitting two seats in front of my across the aisle.

Oscar's behavior went beyond boisterous, well into the realm of obnoxious. He was loud, he was rude, he was up out of his seat repeatedly, unable to sit still. I'm a seasoned air traveler and very used to ignoring the various mid-behaviors of my fellow passengers, but there was something about Oscar I found particularly grating. He persisted in this up until the first meal service, at which point some god out of some pantheon smote him but good and he threw up all over himself in spectacular fashion. When he stood up after being directed aft by an attendant, I saw that he was literally covered in vomit--all over his shirt and down his pant legs and in considerable quantity. And there was apparently enough left over to have covered the seat as well, since the attendant later placed a pillow on it to make the seat usable again.

Peace reigned for most of an hour while Oscar presumably cleaned himself up. He then reappeared--shirtless. Perhaps a little quieter, but still with plenty of swagger. Which I must say I viewed with some amusement since his cool demeanor did not jive with the fact that from the rear one could see that the entire crotch of his pants was completely packed with now-drying vomit which he had apparently missed in the clean-up effort. Ah, I thought to myself. This was schadenfreude.

Eventually one his of traveling companions gave Oscar a t-shirt and he fell asleep sitting on his vomit-laden pillow for most of the remainder of the flight. Later, mention of a $120 bar bill lead me to conclude that Oscar had had far too much to drink prior to boarding the flight.

Thursday June 19, 2008

Here is a quick look under the covers of the new Sun Datacenter Switch 3x24, the new InfiniBand switch just announced by Sun at ISC 2008 in Dresden. First some photos and then an explanation of how this switch is used as a Sun Constellation System component to build clusters with up to 288 nodes.

First, the photos:

Nano Magnum's three heat sinks sit atop Mellanox 24-port InfiniBand 4x switch chips. The purple object is an air plenum that guides air past the sinks from the rear of the unit.

Looking down on the Nano, you can see the three heat sinks that cover the switch chips and the InfiniBand connectors along the bottom of the photo. The unit has two rows of twelve connectors with the bottom row somewhat visible under the top row in this photo.

The Nano Magnum is in the foreground. The unit sitting on top of Nano's rear deck for display purposes is an InfiniBand NEM. See text for more information.

You might assume NanoMagnum is either a simple 24-port InfiniBand switch or, if you know that each connector actually carries three separate InfiniBand 4X connections, a simple 72-port switch. In fact, it is neither. NanoMagnum is a core switch and none of the three InfiniBand switch chips is connected to the others. Since it isn't intuitive how a box containing three unconnected switch chips can be used to create single, fully-connected clusters, let's look in detail at how this is done. I've created two diagrams that I hope will make the wiring configurations clear.

Before getting into cluster details, I should explain that a NEM, or Network Express Module, is an assembly that plugs into the back of each of the four shelves in a Sun Blade 6048 chassis. In the case of an InfiniBand NEM, it contains the InfiniBand HCA logic needed for each blade as well as two InfiniBand leaf switch elements that are used to tie the shelves into cluster configurations. You can see a photo of a NEM above.

The first diagram (below) illustrates how any blade in a shelf can reach any blade in any other shelf connected to a NanoMagnum switch. There are a few important points to note. First, all three switch chips in the NanoMagnum are connected to every switch port, which means that regardless of which switch chip your signal enters, it can be routed to any other port in the switch. Second, you will notice that only one switch chip in the NEM is being used. The second is used only for creating redundant configurations and the cool thing about that is that from an incremental cost perspective, one need only buy additional cables and additional switches--the leaf switching elements are already included in the configuration.

If the above convinced you that any blade can reach any other blade connected to the same switch, the rest is easy. The diagram below shows the components and connections needed to build a 288-node Sun Constellation System using four NanoMagnums.

Clusters of smaller size can be built in a similar way, as can clusters that are over-subscribed (i.e. not non-blocking.)

Wednesday June 18, 2008

Last night in Dresden at the International Supercomputing Conference (ISC 2008), Sun unveiled Hercules, our newest Sun Constellation System blade module. Officially named the Sun Blade X6450 Server Module, Hercules is a four-socket, quad-core blade with Xeon 7000 series processors (Tigerton) that fits into the Sun Blade 6048 Chassis, the computational heart of Sun's Constellation System architecture for HPC. According to Lisa Robinson Schoeller, Blade Product Line Manager, the most notable features of Hercules are its 50% increase in DIMM slots per socket (six instead of the usual four), the achievable compute density at the chassis level (71% increase over IBM and 50% increase over HP), and the fact that Hercules is diskless (though it does also support a 16 GB on-board CF card that could be used for local booting.) A single Constellation chassis full of these puppies delivers over 7 TeraFLOPs of peak floating-point performance.

Lisa Schoeller and Bjorn Andersson, Director for HPC, showing off Hercules, Sun's latest Intel-based Constellation blade system

Last night in Dresden at ISC 2008 Sun announced NanoMagnum, the latest addition to the Sun Constellation System architecture. Nano, more properly called the Sun Datacenter Switch 3x24, is the world's densest DDR InfiniBand core switch, with three 24-port IB switches encased within a 1 rack-unit form factor. Nano complements its big brother Magnum, the 3456-port InfiniBand switch used at TACC and elsewhere, and allows the Constellation architecture to scale down by supporting smaller clusters of C48 (Sun Blade 6048) chassis. Nano also uses the same ultra-dense cabling developed for the rest of the Constellation family components, with three 4X DDR connections carried in one 12X cable for reduced cabling complexity.

Here are some photos I took of the unveiling on the show floor at ISC.

Bjorn Andersson, Director of HPC, and Marc Hamilton, VP System Practices Americas, prepare to unveil NanoMagnum at ISC 2008

Ta da. The Sun Datacenter Switch 3x24, sitting in front of a Sun Blade 6048 chassis and on top of a scale model of TACC Ranger, the world's largest Sun Constellation system

A closeup of the new switch

Bjorn uncorks a magnum of champagne to celebrate

Lisa Robinson Schoeller (Blade Product Line Manager) and Bjorn prepare to spread good cheer

The official toast

NanoCork

My notes from IDC's HPC briefing at the ISC 2008 conference here in Dresden. The IDC folks are fast slide flippers so interesting details are missing in some cases--best I could do.

New HPC tracking methods review. HPC market update (2007, forecasts, cluster update) New IDC research findings -- HPC management software

HPC means "all technical servers" -- HPC, HPTC, technical servers, highly computational servers are all synonyms. Anything bigger than a desktop that is computationally or data intensive.

HPC growth stronger than expected. 19% CAGR over last four year. $11.5B in 2007. x86 and Linux dominate. Blades making inroads into all segments. Clusters continue to gain share. Major challenges are power, cooling, realestate, storage and data management, system management. Software continues to be a major hurdle.

Supercomputer now equals anything over $500K. Divisional $250K-$499K. Departmental is $100K-$249K. Workgroup 0-$99K. Biggest growth is in departmental space. 45% CAGR Departmental.

Total server market is about $52B. Of that business servers are $42B and HPC is about $10B.

Look at all server processors, technical versus commercial. Almost one third in HPC at this point. Virtualization has reduced processor counts in commercial. And HPC customers buy lots of processors because they (x86 called out) do a poor job of running required workloads--need more. Many-core processors amplify this.

Clusters still hard to use and manage. System management and growing cluster complexity. Power, cooling, floor space. Third party cost issues -- up to 60-65% of budget in some cases.

Software becoming #1 bottleneck. Better management software needed--new buyers require high ease of use. Parallel software issues related to multicore. Application rewrites required.

HPC vendor market share: HP 32.9%; IBM 32.9%; Dell 17.8%; Sun 4.6%. x86 now at 72% share in 2007. Linux 74% share in 2007. Watching Windows closely -- has a stable 5% share, roughly.

HPC processors are now shipping at a rate of over 3.3M/year. HPC clusters represent over 2.6M/year.

Forecast results based on these assumptions: Worldwide economic downturn will negatively impact overall IT spending, but HPC will be somewhat insulated due to its R&D nature. Commodization will continue to rule. Global petascale initiative will push technical innovation. Major growth areas will be energy, defense, security, manufacturing, and entertainment. Many-core will ignite a new growth curve as customers buy more processors, BUT this has not been factored into forecast.

5-year forecast says that major growth will be in the low to mid parts of the market with double digit growth in those segments. 9.2% overall market growth for market. Storage five-year CAGR predicted to be 11.4%.

Blades optimize on environmental factors. One and two RU rack servers will also continue to be important for their flexibility.

Overall guidance on areas of focus for vendors: work on getting more memory bandwidth to sockets, specialised processors, speed of interconnect, better data management, power and cooling. Pay attention to the mid and low end of market where most growth will occur.

Introduced a new term: "hpc management software" in recognition of fact that this area is becoming a large problem as clusters get larger and more complex and customers take an a la carte approach to systems, putting heavy burden on system administrators and user support personnel.

There then followed a presentation of the joint work done by IDC and the Council on Competitiveness. The Reflect and Reveal studies, which are well worth reading to understand the perspective of new entrants into HPC, a high growth area for the market. The reports are available for download here. This is the same material that was presented at the IDC HPC User Forum meeting in Norfolk, Virginia this past April.

New areas of research for IDC. Extreme computing -- any kind of technology (hw/sw/business models) that may define the datacenter of the future. Adding performance and price performance to quarterly views. HPC storage market sizing and forecasts, including data management research. HPC end-user quarterly view--what is actually going on at sites, including info on what benchmarks customers are using to make selection decisions. Multi-core issues, specifically application impacts. New low-end workgroup computers--expect people moving from a desktop to get their apps running in parallel and then jump in at workgroup level or perhaps higher--not clear at this point. Tracking petascale and exascale initiatives. Country-level HPC tracking -- 17 countries. Market share by application/industry by vendor. Datacenter assessment and benchmarking

HPC User Forum meetings will include OIl/gas meetings and Finance meetings to do drilldowns on technical requirements in these areas.

HPC Storage growth charts. Total $3.7B in 2006. Growth projection showed good growth and absolute market size below compute and above service.

User Forum meetings: Oct 13-14 Stuttgart; Oct 16 London; Sept 8-10 Tucson, AZ.

Tuesday June 17, 2008

My notes from Burton Smith's talk in the Cluster session at ISC 2008 in Dresden.

The Killer Micros II: The Software Strikes Back
Dr. Burton Smith , Microsoft Corporation, USA

Spent 30 years in High Performance Computing, but not his day job at Microsoft though he does some night work. His day job is parallel computing on clients -- general purpose parallel computing -- multicore.

Cluster Software is Primitive. Programming is at too low a level (C++, OpenMP, MPI.) Tools are too few and too thinly supported -- cluster market is not big enough. Applications are too cluster-specific because the infrastructure is different on various clients--islands of cluster specificity.

Big Changes are Coming. First, the many-core inflection point--parallel computing comes to the mainstream. Second, cloud and corporate computing--better data searches and access and service-based application software. SOAP, AJAX, XML. Yes, partly. But it is really about breaking applications into communicating pieces and distributing them.

At SC '89, Eugene Brooks predicted mainstream hardware would dominate the HPC area, "None will survive the attack of the killer micros!" Now it is software's turn. Meaning that HPC will soon be dominated by software from outside of HPC.

Client Many-Core Parallelism. Client computing will soon be parallel, even on mobile devices and phones. Why? More absolute performance, and more performance per watt. Parallel languages and tools are underway--they are need to use the new hardware well. And this does not mean adding parallel for-loops to C++.

Cluster versus mainstream software. SPMD parallelism with OpenMP versus mixed task/data parallelism; fixed processor counts versus variable processor counts; MPI versus Internet and MPI; C++ versus C#, F#, Excel, SQL, C++; file system versus databases, cloud, file system;

Emerging mainstream software has richer capabilities.

Software as a Service. Software services will run everywhere-- in clients, on servers, in the cloud. Which brings distributed computing to the forefront. Service-based apps are distributed with data intensity where data are and computational intensity where processors are.

Conclusions. Cluster software is still primitive because the market is so small. Two major revolutions now underway in computing that will change the landscape. Clusters will be affected by these changes.

At the Sun HPC Consortium gala dinner last night here in Dresden, Marc Hamilton, Sun's Vice President of Systems Practice Americas, recognized Sun's first four Sun Constellation System customers and presented each with a plaque. The text of the plaques is shown below along with some photos of the awards.

The last customer talk at the Sun HPC Consortium here is Dresden was given by Jostein Sundet, who is Program Director for Research Computing Services and Adjunct Professor in Atmospheric Sciences, University of Oslo. He spoke about the University's HPC requirements and infrastructure and about how their new Sun Constellation System enables both capability and capacity computing for the University and its customers.

The University of Oslo is the largest university in Norway. And they now have the largest InfiniBand switch in the world -- the Sun Datacenter Switch 3456 (a.k.a. the Magnum switch,) which is currently used to connect 96 Sun Blade 6048 systems to create a Sun Constellation System. In addition, they have seven Sun Fire x4600 8-way systems with up to 256 GB memory, 450 Sun Fire X2200 m2 nodes with 2*AMD quad core processors, 96 Dell 1425 systems, and about 1 PetaByte of storage.

The user base includes about 250 active users, including national (advanced) users and local University of Oslo users who may not be highly skilled in computing and come from a wide range of disciplines, including politics. The site also acts as a backup site for the Norwegian MET-Office's operational weather forecast and functions as a Tier1 and Tier2 facility for CERN, while also participating in the National GRID.

Operational goals of the facility are to allow easy access for all users, both the elevated top (capability) users and the deepening base of capacity users. To do this, the university has acquired their Sun Constellation System, a homogeneous, InfiniBand fat-tree based HPC cluster system and coupled it with advanced job scheduling capabilities that supports backfilling, suspend/resume/migrate, and resource pooling.

Backfilling is critical for supporting increased system utilization in a mixed-use environment. By introducing such a capability, utilization was raised from about 60% to over 80% since smaller jobs can now be scheduled ahead of larger jobs while the larger jobs are waiting for resources. This has allowed users to temporarily exceed their job quota as in the case in which a user was able to temporarily run 1263 concurrent jobs, well above their usual limit of 384 jobs.

Titan, the Sun Constellation System to be used for both capacity and capability workloads, is a 4000-core cluster built with AMD quad-core processors using the Sun Datacenter 3456 centralized switch to provide InfiniBand connectivity for the system. The parallel file system is centralized and implemented with IBM's GPFS.

Initial performance measurements demonstrated 965 MB/sec between nodes (point-to-point) and 780 MB/sec per node when performing a 96-node all-to-all MPI communication. Latency was measured at about 2 ms.

Primary challenges included some early firmware issues that were solved by Sun and connector and cable issues, an issue also mentioned by Karl Schulz in an earlier Consortium talk.

My notes from Jack Dongarra's talk in the Cluster session at ISC 2008 in Dresden.

Four (maybe 6) Important Concepts to Consider when Using Computing Clusters.
Jack Dongarra, University of Tennessee & Oak Ridge National Laboratory, USA

The concepts are:

• Effective use of Many-Core. Dynamic data driven execution, bock data layout
• Exploiting mixed precision in algorithms
• Self adapting and auto tuning of software to underlying architecture
• Fault tolerant algorithms
• Exploit hybrid architectures
• Communcation-avoiding algorithms

In the "old days" processors became faster each year. Today the clock speed is fixed or slowing. But things are still doubling every 18-24 months. Number of cores will double roughly every two years while clock speed decreases (does not increase.) Need to deal with millions of threads in a system.

TOP500 perspective. Regular, milestone increase in system peak performance every 11 years. 1 GFLOP 22 years ago (1 thread), 1 TFLOP 11 years ago (10^3 threads), 1 PFLOP (10^6 threads). Extrapolate to ExaFLOP: 10^9 threads in about 2019.

How to code for these multicore systems? Fine granularity to support high level of parallelization. Asychrony will be important as granularity becomes finer. Must rethink the design of our software. A bigger disruption than the move from shared memory to message passing. Much resistance then and likely more at this transition point.

Steps in a current LAPACK LU decomposition with partial pivoting. Most of the parallelization comes from the matrid-multiplication step. It is bulk synchronous. Scalar step, parallel, scalar, parallel...in synchrony. Very inefficient use of resources. A more event-driven, multithreaded approach is needed based on an analysis of the directed graph of the computation. Adaptive lookahead. Not a new idea, but reviving it to exploit these new architectures.

PLASMA is a redesign of LAPACK/ScaLAPACK. Asynchonry, dynamic scheduling, fine granularity, locality of reference. He shows a graph of LU performance of PLASMA against LAPACK and AMD and Intel implementations -- does better at all or most problem sizes. Same for QR. And Cholesky. Consonsiderably better performance at all problem sizes.

Performance of single precision on conventional processors. Roughly a factor of two performance over double precision looking at SGEMM/DGEMM. Less compute, less data being moved. Exploit 32-bit as much as possible. Some algorithms can use mixed precision

Intriguing potential. Automatically switch between SP and DP to match desired accuracy. Potential for GPU and FPGA -- use as little precision as you can get away with.

Performance Optimization. Embed self-tuning optimizations into algorithms so codes can adapt themselves to underlying system architecture. Past successes: ATLAS, FFTW, Spiral, Open MPI (will adapt itself to the characteristics of global communication interconnect.)

Fault Tolerance. MPI not up to the task of dealing with faults. Mismatch between hardware and programming. Two kinds of faults important: Erasures--lose a resource/processor and Errors--detect an error and recover. Much work done looking at these issues -- will become more important.

Conclusions. For last decade, research tilted in favor of hardware. Need to rebalance this -- barriers to progress are increasingly in software. Hardware has half-life of a few years, software's is decaded.

Parallelism is exploding. Perfomance will be a software problem. Locality will continue to be important. Massive parallelism required, including pipelining and overlap.

My notes from Thomas Sterling's talk in the Cluster session at ISC 2008 in Dresden.

The Idea of Clusters -- from a personal Beowulf perspective
Thomas Sterling
Louisiana State University

Where we are and how we got here, the drivers pushing commodity clusters forward, and clusters in the sunset of Moore's Law: they will still be clusters, but they will look different.

Definition of a Commodity Cluster. Distributed/parallel computing system, constructed entirely from commodity subsystems with two major subsystems (compute nodes and system area network.)

Use of Commodity Clusters Science and Engineering, Manufacturing, FInancial, Commerce, and a large role in Search Engines. And clusters dominate the TOP500 list--more than 70% of systems.

Early History of Cluster Highlights. SAGE for NORAD was essentially a cluster built in 1957. Ethernet in 1976. First NOW workstation cluster at UC Berkeley in 1993. Myrinet introduced in 1993. Beowulf in 1993. MPI standard 1994. Gordon-Bell prize for price-performance 1997...

UC Berkeley NOW Project. 32-40 SPARCStation 10 and 20 nodes. ATM interconnect and then later Myrinet. First cluster in the TOP500 list.

On the East Coast, NASA Beowulf Project. Three generations between 1994 and 1996. Wiglaf, Hrothgar, and Hyglac. 16 nodes each. Established the vision of low-cost HPC. Empowerment: Users took control and were no longer at the mercy of vendors.

Standardization of interfaces was an important driver of clustering. PCI standard. Replaced VESA and ISA. Fast and Gigabit Ethernet -- cost effective, multiple vendors, clustering able to directly leverage LAN technology and market. And then Myrinet appeared with low latency (11usec), scalable to thousands of hosts, though at a higher price point than ethernet.

Performance wasn't the best, but more scientists could get their hands on these systems. They could build it themselves and stick it in a closet. With considerable pain and effort, they could get the systems to work better. And the cost-performance was 10X better than vendor solutions.

Open Source Software, while not essential, became a motivator and driver of development of clusters. Allowed customers to build their own cluster software.

PVM was the first message-passing standard. And then came MPI though the community coming together to create a standard. It was a joining of the cluster and MPP communities at the software level--important.

More middleware was needed as clusters became more shared resources. Maui, PBS, etc, were developed as workload management systems with support for MPI. Condor for throughput computing.

Basic Principles: Performance to Cost (low hanging fruit), Flexibility (inmates are in control), and Leverage of Technology Opportunities (scum sucking bottom feeders.)

Key driver today is multi-core. All cluster nodes are now parallel computers. How we manage this is a real issue. InfiniBand is taking hold as price comes down, performance goes up. Heterogeneous accelerators like Clearspeed boards, nVIdia Tesla, AMD FireStream, etc.

There is also the potential of FPGAs. Run 10-100 times slower, but they can show exceptional speedups on certain applications.

New things that may be coming next. 3D packaging, lightweight cores, processors in or near memory (PNM), embedded heterogeneous architectures (combining PNM with streaming architectures), smarter memories (transactional memory.)

Clusters are in a Phase Change. Next phase change may be driven by clusters as we deal with model of computation, operating systems, and in programming models.

Goals of a new model of parallel computation. Address the dominant challenges: latency, overheard, starvation, resource contention, and programmability. ParalleX project held out as an exemplar of an approach that attempts to address these issues.

Clusters at Nanoscale. Clusters are forever. It took 15 years to dominance. Technology pressures will drive dramatic change -- component types, usage models, software stack, and programming methods. And classes of applications are about to go through significan change -- knowledge economy, machine intelligence, dynamic directed graphs.

Yesterday afternoon, Thomas Nau who is Head of the Infrastructure Department at the University of Ulm and a self-described Solaris geek, gave a talk titled "Storage the Solaris Way" at the Sun HPC Consortium meeting here in Dresden. The main points of his talk were an overview of the ZFS value proposition and a quick tour through cool things one can do with Solaris out of the box, for example using iSCSI and using the various network attached storage solutions available as part of Solaris.

Thomas first reminded the audience of what he and most other people in the HPC community want in a storage solution: safety and reliability, fast error detection and correction, performance, expandability, and interoperability via open standards. All of which are offered by ZFS.

With respect to safety and reliability, Thomas mentioned the following ZFS attributes:

• 256-bit checksums for everything, not just metadata
• ditto blocks to create copies of mission-critical metadata
• transactional i/o semantics using COW (copy-on-write)
• instant snapshots and clones
• exploitation of on-disk and on-array caches for performance

With respect to built-in Solaris storage options, Thomas took the audience through a whirlwind tour of Solaris network attached storage (NAS) capabilities as well as block-level access using iSCSI. He also managed to demo all of this using his laptop, which was running two virtual machines called Angelina and Brad.

As shipped, Solaris has built-in support for NFSv4, Samba, and (in OpenSolaris) CIFS. As Thomas pointed out, the Samba implementation has been modified to support ZFS as a virtual file system back end and the CIFS server has been implemented in the kernel for maximum performance advantage.

To demonstrate iSCSI, Thomas set up several storage pools and then exported them via iSCSI as normal disks from Angelina to Brad where he mounted the disks in a mirrored configuration, which was all quite easy to do. Your correspondent, however, was not fast enough to capture the details of the demo. I expect that slides will be made available on the HPC Consortium website at some point.

In terms of performance, a test done at Ulm showed that the iSCSI approach which used several small storage arrays onto 2x2 redundant x4500 servers delivered comparable performance to a previous FC-AL solution that had been used with several small storage arrays.

Bernd Dammann, Associate Professor at the Technical University of Denmark spoke yesterday afternoon at the Sun HPC Consortium meeting here in Dresden. His talk focused on the benefits of Sun's Studio compiler and tools suite for education and HPC.

Sun Studio is an important tool for teaching students about HPC programming techniques at DTU, including data flow through cache-based systems, loop-based optimization techniques, pipelining, and general application tuning techniques. The particular programming course that Bernd described focuses on helping students understand how real computers work, how memory and CPUs are glued together -- the details they don't learn in more theoretical courses. However, once the students are exposed to these techniques they are surprised to learn that many of these techniques are applied automatically to their codes by modern compilers. Good news for ease of use, but frustrating for engineering students who typically don't like black boxes and prefer to understand internal details in such cases. Sun Studio solves this problem with its compiler commentary feature.

Compiler commentary allows the programmer to view their source code annotated with compiler-generated comments that describe in detail not just which optimizations were applied to the code, but also cases in which optimizations where not applied. Bernd showed several examples that illustrated how the feature works, including the use of the er_src utility to display source code with interleaved commentary.

The benefit, of course, of the compiler commentary is that a suitably educated programmer can use the information to find additional opportunities for performance improvement by making suitable changes to the code or by activating additional compiler features. For example, one of Bernd's examples showed the use of the -xrestrict compiler flag, which was not familiar to me. It allows the programmer to tell the compiler that pointers in the code are known not to overlap, which can potentially allow the compiler to significantly increase performance with additional optimizations. Cool.

Bernd noted that Sun's compiler commentary is "in the right place," namely in the binaries rather than in separate log files or merely displayed on a screen. By storing the commentary internally, it can be extracted and looked at later-- potentially long after compilation, which can be very useful.

Bernd then gave a brief overview of Sun's primary performance analysis tool, Sun Studio Performance Analyzer. He noted that it can display compiler commentary as well as a variety of application performance metrics (cpu usage, timelines, etc.) He praised it as a tool that both he and his students find very intuitive and easy to use.

Sun Studio is also used in a parallel programming course where it makes teaching OpenMP much easier. Being able to look at performance timelines for each thread to see OpenMP overheads and using the Thread Analyzer tool to detect data races were two examples. He also liked Sun's extension to OpenMP that allows the compiler to perform automatic scoping, which can be very useful in dealing with large, legacy codes with hundreds of variables.

Students at DTU have been using the Sun Studio tools for four years and like it a lot. They wished they could use the tools on Linux. Well, now they can.

Bernd ended his talk with some favorable comparisons of Sun Studio C against GCC and Intel C and showed an example of how easy it was to use the Sun tools to easily parallelize and debug a large, legacy Fortran code and get good parallel performance very quickly. He conclused with the observation that Sun Studio is a world-class product that is easy to learn and use. Bernd is now on my official list of favorite customers. :-)