It seems to me that what you have documented in this paper is the existence of two very different expertise gaps, with different causes and possibly different remedies. One is expertise in the program itself, and the other is expertise in the technology and technique of performance tuning on parallel systems. There's a third gap, which is expertise in the programming tools themselves, but we can handle that pretty well by education.

Expertise in the program itself, exemplified by Don's case, is a problem seen in the maintenance of all large and complex programs. It is actually unrelated to expertise in the tools, since that is easily acquired. Theres a lot of work going into this area, but nobody actually seems to use it. Most tools for understanding programs become shelfware, a fact that should deeply worry tool developers. Of course this is made much worse by complexity added into the program in the tuning process.

I don't think this is the "expertise gap" bemoaned by the HPC community.

I really think that they are talking about the second gap, in the techniques of performance tuning. You don't demonstrate that education is not a potential solution for this problem, though it certainly hasn't been a good solution up to now. In fact, I think that we don't know how to educate people in scaling and tuning, though we have some success with apprenticeships and similar techniques. Maybe we need performance tuning workshops, similar to writers' workshops.

Reducing complexity helps both gaps, of course, but reducing the complexity of performance tuning has double leverage. It aids the complex job of performance tuning directly, and it reduces the complexity of the code itself, reducing the first gap. If programmers didn't have to worry about performance, they could all write compact, beautiful code.

I think that you need to call out both problems explicitly, and propose separate remedies for both of them. The different solutions may well be at odds with one another, too. In particular, the solution to individual program expertise is often more abstraction. The solution to tuning is often less abstraction. We need to be very explicit about what part of the problem we are attacking with what tools. I think separating the kinds of "expertise gap" might be a good start.

There really is an "expertise gap" out there in the High Performance Computing (HPC) community. One of the goals of Sun's HPCS Productivity Team has been to understand what's really going on, as opposed to what people like to complain about. Susan Squires, our group's anthropologist, would be the first to tell you that those aren't always the same thing.

Some key people in the HPC community have been telling us (in various ways) that they are constrained by the expertise needed for HPC application development, so we looked into it. We gathered several kinds of data and analyzed it using a variety of methods. It turns out that this "expertise gap" idea is right on target, and we can now say quite a bit about how it looks. It takes lots of education and years of experience for people to learn this kind of programming, and only a very few ever get really good at it. We see the problem as an inevitable consequence of the way HPC software gets developed; this will have to change if we're going to get the kind of dramatic productivity increase that DARPA is seeking with their funding of the HPCS program.

Sue Squires, Larry Votta, and I wrote up some of these conclusions for the recent Workshop on Productivity and Performance in High-End Computing (P-PHEC) in a paper we titled "Yes, There Is an 'Expertise Gap' in HPC Application Development" (download the PDF). Here's the abstract:

The High Productivity Computing Systems (HPCS) program seeks a tenfold productivity increase in High Performance Computing (HPC), where productivity is understood to be a composite of system performance, system robustness, programmability, portability, and administrative concerns. Of these, programmability is the least well understood and perceived to be the most problematic. It has been suggested that an “expertise gap” is at the heart of the problem in HPC application development. Preliminary results from research conducted by Sun Microsystems and other participants in the HPCS program confirm that such an “expertise gap” does exist and does exert a significant confounding influence on HPC application development. Further, the nature of the “expertise gap” appears not to be amenable to previously proposed solutions such as “more education” and “more people.” A productivity improvement of the scale sought by the HPCS program will require fundamental transformations in the way HPC applications are developed and maintained.

I'm reading an interesting book by Don Knuth, Things a Computer Scientist Rarely Talks About. A lot of it has nothing to do with computer science, though it is very interesting, but I found the following gem:

One of the main characteristics of a computer science mentality is the ability to jump very quickly between levels of abstraction, between a low level and a high level, almost unconciously. Another characteristic is that a computer scientist tends to be able to deal with nonuniform structures — case 1, case 2, case 3 — while a mathematician will tend to want one unifying axiom that governs an entire system. ... Experience shows that about one person in 50 has a computer scientist's way of looking at things.

That's a fascinating observation that tends to align a lot of the data about the expertise gap. It's not just expertise, so training won't really handle it. Education, in the true sense might, and apprenticeship might help, but it might just take genetics. It also fits what I was saying about levels of abstraction.

It makes you really want something like Fortress, so the library writer, who is a computer scientist, can paper over the gaps while the mathematician/physicist can think at his own level.