The Former Weblog of Peter Ahé

Today is my last day at Sun. This has been a tremendous experience, and you, the readers of my blog, have helped make it so. I plan to follow the OpenJDK compiler, Kitchen Sink Language, and OpenJFX projects, but I have a full time job, so my contributions will be small.

I still have a lot to say about Java technology, and I hope I will be able to say a bit about my next project. I invite you to follow me to my new place on the web.

The JavaOne schedule builder has been improved since last year, and it is now easier to resolve conflicts. This is much appreciated. However, the builder should have been written in Java or AJAX to limit the time required to reload the page after adding a session.

Anyways, I just finished building my schedule, and I found the print option to be somewhat lacking. However, I also mailed the schedule to myself and that version did not include all the headers. After printing the mail in two-pages-per-sheet mode, I noticed that I could fit it in my trusted moleskine® notebook:

JavaOne schedule in notebook.

I apologize about the quality; my real camera was stolen on my way to FOSDEM 2007. This picture is from my Razr V3.

Come to our BOF sessions at JavaOne and hear more about the Java programming language and the Java technology tools (including javac).

BOF-2899: Java Programming Language Features in JDK Release 7

Track: Java SE
Room: Gateway 102/103
Date: 08-MAY-07
Start Time: 22:00

BOF-2898: Java Programming Language, Java Compiler, and Related Tools and API

Track: Java SE; Open Source
Room: Room 105
Date: 09-MAY-07
Start Time: 19:55

I was recently interviewed by Jan for java.sun.com; where I said: Although most developers do not want to know how to use the Java Compiler API, it will provide a lot of indirect benefits. This was discussed on comp.lang.java.programmer, and I think it may have been misunderstood.

99% of Java developers will newer need to know anything about the Java Compiler API (JSR 199). A small number of Java developers will use the API directly. However, the software these few developers make is used by all Java developers. I'm talking about software such as IDEs (NetBeans, Eclipse, IntelliJ IDEA, etc), Java EE application server, build infrastructures such as Maven and Ant, and test frame works. All these pieces of software have one thing in common: they all need to invoke a compiler to translate Java source code to class files. However, 99% of all developers will not and should not care about how the compiler is invoked.

The compiler API becomes interesting for all developers when we start measuring build times in milliseconds instead of seconds (when you invoke Ant from NetBeans), or when start measuring deployment time in seconds, not minutes. These are the indirect benefits you will all notice, without learning how to use the API, as soon as your favorite tool migrates to the API.

If questions about the compiler API ever shows up on a Java certification exam, please have the examiners call me and I'll set them straight

That being said, if you are interested in compiler technology and want to hack on the compiler, please come and join us at https://openjdk.dev.java.net/compiler/.

Open sourcing of javac (and the JDK) is an opportunity to revisit our current practices and think about what is missing, what needs to be updated, and what should stay the same. One issue is how to attribute source code with @author tags and there are different policies in use in the open source community.

The position of the Apache Software Foundation (ASF) was explained by the then president of the ASF. To summarize, the ASF recommends against putting @author tags in source code for these reasons:

• Attributions in source files are not up to date.
• Software is a team effort.
• ASF sells the code base as an ASF brand.
• Shield against law-suits.

However, those concerns are not that convincing to me and are far out-weighed by the need for programmers to take responsibility for the code they write. In Pragmatic Programmer: from journeyman to master, Hunt and Thomas recommend that programmers sign their work, tip 70. They further say:

We want to see pride of ownership. I wrote this, and I stand behind my work.

The former president of ASF has a good point that @author tags should be kept up to date. I agree and think the solution is to ensure they are kept up to date with a clear policy and code reviews. For example, if you create a new class or make significant changes to its API add an @author tag. If you make a minor change make sure your name is in the project's contributor file but do not add an @author tag.

Another point made by the ASF is that software is created by a team and the strength of the brand. I agree that software is created as a team and I think that all the team members deserves to be mentioned in a file that lists all contributors. Such a file could be part of the source bundle and also be prominently linked from the website. I don't think the sense of team effort is harmed by adding @author tags to source files and I don't see how it harms the brand, after all, the copyright statements remain and prominently identify the organization.

I am not a lawyer and this is not legal advice: it is plausible that there are people in other countries that should be careful about putting their names in source files because they can be sued. However, I do not see how this affects people that does not live in those countries.

I have long followed the Emacs lisp library header conventions as described in the GNU Emacs Lisp Reference Manual. As a practical matter, I think these can be directly applied to Java source code. When the Emacs lisp convention talks about file header, I think top-level class and package comment. In other words, only package and top-level class comments should contain @author tags.

It is possible to argue that we have a unique opportunity to go and clean up our sources right now before they are released to the public. However, I'm not convinced that this opportunity has not already passed. Some parts of the JDK were open sourced even before javac and HotSpot was opened in November and the public API has been part of the JDK (src.zip) for many releases. Similarly, all the sources of the JDK have been available for download under JRL since late 2004. There are people that are very sensitive to having their @author tags removed, for example, consider how the developer exa felt after he handed over a project and discovered that the new maintainers had removed his name from the source code.

Although we have not consistently used @author tags in the JDK, or even javac, I think we should keep the ones we have already and develop a consistent policy (for example inspired by the Emacs lisp conventions). It may be a good idea to hide email addresses by obfuscating existing @author tags if they contain email addresses.

Alex and I are on stage at EclipseCon 2007 waiting to give our presentation, unfortunately this means that we are missing Scott Adams who we can hear from the conference room next to ours.

So if you're interested in hearing our talk on Java SE 7 language features, now would be a good time to head down to Santa Clara Convention Center. We start at 10:10am.

Thanks to Randall Munroe for generously sharing his comics with the rest of the world.

Forget about my wish list from last month. There are a few really good ideas on that list (closures and super packages). I gave up on properties. The rest is about making generic types easier to use. However, during my recent hiatus in Europe, I have been reflecting on generics. What is the problem with generics? Is it too complicated?

No doubt, generic types are relatively new in mainstream programming languages and wildcards are truly unique: it was literally invented for the Java™ programming language. This means that a lot of developers are not yet comfortable with generics. When I first heard about Object-Oriented Programming, I found it really scary. All that about inheritance, overriding, overloading, is-a, and has-a. Frankly, there is still a lot of confusion about something as fundamental as is-a, and has-a.

Most developers are very comfortable with OOP. In five years, most developers will be comfortable with generics.

Generic types are taking a lot of heat on various blogs. Not all of the critisism is without substance. I see that there is a problem when I talk to developers and close compiler bug reports as not being a bug. I see that there is a problem when trying to teach generics. I see that is a problem that causes otherwise diligent developers ignore a whole category of warnings about potential type unsafety in their programs.

The problem is the lack of reification. In other words, the lack of runtime type-information about generic types (at runtime, you cannot tell an ArrayList<String> from a ArrayList<Integer>). We need to do something about that. It is indeed time to erase erasure!

In the next weeks I'll explain what it buys us and how we can accomplish it.

Let me jump right into it. I propose an alternative to type inference for local variables. I'll explain why later. Everybody knows this example:

Map<String,List<Integer>> map = new HashMap<String,List<Integer>>();

I propose this solution:

Map<String,List<Integer>> map = HashMap.new();

Before I get to caught up in the details, let me examine why we are talking about this in the first place. As far as I'm aware, there are primarily two reasons for looking at examples as the above:

1. DRY—Don't repeat yourself.
2. Inspiration from scripting languages (or dynamically typed languages): JavaScript, Ruby, Smalltalk, Scheme, etc. Moving to Java from such languages can feel very constraining because you have to type so many types.

I'm currently aware of two serious proposals, the first from James:

map := new HashMap<String,List<Integer>>();

The second is from Christian:

final map = new HashMap<String,List<Integer>>();

Rémi has implemented both of these proposals. But let us not forget what is going on in other languages:

var map = new HashMap<String,List<Integer>>();

This is well known from JavaScript and has recently been added to C#. However, adding it to Java would require adding a new keyword (which is bad) or having some really strange rules. So var is not really a contender for Java.

All of these proposals (mine excluded) essentially solve the problem using the same technique: type inference. The problem is that this promotes bad behavior:

• Forces use of class types, not interface types on local variables. Generally the opposite is recommend as implementations are more flexible when they rely on interfaces instead of specific implementation classes. This is mediated by only allowing this syntax on local variables so no API is affected.
• Code could be slightly harder to read. In many cases type declarations on variables are helpful when reading some code. Most will argue that some of the really nasty examples with long type arguments are fun to neither read nor write. So eliding the type on a variable can sometimes improve readability. However, since the language can't enforce a readability rule, some developers will misuse such a feature.

This motivated Neal to suggest:

Map<String,List<Integer>> map = new HashMap<>();

Neal and I just had a chat about this. I think we both agree that it is ugly but the alternative could have compatibility problems (as well as other problems):

Map<String,List<Integer>> map = new HashMap();

So you can probably see how I got the idea:

Map<String,List<Integer>> map = HashMap.new();

However, there is more to it than just a simple syntax for constructing instances. My initial reaction to this problem was: we need more static factories throughout the JDK. For example:

Map<String,List<Integer>> map = HashMap.create();

Where the definition of create is:

public static <K,V> HashMap<K,V> create() { return new HashMap<K,V>(); }

What I propose is actually a new syntax for declaring static factory methods and that the compiler adds them by default (as it does with constructors). For example, if you declare a class:

public class Box<T> {
    public Box() {}
}

The compiler would automatically add this method:

public static <T> Box<T> new() { return new Box<T>(); }

The programmer can specify any number of new methods just as it is possible to have multiple constructors. If the programmer provides a new method with a signature that matches that of a public constructor, that will be used instead of the compiler providing one.

Thanks to Neal M Gafter for his input on this idea and this posting.

Hans Muller just convinced me that I didn't want a short hand syntax for declaring properties after all. Hope it isn't too late.

I got a few questions from a reporter on the Kitchen Sink Language. He wanted to know what prompted me to start the project and what my goal is. Here is my reply:

The Java™ programming language is defined by the JCP. This is a good thing as it ensures that the entire community is heard. We should also be conservative when selecting which features are added to the language. Otherwise, the language may become a mess. This was explained in Evolving the Java Language by Bracha, Hamilton, and Reinhold.

We want the main branch of javac to be stable and primarily focus on implementing approved features. On the other hand, we must experiment and have fun to get a feel for new language features. Most people seems prefer evaluating new language proposals by using them on their own code, rather than reading abstract specifications and proposals.

As you may know, I'm the Java compiler (javac) tech lead. I see a lot of proposal for enhancing the language and our team has to turn down most. So how can we experiment? James solution is the Kitchen Sink Language which will be a forum for trying out crazy (and not so crazy) ideas for language enhancements.

Over the winter holidays, Rémi Forax implemented two different proposals for language enhancements. This is exactly the kind of community James and I would like to encourage so we were happy to see Rémi's work. On the compiler-dev mailing list, Rémi's work prompted Ted Neward to suggest that we do something to coordinate future experimental features. So I felt this was the right time to implement James' idea.

Now it is time for me to create a vendor branch so we can start experimenting.

Dear Santa,

This year I wish that Java...

Back in May, I described how we would update the specification and implementation of @Override. Unfortunately, I messed up when integrating the changes and the specification of @Override was not updated. The compiler's behavior is changed but the documentation does not reflect that.

I will make sure to get the documentation fixed in JDK 7 as soon as possible but it is more problematic updating the documentation of JDK 6 because JSR 270 has been finalized. In the meantime, here is what it was supposed to have looked like:

Indicates that a method declaration is intended to override a method declaration in a supertype. If a method is annotated with this annotation type compilers are required to generate an error message unless either of the following conditions hold:

• The method does override or implement a method declared in a supertype.
• The method has a signature that is override-equivalent to that of any public or protected method declared in Object.

In case Bruce is still reading my blog, I just want to make it perfectly clear that Joe had absolutely nothing to do with this snafu.

Neal directed my attention to Laird Nelson's struggle with generics.

Many people have found themselves in a similar situation and I can certainly empathize. Generally speaking, the problem is the need to refer to the type of the current class. This is called self-types or the type of this. But I am getting ahead of myself, let us first examine if there are ways to solve the problem in the current language.

Design with Generic Types

Software development starts with the design phase. This is when you design your software in a small group by the whiteboard and draw informal diagrams. The boxes you draw on the whiteboard represents concepts in the application domain and will eventually result in classes and interfaces. You may also draw lines between concepts that are related. At this time you are not too concerned about how to implement the behavior but how the concepts are related.

The design phase is the ideal time to decide what type variables you need. In most cases you will need a type parameter to when a class aggregates (behaves as a collection of) other objects of varying types depending on usage. For example, consider event handlers: if you design a general event handler that can handle all sorts of events this event handler does not need to be parameterized with the type of event it handles. On the other hand, if you have multiple kinds of event handlers, one for mouse events, one for keyboard events, and one for timer events, then it may make sense to have a single event handler class that is parameterized with the event type.

Ignore how to implement the classes. If a type variable does not make sense on the whiteboard, it does not belong in your code.

It does not matter if you are new to generic types, a brilliant type theoretician, or a recovering C++ template meta-programming addict: it is too easy to use type variable for things they are not suited for!

This lesson is particularly hard to learn if you are used to C++ templates. Generic types are not templates and you should not use type parameters for implementation convenience.

Laird Nelson's Scenario

Laird Nelson describes a scenario with objects, references, and adapters. Objects can be canonically identified and references can be dereferenced. I do not know why references are needed and I would personally go for something simpler. However, I will assume that there are good reasons for having all the classes and interfaces described by Laird Nelson but note that I do not fully understand the purpose of all of them.

The goal is to be able to reference objects type-safely as illustrated by this example:

Person p = null;
Reference</* what goes here? */> ref = p.getCanonicalReference();
Person p2 = ref.dereference();

My Design

So I put all the above theory to the test and started drawing Laird Nelson's example on my whiteboard. Types like Dereferenceable and Reference are naturally parameterized to specify the type of what they reference. Similarly, a BaseObjectAdapter can be parameterized over what it adapts. I am not so sure about CanonicallyIdentified so I chose not to parameterize it. It is easy to do so if it makes sense, though. Since I do not know anything about BaseObject I do not see a need to add any parameters. Certainly neither a Party nor a Person are parameterized.

So after the design phase, it could look like this:

class DereferenceException extends Exception {}

interface Dereferenceable<T extends BaseObject> {}

class Reference<T extends BaseObject> implements Dereferenceable<T> {}

interface CanonicallyIdentified {}

interface BaseObject extends CanonicallyIdentified {}

interface Party extends BaseObject {}

interface Person extends Party {}

class BaseObjectAdapter<T extends BaseObject> implements BaseObject {}

Compare to where Laird Nelson gave up:

class DereferenceException extends Exception {}

interface Dereferenceable<T extends BaseObject> {}

class Reference<T extends BaseObject>
    implements Dereferenceable<T> {}

interface CanonicallyIdentified {}

interface BaseObject extends CanonicallyIdentified {}


interface Party extends BaseObject {}


interface Person extends Party {}


class BaseObjectAdapter<T extends BaseObject>
    implements BaseObject {}

class DereferenceException extends Exception {}

interface Dereferenceable<T extends BaseObject<T>> {}

class Reference<T extends BaseObject<T>>
   implements Dereferenceable<T> {}

interface CanonicallyIdentified /* crap. */ {}

interface BaseObject<T extends BaseObject<T>>
    extends CanonicallyIdentified<T> {}

interface Party<T extends BaseObject<T>>
    extends BaseObject<T> {}

interface Person<T extends Person<T>>
    extends Party<T> {}

class BaseObjectAdapter<T extends BaseObject<T>>
    implements BaseObject<T> {}

Implementing My Design

During the design phase implementation details are not too important. However, once the design is mature we do need to worry about how to implement it? There are a few tricks to learn but most of it is straightforward, at least Dereferenceable, and Reference are:

interface Dereferenceable<T extends BaseObject> {
    T dereference() throws DereferenceException;
}

class Reference<T extends BaseObject> implements Dereferenceable<T> {
    public T dereference() throws DereferenceException {
        return null; // or something
    }
}

CanonicallyIdentified may not seem as straightforward and what about Laird Nelson's example:

Person p = null;
Reference</* what goes here? */> ref = p.getCanonicalReference();
Person p2 = ref.dereference();

So clearly, I must parameterize CanonicallyIdentified? No. I was unsure at the whiteboard and decided not to add parameterize CanonicallyIdentified and it is not used in the example anyway. I will use a wildcard:

interface CanonicallyIdentified {
    Reference<?> getCanonicalReference();
}

On the other hand, had I decided (at the whiteboard) that it did make sense to parameterize CanonicallyIdentified it could look like this:

interface CanonicallyIdentified<T> {
    Reference<? extends T> getCanonicalReference();
}

This decision is based on the the design of the object hierarchies, not on where types flow. BaseObject looks like this:

interface BaseObject extends CanonicallyIdentified {}

If I chose to parameterize CanonicallyIdentified, it would look like this:

interface BaseObject extends CanonicallyIdentified<BaseObject> {}

BaseObject is not parameterized and should not be.

Party and Person are not parameterized either but we do want to know the type of the canonical reference. The answer is to override the method and specialize the return type (covariant return types):

interface Party extends BaseObject {
    String getSortName(); // or whatever
    Reference<? extends Party> getCanonicalReference();
}

interface Person extends Party {
    Reference<? extends Person> getCanonicalReference();
}

This also answers the question about what type argument to use in the example above:

Person p = null;
Reference<? extends Person> ref = p.getCanonicalReference();
Person p2 = ref.dereference();

Finally, BaseObjectAdapter:

class BaseObjectAdapter<T extends BaseObject> implements BaseObject {
    /* various instance fields... */
    private Reference<T> canonicalReference; // with the usual getters and setters
    public Reference<T> getCanonicalReference() {
        return canonicalReference;
    }
}

Is there a lesson to learn from this? Generic types are not C++ templates. Design your types to have the type parameters they naturally need and do not add unnecessary type parameters to save yourself from typing. Joe and I have been talking about best practices when using generics for software design at JavaOne 2005 and 2006. We recommend that you try to avoid unnecessary type variables. Sometimes the solution is to not use generics.

Generic Methods

Type parameters on generic methods are different. However, not too much. Never use type parameters on public methods if they only benefit the implementation. Instead use a wildcard and a private generic method if you need to name a type when implementing the behavior. For example, consider how to implement Collections.reverse:

public static void reverse(List<?> list) {
    reverse0(list);
}
private static <T> void reverse0(List<T> list) {
    ListIterator<T> fwd = list.listIterator();
    ListIterator<T> rev = list.listIterator(list.size());
    for (int i=0, mid=list.size()>>1; i<mid; i++) {
        T tmp = fwd.next();
        fwd.set(rev.previous());
        rev.set(tmp);
    }
}

Self-Types

What about self-types? The Java™ programming language does not have self-types right now. Self-types would allow you to refer to the type of the receiver (the type of this, the current class). Imagine we used this to denote a self-type and we could avoid overriding getCanonicalReference:

interface CanonicallyIdentified {
    Reference<? extends this> getCanonicalReference();
}

interface Party extends BaseObject {
    String getSortName(); // or whatever
}

interface Person extends Party {}

Compare this to when I used covariant return types:

interface CanonicallyIdentified {
    Reference<? extends this> getCanonicalReference();
}

interface Party extends BaseObject {
    String getSortName(); // or whatever
}


interface Person extends Party {}

 

interface CanonicallyIdentified {
    Reference<?> getCanonicalReference();
}

interface Party extends BaseObject {
    String getSortName(); // or whatever
    Reference<? extends Party> getCanonicalReference();
}

interface Person extends Party {
    Reference<? extends Person> getCanonicalReference();
}

The Strongtalk type system for Smalltalk has self-types. You can download the Strongtalk system from www.strongtalk.org. The LOOJ paper by Bruce and Foster includes a proposal for adding ThisClass to the Java programming language.

If self-types were added to the Java programming language, it would be obvious to consider retrofitting this onto Object.clone():

protected native this clone() throws CloneNotSupportedException;

Unfortunately, this is not possible because the specification of that method does not require:

x.clone().getClass() == x.getClass()

It is only recommended and such a change could then break existing programs that follow the specification. Although we sometimes have to break source compatibility, breaking programs that follow the specification is not a viable option. In the situation where new API is defined or the subtypes of a class are controlled, it is possible to take advantage of self-types on the clone method:

class NewClass implements Cloneable {
    protected this clone() {
        return (this)super.clone(); // cast required as we cannot retrofit Object.clone()
    }
}

Since it is already possible to use covariant return types to simulate this behavior today and we cannot retrofit Object.clone() I consider it unlikely that we will add self-types to the Java programming language anytime soon.

Acknowledgments

Joseph D. Darcy and Alex Buckley provided me with a lot of useful feedback on the early drafts and helped me get the flow better.

Starting tomorrow, the Java™ compiler, javac, is available under the GPLv2 from openjdk.dev.java.net/compiler.

I cannot help you interpret the license because I am not a lawyer. For the same reason, nothing on this blog is legal advice. Sun's lawyers cannot help you either because that could create a conflict of interest. Be sure to read the license carefully, the use of the Java trademark is restricted. Retain your own lawyer if you have any legal questions.

We continue to accept patches to the compiler. Before you submit a compiler patch, I recommend that you read my rules of thumb about maximizing your chances for getting a patch accepted.

All that being said, I am really excited about this and it has been real hard not to be able to talk about this. I hope to see a lot of projects prototyping new programming language constructs, and tighter integration with IDEs and other applications that use a compiler. We have already seen very cool things such as the Jackpot project and performance in GlassFish. Since both projects are open sourced, relying on closed source may have been a problem for some members of their respective communities. However, any such concerns should be history now.

Earlier this week JSR 199 passed the Final Approval Ballot. This means that the JSR is approved and we just need to do the final release (which will be part of JDK 6). Managing a JSR has not been easy but fortunately Ian and Azar stepped in and helped me with the final paperwork. The JSR would not have been possible without the support from the entire compiler team and especially Jon, Seetharam, and Igor's contributions. Similarly, the feedback from the Sun-internal reviewers, especially the NetBeans team, has been invaluable. I should not forget to mention the expert group.

We celebrated locally by drinking a glass of champagne but since our team is spread across the world, I decided to further the celebration with a new look for my blog (please let me know if you notice any problems, I have only tested the look with the Firefox and Opera web browsers).

You may have noticed that I stopped posting my bi-weekly summary of compiler fixes. I plan to pick it up again when JDK 7 development takes off.

Right now is kind of a slow period and that is nice. Later today, JSR 199 should move to the final approval ballot (FAB). This is the last stage it needs to go through before the final release which is delivered in JDK 6. All JSRs including the umbrella JSR for the release must go through this process before we can ship the release.

The only changes that we will make to JDK 6 at this time is to fix show stopper issues. This means that most of my team is looking at either improving documentation, writing sample code, making it easier to build the compiler, or JDK 7.

Personally, I'm procrastinating on writing compiler documentation by fixing bugs to go into JDK 6 update 1 and JDK 7. So far I have lined up a little over twenty fixes. I'll give a summary when I know more about how you can get them. It is most likely JDK 7 b02 but we do not have weekly builds since we are focusing our release engineering and quality resources on JDK 6. So I don't know when.

Besides pruning my bug list, I'm also helping with figuring out which language features Sun would like to see in JDK 7. Clearly, we're looking at code abstraction but there is a lot of other ideas floating around. Another thing we are investigating is if it is possible make generic types a little more convenient. For example, code like this is pretty annoying:

Map<Comparable<? extends Number>, List<Number>> x =
    new HashMap<Comparable<? extends Number>, List<Number>>();

James Gosling has returned to the Java programming language and I really like his approach. I think the only problem we have is that we have too many good ideas and just one tiny release to put them into.