The Former Weblog of Peter Ahé

Forget about my wish list from last month. There are a few really good ideas on that list (closures and super packages). I gave up on properties. The rest is about making generic types easier to use. However, during my recent hiatus in Europe, I have been reflecting on generics. What is the problem with generics? Is it too complicated?

No doubt, generic types are relatively new in mainstream programming languages and wildcards are truly unique: it was literally invented for the Java™ programming language. This means that a lot of developers are not yet comfortable with generics. When I first heard about Object-Oriented Programming, I found it really scary. All that about inheritance, overriding, overloading, is-a, and has-a. Frankly, there is still a lot of confusion about something as fundamental as is-a, and has-a.

Most developers are very comfortable with OOP. In five years, most developers will be comfortable with generics.

Generic types are taking a lot of heat on various blogs. Not all of the critisism is without substance. I see that there is a problem when I talk to developers and close compiler bug reports as not being a bug. I see that there is a problem when trying to teach generics. I see that is a problem that causes otherwise diligent developers ignore a whole category of warnings about potential type unsafety in their programs.

The problem is the lack of reification. In other words, the lack of runtime type-information about generic types (at runtime, you cannot tell an ArrayList<String> from a ArrayList<Integer>). We need to do something about that. It is indeed time to erase erasure!

In the next weeks I'll explain what it buys us and how we can accomplish it.

Let me jump right into it. I propose an alternative to type inference for local variables. I'll explain why later. Everybody knows this example:

Map<String,List<Integer>> map = new HashMap<String,List<Integer>>();

I propose this solution:

Map<String,List<Integer>> map = HashMap.new();

Before I get to caught up in the details, let me examine why we are talking about this in the first place. As far as I'm aware, there are primarily two reasons for looking at examples as the above:

1. DRY—Don't repeat yourself.
2. Inspiration from scripting languages (or dynamically typed languages): JavaScript, Ruby, Smalltalk, Scheme, etc. Moving to Java from such languages can feel very constraining because you have to type so many types.

I'm currently aware of two serious proposals, the first from James:

map := new HashMap<String,List<Integer>>();

The second is from Christian:

final map = new HashMap<String,List<Integer>>();

Rémi has implemented both of these proposals. But let us not forget what is going on in other languages:

var map = new HashMap<String,List<Integer>>();

This is well known from JavaScript and has recently been added to C#. However, adding it to Java would require adding a new keyword (which is bad) or having some really strange rules. So var is not really a contender for Java.

All of these proposals (mine excluded) essentially solve the problem using the same technique: type inference. The problem is that this promotes bad behavior:

• Forces use of class types, not interface types on local variables. Generally the opposite is recommend as implementations are more flexible when they rely on interfaces instead of specific implementation classes. This is mediated by only allowing this syntax on local variables so no API is affected.
• Code could be slightly harder to read. In many cases type declarations on variables are helpful when reading some code. Most will argue that some of the really nasty examples with long type arguments are fun to neither read nor write. So eliding the type on a variable can sometimes improve readability. However, since the language can't enforce a readability rule, some developers will misuse such a feature.

This motivated Neal to suggest:

Map<String,List<Integer>> map = new HashMap<>();

Neal and I just had a chat about this. I think we both agree that it is ugly but the alternative could have compatibility problems (as well as other problems):

Map<String,List<Integer>> map = new HashMap();

So you can probably see how I got the idea:

Map<String,List<Integer>> map = HashMap.new();

However, there is more to it than just a simple syntax for constructing instances. My initial reaction to this problem was: we need more static factories throughout the JDK. For example:

Map<String,List<Integer>> map = HashMap.create();

Where the definition of create is:

public static <K,V> HashMap<K,V> create() { return new HashMap<K,V>(); }

What I propose is actually a new syntax for declaring static factory methods and that the compiler adds them by default (as it does with constructors). For example, if you declare a class:

public class Box<T> {
    public Box() {}
}

The compiler would automatically add this method:

public static <T> Box<T> new() { return new Box<T>(); }

The programmer can specify any number of new methods just as it is possible to have multiple constructors. If the programmer provides a new method with a signature that matches that of a public constructor, that will be used instead of the compiler providing one.

Thanks to Neal M Gafter for his input on this idea and this posting.

Dear Santa,

Hans Muller just convinced me that I didn't want a short hand syntax for declaring properties after all. Hope it isn't too late.

Dear Santa,

This year I wish that Java...

[Read More]

Back in May, I described how we would update the specification and implementation of @Override. Unfortunately, I messed up when integrating the changes and the specification of @Override was not updated. The compiler's behavior is changed but the documentation does not reflect that.

I will make sure to get the documentation fixed in JDK 7 as soon as possible but it is more problematic updating the documentation of JDK 6 because JSR 270 has been finalized. In the meantime, here is what it was supposed to have looked like:

Indicates that a method declaration is intended to override a method declaration in a supertype. If a method is annotated with this annotation type compilers are required to generate an error message unless either of the following conditions hold:

• The method does override or implement a method declared in a supertype.
• The method has a signature that is override-equivalent to that of any public or protected method declared in Object.

In case Bruce is still reading my blog, I just want to make it perfectly clear that Joe had absolutely nothing to do with this snafu.

I posted a proposal on closures a few days ago and the comments can so far be partitioned in two categories: "I like closures" and "I don't understand closures".

The strange thing is that I haven't seen any comments from people that understands closures saying they don't like them. So this makes you think if closures is just a matter of understanding them or not.

This must be what Neal is thinking because he tries to answer the question: what is the point of closures?

I would also recommend that you play a little with Smalltalk or Ruby.

I'm not allowing comments on this one as you can comment on my previous entry on closures or on Neal's blog.

Some clever guys have written up a proposal on closures and been kind enough to put my name on it. I was just sitting in the room trying my best not to look too stupid

A closure is an anonymous function (aka lambda expression) in which all free variables can be accessed (even if they are not final). Furthermore, in the tradition of Smalltalk, the closure should be able to return directly from the method enclosing the closure definition (aka non-local return).

In the Java™ programming language, closures can be simulated to some extent with anonymous classes but these cannot access non-final local variables and have no way to exit the method containing the definition besides throwing an exception (which should not be used for control flow).

Closures have been a favorite of most Smalltalk freaks and has gotten renewed interest in languages such as Ruby.

Anonymous classes, although useful, can be a tad verbose and tiresome to type so this proposal must address the proper closure of free variables and non-local return as well as suggest a consice syntax.

I'm currently looking at how javac handles @Override. Consider the current specification of @Override:

Indicates that a method declaration is intended to override a method declaration in a superclass. If a method is annotated with this annotation type but does not override a superclass method, compilers are required to generate an error message.

There are a few subtle points that may escape the casual reader:

• it says superclass not supertype
• override and implements are not mutually exclusive

There are a few surprising consequences of this:

    abstract class C1 {
        public abstract void m();
    }
 
    class C2 extends C1 {
        @Override
        public void m() {}
    }

Although the method C2.m implements the method C1.m, it also overrides it. So this is OK according to the current specification and javac accepts it. The next example is even more convoluted:

    public interface Test {
        void m();
    }

    abstract class A implements Test {
        // m() is inherited from Test
    }

    class B extends A {
        @Override
        public void m() {}
    }

This is also legal since m is a member of the abstract class A, B.m is overriding (and implementing) a method in a superclass. However, javac fails to accept this program.

So thought about it and came up with this specification:

Indicates that a method declaration is intended to override a method declaration in a supertype. If a method is annotated with this annotation type but does not override a supertype method, compilers are required to generate an error message

However, this doesn't work as you would expect: although override and implements are not mutually exclusive, override is not a "superset" of implements:

    interface A {
        void m();
    }

    class B implements A {
        public void m() {} // implements but doesn't override
    }

Whereas:

    interface A {
        void m();
    }

    interface B extends A {
        void m(); // overrides but doesn't implement
    }

Confused? Then you're getting there ;-)
So we tried this one:

Indicates that a method declaration is intended to override a method declaration in a supertype. If a method is annotated with this annotation type but does not override or implement a supertype method, compilers are required to generate an error message

However, what about (currently accepted by javac):

    interface Foo {
        @Override
        String toString();
    }

So does Foo.toString override or implement a method in a supertype? Not really, the JLS is a bit convoluted here but basically says that although Object is a direct supertype of Foo, Foo does not inherit Object.toString because Foo itself declares a toString method...

Now what? We thought about it and came up with:

Indicates that a method declaration is intended to override a method declaration in a supertype. If a method is annotated with this annotation type but is not in fact override-equivalent to any method declared in a supertype, compilers are required to generate an error message.

So how is this different? Rather than relying on the definition of override or implements in the JLS we simply say what we intended to say all along: if a similar method exists in a supertype, then you may use @Override.

So then Eugene says: what about visibility. Oh boy:

Indicates that a method declaration is intended to override a method declaration in a supertype. If a method is annotated with this annotation type compilers are required to generate an error message unless either the method is override-equivalent to a [update: public or protected] method in Object, or the method does override or implement a method declared in a supertype.

I hope to have this ready for b86.

Least wanted features collecting dust and votes in the bugdatabase include const keywords like C++ and multiple return values.

We never planned or dreamed of adding these features to the Java Programming Language. So why did we keep them open? I guess it was a misguided attempt to be open. But openess is not pretending we're listening, openess is being open about our decisions. Not everyone will agree on our decisions but at least they know where we stand.

This is one of the great things about open source: if you disagree with the developers, you can fork of your own branch. I think the biggest value of Java is being a "standard" platform, so I don't think that anyone should be allowed to distribute modified JDKs unrestricted. This is why I think we have found the perfect balance with the Peabody effort. This is not free software, but you can download all the sources and try out all your vacky ideas. With some restrictions, you can even share them with other developers.

But best of all, if you can fix your most (least) favorite bug, then you get a chance to get the fix back into the JDK. If you submit the right paperwork as decribed in the link above, we can now accept your changes and incorporate them into the JDK. I imagine that this could also happen with your most wanted features, although big features will have to go through the Java Community Process. Or we might say no, just as we did with const keywords and multiple return values. Saying no is the hardest part of being open.

We have published a paper on wildcards. The paper should be fairly accessible as it is focused on the use and implementation of wildcards. However, if you are looking for a tutorial, you will be better of with the Generics Tutorial.

Keith Lea was kind enough to provide me with this example:

    List<Class<?>> classes =
        Arrays.asList(String.class, Boolean.class);

This small example does not compile, but Keith had figured out that changing it helped:

    List<? extends Class<?>> classes =
        Arrays.asList(String.class, Boolean.class);

Of cause, Keith would never have written the first example if he had read my previous entry on "Why is the Capture of a Wildcard Incompatible with Itself?".  However, the compiler still complains, though not as loudly:

warning: [unchecked] unchecked generic array creation of type java.lang.Class<
    ? extends java.lang.Object&java.io.Serializable&java.lang.Comparable<
        ? extends java.lang.Object&java.io.Serializable&java.lang.Comparable<?>>>[]
for varargs parameter
        Arrays.asList(String.class, Boolean.class);
                     ^
1 warning

As any good programmer, Keith was concerned about this warning.  Anyone should strive to remove warnings from their programs.

So what is all that goo in the warning?  Why would the compiler claim that this is unchecked?

First, notice that the compiler wants to create an array of type:

java.lang.Class<
     ? extends java.lang.Object&java.io.Serializable&java.lang.Comparable<
         ? extends java.lang.Object&java.io.Serializable&java.lang.Comparable<?>>>[]

We don't need to see java.lang and java.io.  Also, Object is redundant in these types so let's clean it up:

Class<? extends Serializable&Comparable<? extends Serializable&Comparable<?>>>[]

Now that's almost easy to read.  But where does this strange type come from?  The answer is inference.  Inference is new to Java but has been around for a long time in other programming languages such as Standard ML.  Take a closer look at the java.util.Arrays.asList method declaration:

public static <T>List<T> asList(T... a);

This method creates a java.util.List view of an array.  The method uses some of the new features of Java: generic methods and varargs.  Varargs is the easy one to understand: I you write ... after the last type in a method declaration, the compiler will accept a variable number of arguments (hence the name varargs).  The compiler will then create an array of these arguments an pass that on to the method as if it was declared as:

public static <T> List<T> asList(T[] a);

This explains why the compiler is creating an array.  But what about the element type of that array?  Notice the <T> before the actual method declaration.  This is a type variable.  Since we want to be able to create a List view of any array, we are asking the compiler to figure out what the actual type is.  That process is called inference.  Inference is mostly done without looking at the expected result type (otherwise the inference would be really complicated when the expected result type is itself inferred).  All this means that the compiler must find a type T such that:

Class<String> <: T and Class<Boolean> <: T

Also we want to be as precise as possible.  One candidate for T is Class<?> but that is somewhat imprecise.  String and Boolean has more in common than that.  E.g., they both implement the interfaces Serializable and Comparable.  So a more precise candidate would be:<

Class<? extends Serializable & Comparable>

The & above is called a compound type and should be considered as an anonymous class that implements both interfaces.

But wait a minute...  Isn't Comparable a generic interface itself?  It is and we are looking for what Boolean and String have in common:

Boolean implements Comparable<Boolean>
String implements Comparable<String>

Let us improve our candidate for T:

Class<? extends Serializable & Comparable<? extends Serializable & Comparable>>

Still no good.  We need parameters for the Comparable interface.  At this point the compiler gives up and says "I don't know":

Class<? extends Serializable & Comparable<? extends Serializable & Comparable<?>>>

The most precise solution would actually have been:

Class<? extends X> where X extends Serializable & Comparable<? extends X>

But the compiler is not quite clever enough to handle this type yet...

But this inferred type is generic and arrays of generic types are quite troublesome (could be a topic for a new blog entry).  Thus the compiler will give a warning each time an array of a generic type is created.  However, unbounded wildcards are quite benign, so the compiler will not complain about array types such as Class<?>.  Incidentally, this was actually the type Keith was looking for, but the compiler cannot use that information unless told explicitly:

    List<? extends Class<?>> classes =
        Arrays.<Class<?>>asList(String.class, Boolean.class);

Now we give an explicit type parameter for T so the compiler doesn't need to infer any types.  An array of Class<?> quite benign, so the warning disappears.

Sometimes you get an error like this:

MyClass.java:9: incompatible types
found   : java.util.Iterator<java.util.Map.Entry<java.lang.String,capture of ?>>
required: java.util.Iterator<java.util.Map.Entry<java.lang.String,?>>
	Iterator<Entry<String,?>> it = map.entrySet().iterator();
                                                              ^

In short: Iterator<Entry<String,capture of ?>> is not compatible with Iterator<Entry<String,?>>. The relevant lines of my program are:

    Map<String,?> map;
    Iterator<Entry<String,?>> it = map.entrySet().iterator();

Is "capture of ?" not a subtype of "?"? Yes, but that is not what we need here. We need the type arguments to be of the same type (you should already know that a List<String> is not a subtype of List<Object>).

This seems silly, isn't a wildcard the same type as it self? Yes, but here we have a wildcard and a type variable ("capture of ?" actually means that a wildcard has been converted into a type variable using capture conversion).

Then why bother with this capture conversion, my program is obviously correct and capture conversion gets in my way? No, your program is wrong. Capture conversion can actually help you understand why.

How? First of all you must learn how to read a wildcard aloud, e.g. read "Map<String,?>" as a Map from String to an unknown type. The second argument to Map is unknown but at some point the compiler needs to reason about a specific type, e.g., when you write "map.entrySet()". The compiler must reason about the specific type of the object "map" refers to at this point at runtime. The wildcard must not longer be wild (unknown) so the compiler captures a snapshot of "?" in an anonymous type variable at that point in the program, i.e., "capture of ?". So the difference between "?" and "capture of ?" is that the former refers to all possible types and the latter refers to a particular type at a particular point in you program.

I still do not see the point of all this? Ok, lets extend the above program a little bit, first we need a special (but perfectly sensible iterator):

    class MyIterator<E> implements Iterator<E> {
	Set<E> iteratedSet;
	// other stuff left out for brevity
    }

Now do this:

	Entry<String,Number> unrelatedEntry;
	...
	MyIterator<Entry<String,?>>mit = (MyIterator<Entry<String,?>>)it
	mit.iteratedSet.add(unrelatedEntry);

Now all of a sudden you have saved an Integer somewhere it might not belong!!! This will almost certainly lead to a class cast exception later in your program (e.g, if map referred to an instance of HashMap<String,String>).

OK, now I understand my problem. How do I solve it? That is simple: just write:

    Map<String,?> map;
    Iterator<? extends Entry<String,?>> it = map.entrySet().iterator();

Notice the added "? extends". UPDATE: I have disabled comments. Please visit the generics forum instead.