Thursday Feb 16, 2006

Annotations: Toward Pluggable Types

One of the new language constructs introduced in Java 5 was annotations. The potential of this feature is larger than many people realize. At the same time, like any construct, it is open to abuse. Hence this blog entry.

One of the more interesting things one can do is define annotations that support optional static analyses. For example, consider an annotation like @NonNull, that signifies that a variable should never hold null. If this annotation is used consistently, a tool can check that such variables indeed never become null at run time. You could say good bye to null pointer exceptions!

This is just one example (albeit, perhaps the most important one) of annotations being used to define an optional type system. It is very important to understand what one can and cannot accomplish in this way.

Since I like being negative, let me start by saying what you cannot do:

You cannot change the run-time behavior of the program

You cannot prevent an otherwise legal program form compiling or executing


Annotations can never be used to change the semantics of a program, or to cause compilation to fail. All annotations do is attach information (metadata) to parts of the program. Tools can then interpret the metadata as they see fit; however, the meaning of a program is fixed by the Java language specification, and cannot be altered.

Occasionally I get asked where in the specs is this stated. The answer, of course, is that the specs don’t have to state any such thing. They already give a definition of what any piece of source code means, and a conformant Java compiler must abide by that specification, and cannot change that meaning.

The fact that a program means the same thing everywhere is at the core of the value proposition of the Java platform. Everyone can read a program and understand what it is going to do; the meaning doesn’t change because of a given installations preferences, local libraries or configurations. That is what helps the Java programming language be such a valuable lingua franca across the industry.

Now, on the positive side, you can plug in a wide variety of semantic analyses or type systems, and get reports on potential errors in your programs. Optional type systems are type systems where the type annotations are syntactically optional, and (more important) the types never influence run-time semantics. Optional types are a perfect fit with annotations.

By definition, one can’t dynamically test for an optional type property of an object. While this may seem to be a limitation, the fact that optional types don’t effect the run time means that different optional type disciplines can co-exist without interacting with each other. That means you can plug in multiple, distinct analyzers/tools//typecheckers. This has the potential to enable pluggable types.

The annotation system may need a bit of a boost in some cases, because an analyzer may need annotations not just on declarations, but on additional language constructs. For example, the Javari research project at MIT, which allows you to distinguish between ordinary and read-only references, would require annotations wherever types can appear (including casts and type parameters, which are not currently supported).

In principle, it would be nice if you could attach annotations to every node in the abstract syntax tree. Indeed, even comments could be viewed as annotations. As Michael Van de Vanter has noted, if comments are treated as meaningful metadata belonging to well defined nodes in the AST, refactoring tools can preserve them reliably. However, adding annotations at essentially arbitrary program points poses significant syntactic challenges, so don’t expect that anytime soon.

Comments:

Gilad, it would be really neat if we could apply annotations to AST nodes or at least to the code blocks like I proposed some time ago. http://forums.java.net/jive/thread.jspa?threadID=988&tstart=0 More over, these annotations can be brought into the bytecode. It should be quite simple to implement a new bytecode attribute (that will be ignored by old JVMs), which will have start and end offsets within bytecode and link to the annotation construct for that bytecode range. This will be somehow similar to the bytecode attibutes that are representing exception handlers or local variables. This feature would open a huge range of possibilities for external tools and not be limited to checking.

Posted by Eugene Kuleshov on February 16, 2006 at 10:22 PM PST #

Gilad, I'm not sure I understand this @NotNull tool. It's going to check at runtime that the variable never becomes null, and if it happens, what is this tool going to do?... throw an exception? What's the difference with getting a good ole' NullPointerException?

Posted by John Smith on February 16, 2006 at 10:53 PM PST #

JML is a way of using comments to do this knid of things. It uses comments because it was designed for older versions of Java that didn't had annotations. It's also more expressive that way. And there are tools that check that "non-null" thing and many other possible errors for a long time (e.g., ESC/Java). Ironically, they just need to get better and more robust. (ironically because that's what they are supposed to help with)

Posted by Radu Grigore on February 17, 2006 at 06:05 AM PST #

"You cannot change the run-time behavior of the program" Doesn't recent EJB3 spec violate this? "You cannot prevent an otherwise legal program form compiling or executing" Doesn't @Override annotation violate this?

Posted by Maxim Shafirov on February 17, 2006 at 06:51 AM PST #

A few quick responses: Eugene: I'd love to discuss further extensions to make annotations more useful. I'm hoping we'll have some sort of JSR for this, but I can't commit to any such thing yet.

John: You misunderstand my inent. @NonNull doesn't do anything at runtime, nor should it. It makes it possible for a tool to check statically that your program simply doesn't ever cause a variable marked with @NoNull to take on the value null.

Radu: ESC/java is really cool technology. It is also very heavyweight, and can do theorem proving. JML is in that same space. They are also intellectually heavyweight things, that won't get used by most programmers. @NonNull would use the xistig framework of APT, hat is widely deployed and well integrated with javac, to do a very simple check. Which means it gets used by many people, and is beneficial to many people.

Maxim: @Override is specified in the JLS, and so it can and does violate the general notion I've outlined - just as the whole type system of the language does. However, you cannot do this for an annotation that is not specified by the JLS.

Posted by Gilad Bracha on February 17, 2006 at 12:10 PM PST #

The problem with many pluggable type systems, including the aforementioned @NotNull analysis is its indecidability in case not all codebase is annotated, which is inevitable in practise when annotating existing code. Thus it is either necessary to blindly believe the annotation, or to provide at least runtime checks. These checks for @NotNull are really not the same as old NPEs, since they are going to signal contract violation right when it is violated, not at some time afterwards. The same could be also true for some other analyses.

Posted by Eugene Vigdorchik on February 17, 2006 at 12:47 PM PST #

How should annotations to be checked in practice? The annotation processing tool does not give access to the syntax trees for method bodies, by design as I understand it. But you need to get out into all corners of the source code to check something like @NonNull. Are there any plans to extend APT to allow stuff like this or would you use a different approach?

This is really neat, by the way: NonNull in IntelliJ IDEA.

Posted by Christian Plesner Hansen on February 18, 2006 at 03:04 AM PST #

Christian, Yes, there are plans in this direction. Peter Ahe is working on a tree API for javac.

Posted by Gilad Bracha on February 18, 2006 at 07:57 AM PST #

What would be your number one request on ESC/Java to make it usable by more programmers? I really am curious.

Posted by Radu Grigore on February 18, 2006 at 08:36 AM PST #

Radu, Your question is biased: it assumes there is one major thing that needs to change. What is the #1 thing you would change in an elephant to make it a turtle? ESC/Java is simply beyond the scope of the average programmer. It's a great tool for sophisticated programmers. Ordinary programmers do not prove theorems or even invariants, and they are not about to start. Sad but true.

Posted by Gilad Bracha on February 18, 2006 at 12:24 PM PST #

I love the idea, but pluggable types as something ordinary Java developers would grasp and be able to use, and in particular be able to debug in case of conflicting optional type information? I'm a little sceptical, given how much trouble getting 'const correctness' annotations right causes in C++ for ordinary developers.

Posted by Dalibor Topic on February 21, 2006 at 08:51 PM PST #

Post a Comment:
Comments are closed for this entry.