Joseph D. Darcy's Sun Weblog

Properties via Annotation Processing

The annotation processing APIs provided by apt in JDK 5 and javac in JDK 6 both present a read-only view of source code; by design directly modifying the input sources is not supported through either annotation processing API. Recently, Project Lombok has used the hook of being able to run an annotation processor inside javac to start an agent which rewrites javac internal classes to change the compiler's behavior; yikes! Such extreme measures are not needed to get much of the effect of modifying the input sources. As I've outlined in the annotation processing forum over the years, just using standard annotation processing to generate a subclass or superclass of a type being processed is a very powerful technique for controlling the ultimate semantics and behavior of the original class. For example, I've hacked up a proof-of-concept annotation type and matching processor to generate property-style getter and setter methods based on annotations on fields.

Concretely, the programmer writes something like

public class TestProperty extends TestPropertyParent {
    protected TestProperty() {}

    @ProofOfConceptProperty
    protected int property1;

    @ProofOfConceptProperty(readOnly = false) // Generate a setter too.
    protected long property2;

    @ProofOfConceptProperty
    protected double property3;
    
    public static TestProperty newInstance(int property1,
                                           long property2,
                                           double property3) {
        return new TestPropertyChild(property1, property2, property3);
    }
}

and, after suitable annotation processing, using the TestProperty type as in

public class Main {
    public static void main(String... args) {
        TestProperty testProperty = TestProperty.newInstance(1, 2, 3);
        output(testProperty);
        testProperty.setproperty2(42);
        output(testProperty);
    }

    private static void output(TestProperty testProperty) {
        System.out.format("%d, %d, %g%n",
                          testProperty.getproperty1(),
                          testProperty.getproperty2(),
                          testProperty.getproperty3());
    }
}

produces the expected output:

prompt$ java Main 
1, 2, 3.00000
1, 42, 3.00000

This approach does have limitations; primarily the annotated class like TestProperty needs to be written to allow its superclass and subclass(es) to be generated. Since it runs as part of the build, the annotation processor needs to be built separately beforehand. The javac command to run the annotation processor looks like:

javac -s ../gen_src/ -d ../bin -processor foo.PocProcessor -cp ../lib TestProperty.java Main.java 

Good practice sets an output location for generated source code, TestPropertyParent and TestPropertyChild in this case, separate from the output location for class files. When compiling this, javac emits an error before the superclass and subclass are generated, but the entire compilation process completes fine correctly generating the source files and compiling all the files together. Java IDEs have varying levels of support for annotation processing; check your IDEs' documentation for details.

The annotation type and processor is only a proof of concept; many possible refinements are left as "exercises for the reader" including:

• Developing a second annotation type to mark a class separate from the annotation to configure how each field should be treated.

• Additional structural checks on the annotated code, proper modifiers on fields and constructors, etc.

• Generation of equals and hashCode methods.

While using an annotation processor to approximate properties is awkward compared to built-in language support, annotation processors can be used today as part of some toolchains and they are configurable by the user. The code provided should be enough of a starting part for others to experiment with using annotation processors in this fashion; have fun.

(2009-09-02 20:29:30.0) Permalink Comments [8]

Annotation Processing Build Advice: Set source and class file destinations

In addition to following the general build advice of setting source, target, and encoding, when running annotation processors the source and class file destination directories should be set via javac's -s directory and -d directory options, respectively. Using the Filer, annotation processors can generate new source code to be compiled as part of the project and can also even generate new class files directly.

As recommended on the annotation processing forum, a much more hygienic build results when different kinds of generated output files are not intermixed with each other and any kind of output file is not mixed with any input files. (Input files would presumably be tracked under version control where generated files would not be.) In more detail:

• Set the directory for outputting generated source files using -s

• Set the directory for outputting generated class file code using -d

• Do not have the source and class file output locations overlap with each other and do not have either of them overlap with the sourcepath or classpath.

Following these guidelines clearly delineates inputs to the compiler and outputs from the compiler.

Although apt has been deprecated, the same recommendations hold there too.

(2009-08-07 09:30:00.0) Permalink

An apt replacement

Showing that no rule is so broad as to not admit an exception, the apt tool and its associated API, com.sun.mirror.*, are being deprecated in JDK 7. The plan is to remove the tool and its API to the next major JDK release after JDK 7.

As a com.sun.* API, the apt API is not governed by the JCP; however, we don't usually mass-deprecate com.sun.* APIs either. We introduced apt in JDK 5 to gain experience with annotation processing before standardizing this facility with JSR 269 in JDK 6, which added the javax.annotation.processing and javax.lang.model.* packages and added annotation processing options to javac.

As the lead engineer for both apt and JSR 269, the JSR 269 API and tool experience should be uniformly better than apt; the newer API is easier to use, more flexible, and should run faster as well. I unconditionally recommend transitioning to the JSR 269 API and javac (or its compiler API) for all your annotation processing needs.

The table below summarizes the apt deprecation information.

Summary of deprecated apt API replacements

apt type	Standard Replacement
Package com.sun.mirror.apt
AnnotationProcessor	javax.annotation.processing.Processor
AnnotationProcessorEnvironment	javax.annotation.processing.ProcessingEnvironment
AnnotationProcessorFactory	javax.annotation.processing.Processor
AnnotationProcessorListener	No analog.
AnnotationProcessors	No analog.
Filer	javax.annotation.processing.Filer
Filer.Location	javax.tools.StandardLocation
Messager	javax.annotation.processing.Messager
RoundCompleteEvent	No analog.
RoundCompleteListener	No analog.
RoundState	javax.annotation.processing.RoundEnvironment
Package com.sun.mirror.declaration
AnnotationMirror	javax.lang.model.element.AnnotationMirror
AnnotationTypeDeclaration	javax.lang.model.element.TypeElement
AnnotationTypeElementDeclaration	javax.lang.model.element.ExecutableElement
AnnotationValue	javax.lang.model.element.AnnotationValue
ClassDeclaration	javax.lang.model.element.TypeElement
ConstructorDeclaration	javax.lang.model.element.ExecutableElement
Declaration	javax.lang.model.element.Element
EnumConstantDeclaration	javax.lang.model.element.VariableElement
EnumDeclaration	javax.lang.model.element.TypeElement
ExecutableDeclaration	javax.lang.model.element.ExecutableElement
FieldDeclaration	javax.lang.model.element.VariableElement
InterfaceDeclaration	javax.lang.model.element.TypeElement
MemberDeclaration	javax.lang.model.element.Element
MethodDeclaration	javax.lang.model.element.ExecutableElement
Modifier	javax.lang.model.element.Modifier
PackageDeclaration	javax.lang.model.element.PackageElement
ParameterDeclaration	javax.lang.model.element.VariableElement
TypeDeclaration	javax.lang.model.element.TypeElement
TypeParameterDeclaration	javax.lang.model.element.TypeParameterElement
Package com.sun.mirror.type
AnnotationType	javax.lang.model.type.DeclaredType
ArrayType	javax.lang.model.type.ArrayType
ClassType	javax.lang.model.type.DeclaredType
DeclaredType	javax.lang.model.type.DeclaredType
EnumType	javax.lang.model.type.DeclaredType
InterfaceType	javax.lang.model.type.DeclaredType
MirroredTypeException	javax.lang.model.type.MirroredTypeException
MirroredTypesException	javax.lang.model.type.MirroredTypesException
PrimitiveType	javax.lang.model.type.PrimitiveType
PrimitiveType.Kind	javax.lang.model.type.TypeKind
ReferenceType	javax.lang.model.type.ReferenceType
TypeMirror	javax.lang.model.type.TypeMirror
TypeVariable	javax.lang.model.type.TypeVariable
VoidType	javax.lang.model.type.NoType
WildcardType	javax.lang.model.type.WildcardType
Package com.sun.mirror.util
DeclarationFilter	javax.lang.model.util.ElementFilter
DeclarationScanner	javax.lang.model.util.ElementScanner6
DeclarationVisitor	javax.lang.model.element.ElementVisitor
DeclarationVisitors	No replacement.
Declarations	javax.lang.model.util.Elements
SimpleDeclarationVisitor	javax.lang.model.util.SimpleElementVisitor6
SimpleTypeVisitor	javax.lang.model.util.SimpleTypeVisitor6
SourceOrderDeclScanner	javax.lang.model.util.SimpleElementVisitor6
SourcePosition	No replacement.
TypeVisitor	javax.lang.model.element.TypeVisitor
Types	javax.lang.model.util.Types

(2009-07-27 11:27:01.0) Permalink Comments [7]

Language Model Changes as of JDK 7 Build 50

To date, there have been a few API changes to javax.lang.model.* in JDK 7. Early in the release, SourceVersion.RELEASE_7 was added to correspond to any new source-level changes coming in JDK 7 (6458819). Eventually, there will be changes to support the modulue construct being added by JSR 294; changes may or may not be needed for language changes coming from Project Coin and JSR 308. Once modulues are added, the JSR 269 API elements meant to cope with the expression problem will be tested. In the mean time, some minor API cleanups have occurred, one to make handling exceptions easier (6794071), another to group common functionality in mixin interfaces (6460529), and some minor API clarifications (6501749). Along the way, there has been a little general bug fixing too (6478017, 6583626, 6498938).

Small tweaks and bug fixing will continue to occur in the javax.lang.model.* and javax.annotation.processing packages throughout JDK 7 in addition to changes needed to support new language features.

(2009-03-12 13:36:36.0) Permalink

JSR 269 Passes Final Approval Ballot

The votes are in and JSR 269 has passed its final approval ballot!

Now that the initial JSR 269 API is defined, I'm looking forward to using the API for various meta-programming tasks, like writing coding convention checkers and incrementally refining the API in JDK 7.

(2006-11-07 23:50:28.0) Permalink Comments [3]

JSR 269 in Final Approval Ballot

I'm very pleased to report that JSR 269 has reached the final approval ballot stage of the JCP process.

The only change since the proposed final draft was to correct the specification of VariableElement.getConstantValue.

(2006-10-24 17:08:39.0) Permalink

JDK 6 Build 102 JSR 269 API Changes

Build 102 included a very small change to the JSR 269 API: the specification of the VariableElement.getConstantValue method was changed so that a value was returned independent of the static-ness of a final field. The implementation already behaved that way; the specification mistakenly excluded the non-static case.

This is the last planned change to the JSR 269 API before its final approval ballot.

(2006-10-24 12:26:34.0) Permalink

JSR 269 Interview on Artima

Artima has an interview with me about JSR 269.

(2006-10-10 15:19:53.0) Permalink

JDK 6 Build 101 JSR 269 API Changes

There haven't been any API changes in JSR 269 since build 98 and the proposed final draft; however, build 101 includes a sample annotation processor showing how the API can be used to check program structure.

The sample file is under

sample/javac/processing/src/CheckNameProcessor.java

from the root JDK installation directory.

As the name suggests, this processor checks that the source being processed follows the naming conventions discussed in the language specification. This name checking is meant to be representative of a class of properties that could be checked for at build time by annotation processors.

A few stylistic points to note in the sample:

• Extending AbstractProcessor: The AbstractProcessor class is provided to make writing annotation processors more convenient by eliminating repeated boilerplate code. Generally, the results of the getSupportedFoo methods can be specified by applying a @SupportedFoo annotation to the class. The @Override annotations help ensure the methods of the class are behaving as expected. To initialize fields for objects that will persist for life of the processor object, such as the Messager and Filer retrieved from the ProcessingEnvironment, assign the fields in the overridden init method after super.init has been called.
• Implement functionality with a visitor, but expose the functionality with its own entry point: The NameChecker class provides a checkNames method for clients to pass an element to get its names checked. While this checking is implemented using a visitor, exposing that implementation detail is not friendly to clients.
• Have a policy for future language versions: The JSR 269 API models the current language, but the language will be changing in some future release and annotation processors written today could be run against new language constructs of the future. Two basic policies are:
  1. Write the processor to only work with the current language version.
  2. Write the processor to cope with unknown future constructs.
  The sample processor does the latter. It does this by returning SourceVersion.latest rather than a particular version constant and by having the visitor's visitUnknown method be non-fatal if called. The visitUnknown method will be called when encountering a construct newer than the platform version the visitor was written to.

For production use, the heuristicallyConstant method probably needs some tweaking; leave comments detailing any changes you find useful.

(2006-10-05 17:15:00.0) Permalink

JSR 269 in proposed final draft I'm happy to announce that JSR 269 has progressed to the proposed final draft stage of the JCP process. This draft corresponds to the version of JSR 269 implemented in build 98 of JDK 6. Please send comments to jsr-269-comments@jcp.org.

Changes from the public review draft include:

• Adding originating elements var-args parameters to the Filer create methods to enable better management of dependencies.
• Changing the return type of the getFooName methods to a separate Name extends CharSequence interface.
• Revised Type visitor structure
• Included a description of the annotation processing discovery process.
• Various specification clarifications:
  • More explicit null pointer defaults.
  • The modeling API is meant to be used for multiple purposes, including but not limited to annotation processing.
  • More notes on anticipated evolution of the API.

(2006-09-18 17:44:45.0) Permalink Comments [2]

JDK 6 Build 97 and 98 JSR 269 API Changes Compared to build 96, build 97 only contained an assortment of minor corrections and clarifications to the API, such as fixing typos and adding additional cross references to sections of the JLS. The most systematic change was updating the javadoc of the methods in the utility visitor classes to follow the recommended pattern for methods intended to be overridden: state the general method contract (in this case with {@inheritDoc}) following by "This implementation...". This pattern is discussed in Effective Java (Item 15: Design and document for inheritance or else prohibit it) and is used in the skeletal collection implementations.

Build 98

• Tightened the specification of the Filer to disallow overwriting files in more situations.
• Added a description of the discovery process to Processor.
• Explicitly stated the language modeling hierarchy javax.lang.model.* can be used for purposes other than annotation processing.

(2006-09-07 10:22:53.0) Permalink

JDK 6 Build 96 JSR 269 API Changes Build 96 of JDK 6 include a few changes since build 95:

• An originatingElements parameter was added to the Filer create methods. The originating elements serve as hints to the tool infrastructure to better manage dependencies; they are the types or packages (representing package-info files) which caused an annotation processor to attempt to create a new file. See the Filer specification for more information.

  This is a source compatible change, but a recompile of existing annotation processors which use a Filer is required.

• Elements.getPackageOf(Element) was added.
• Various minor specification changes:
  • Element.getSimpleName() result redefined for constructors and initializers
  • TypeParameterElement.getBounds() result when no explicit bound redefined.

(2006-08-21 16:32:33.0) Permalink

JDK 6 Build 95 JSR 269 API Changes Build 95 of JDK 6 include a few changes since build 92:

• The return type of the Filer.{create, get}Resource methods was changed from JavaFileObject to FileObject. This less-specific return type is more appropriate for files that are not being treated as source or class files by the annotation processing infrastructure.
• The RoundEnvironment.getElementsAnnotatedWith methods were clarified to state that non-annotation types are illegal arguments.
• The method PackageElement.isUnnamed was added so that the text of a package's name does not need to be inspected to determine if a package is unnamed.

(2006-08-14 20:06:40.0) Permalink

Mustang Build 92 JSR 269 API Changes Build 92 of Mustang includes API changes from the JSR 269 public review draft:

• The Filer methods return JSR 199 objects instead of a Writer or OutputStream. This revised Filer API gives clients more flexibility over how a file is written. There is also a getResource method to non-destructively read the contents of an existing file.
• Annotation processing is supported on package-info files, which can hold package annotations. While classes and interfaces (and their contents) are the most commonly annotated entities, packages can be annotated as well so the annotation processing API should consider those annotations too. More broadly, the new API can also accommodate any additional kinds of entities that will be able to be annotated in future language versions (like superpackages in Dolphin?).

There are small code deltas to adapt existing annotation processing code to use the new API. For Filer uses:

• The return type of createSourceFile was changed from Writer to JavaFileObject.

  Replace	filer.createSourceFile()	// Pre-build 92
  With	filer.createSourceFile().getWriter()	// Build 92 and later

• The return type of createClassFile was changed from OutputStream to JavaFileObject.

  Replace	filer.createClassFile()	// Pre-build 92
  With	filer.createClassFile().getOutputStream()	// Build 92 and later

• The createBinaryFile method was removed; its functionality can be gotten from createResource. The Filer.Location enum was removed and JavaFileManager.Location is used instead.

  Replace	filer.createBinaryFile(loc, pkg, relPath)	// Pre-build 92
  With	filer.createResource(javaFileManagerLoc, pkg, relPath).getWriter()	// Build 92 and later

• The createTextFile method was removed; its functionality can be gotten from createResource.

  Replace	filer.createTextFile(loc, pkg, relPath, charsetName)	// Pre-build 92
  With	filer.createResource(javaFileManagerLoc, pkg, relPath).getWriter()	// Build 92 and later
  Or	new OutputStreamWriter(filer.createResource(javaFileManagerLoc, pkg, relPath).getOutputStream(), charsetName)	// Build 92 and later

For RoundEnvironment uses:

• The getSpecifiedTypeElements method returning a Set<? extends TypeElement> was replaced with getRootElements returning a Set<? extends Element>.

  Replace	roundEnv.getSpecifiedTypeElements()	// Pre-build 92
  With	typesIn(roundEnv.getRootElements())	// Build 92 transition

Whether or not it is reasonble to wrap getRootElements with ElementFilter.typesIn depends on the circumstances and what the processor is trying to do. Using the new API, processors should handle annotations on any kind of element. If a processor is only examining classes and interfaces, applying the typesIn filter is reasonable. If other kinds of elements need to be processed as well, the getKind method can be used to find out what an element is or a visitor can be written to implement element-specific or kind-specific operations. (2006-07-24 16:30:54.0) Permalink

Mustang Build 91 JSR 269 API Changes In build 91 of Mustang, to address bug 6425592 we added a method to RoundEnvironment:

Set<? extends Element> getElementsAnnotatedWith(Class<? extends Annotation> a)

which parallels the existing method

Set<? extends Element>getElementsAnnotatedWith(TypeElement a)

The new method is similar in spirit to the Element.getAnnotation; while it violates meta and base level separation, used with care, it can make programming more convenient. (2006-07-18 13:20:35.0) Permalink

JSR 269 in public review I'm happy to announce that JSR 269 is now in public review. Please send comments to jsr-269-comments@jcp.org.

(The public review draft has several differences from the API specification and implementation in Mustang build 84:

• The Filer methods return JSR 199 JavaFileObject objects instead of java.io Writer and OutputStream objects.
• The legal input names to the createSourceFile and createClassFile methods in Filer were expanded to include syntax for creating package-info files.
• The method
  Set<? extends TypeElement> getSpecifiedTypeElements()
  was replaced with
  Set<? extends Element> getRootElements()
  

Several of these changes relate to enabling annotation processing on package-info files in additional to normal source and class files.) (2006-05-19 15:17:00.0) Permalink

apt mirror API Open Sourced I'm happy to announce that we have open sourced the apt mirror API. The classes and interfaces in com.sun.mirror.apt.* introduced in Sun's JDK 5.0 are now available under a BSD license from the aptmirrorapi java.net project; see the "Documents & files" page to download an archive of the source files.

By compiling the source files in the archive and creating the corresponding jar file, the resulting jar file can be used on a Java compiler's classpath instead of the tools.jar in Sun JDK when developing annotation processors. (2006-01-10 19:48:36.0) Permalink Comments [1]

JSR 269 in Mustang Build 57 The initial reference implementation of JSR 269 shipped as part of Mustang build 57! JSR 269 has two basic pieces, an API that models the Java programming language and an API for writing annotation processors. Those pieces are in the new packages javax.lang.model.* and javax.annotation.processing, respectively. The JSR 269 functionality is accessed by using new options to the javac command. By including JSR 269 support, the javac command now acts analogously to the apt command in Sun's JDK 5.

To get an overview of annotation processing, see our 2005 JavaOne session on that topic.

Annotation processing is now enabled by default in javac; we designed our implementation of JSR 269 so that there should only be negligible speed impact on javac when no annotation processors are run. To just run annotation processing without compiling source code to class files, use the -proc:only option.

Writing your first annotation processor

Annotation processing is a form of meta-programming, that is, programming based on the structure of a program. However, for starters, let's consider a "Hello World" annotation processor:

import javax.annotation.processing.*;
import static  javax.lang.model.SourceVersion.*;
import javax.lang.model.element.*;
import javax.lang.model.type.*;
import javax.lang.model.util.*;
import java.util.Set;

@SupportedAnnotationTypes("*")	// Process all annotations
@SupportedSourceVersion(RELEASE_6)
public class HelloWorldProcessor extends AbstractProcessor {
    public boolean process(Set<? extends TypeElement> annotations,
			   RoundEnvironment roundEnv) {
	if (!roundEnv.processingOver())
	    processingEnv.getMessager().printNotice("Hello World");
	return false; // Don't claim any annotations
    }
}

To compile this program, add tools.jar to the classpath of javac:
javac -cp Path-to-tools.jar HelloWorldProcessor.java

(We plan to make adding tools.jar to the classpath more convenient in the future.) To run the annotation processor, use the new -processor option to javac:
javac -processor HelloWorldProcessor Foo.java

The output should include:

Note: Hello World

(HelloWorldProcessor can be run against HelloWorldProcessor.java; to do so, both tools.jar and the class files for HelloWorldProcessor need to be on the classpath.) This simple annotation processor doesn't actually examine the structure of a program, but it illustrates a few points about how to write and run annotation processors:

• Annotation processors are run by javac before the input source files are compiled.
• Annotation processors must implement the javax.annotation.processing.Processor interface to get registered with javac. This interface has methods that inform javac about what the annotation processor does, including what annotations it processes and what source version it supports.
• Extending the javax.annotation.processing.AbstractProcessor class is a convenient way to write a processor since annotations can be used to provide the value the methods return. When extending AbstractProcessor, the only method that needs to be implemented is process.
• A processor can get run multiple times in one javac invocation. If the test (!roundEnv.processingOver()) is removed, "Note: Hello World" will be printed twice. The javac infrastructure can call a processor's process method once per round. A round corresponds to analyzing a set of types.
• Processors get access to resources, like a Messager, by getting helper objects from different environments. The ProcessingEnvironment (the processingEnv field in AbstractProcessor) provides round-independent utilities while the RoundEnvironment argument to process provides round-dependent ones.

Future Topics

"Hello World" just scratches the surface of what annotation processors can do! For example, annotation processors can

• process annotations
• recursively generate new source files and class files
• analyze class files (try javac -Xprint java.lang.Object)

In future blog entries, I'll discuss these topics as well as providing a comparison between apt and JSR 269 annotation processing, best practices for writing and running annotation processors, suggestions on how to use JSR 269 visitors, and describing how the API copes with the expression problem. (2005-10-21 13:20:21.0) Permalink Comments [10]

Annotation Processing Forum Created To discuss annotation processing, consider using this newly created forum. Topics should include both apt-based processing as well as processing based on the forthcoming JSR 269. (2005-08-12 17:25:22.0) Permalink