The Former Weblog of Peter Ahé

I was recently interviewed by Jan for java.sun.com; where I said: Although most developers do not want to know how to use the Java Compiler API, it will provide a lot of indirect benefits. This was discussed on comp.lang.java.programmer, and I think it may have been misunderstood.

99% of Java developers will newer need to know anything about the Java Compiler API (JSR 199). A small number of Java developers will use the API directly. However, the software these few developers make is used by all Java developers. I'm talking about software such as IDEs (NetBeans, Eclipse, IntelliJ IDEA, etc), Java EE application server, build infrastructures such as Maven and Ant, and test frame works. All these pieces of software have one thing in common: they all need to invoke a compiler to translate Java source code to class files. However, 99% of all developers will not and should not care about how the compiler is invoked.

The compiler API becomes interesting for all developers when we start measuring build times in milliseconds instead of seconds (when you invoke Ant from NetBeans), or when start measuring deployment time in seconds, not minutes. These are the indirect benefits you will all notice, without learning how to use the API, as soon as your favorite tool migrates to the API.

If questions about the compiler API ever shows up on a Java certification exam, please have the examiners call me and I'll set them straight

That being said, if you are interested in compiler technology and want to hack on the compiler, please come and join us at https://openjdk.dev.java.net/compiler/.

I got a few questions from a reporter on the Kitchen Sink Language. He wanted to know what prompted me to start the project and what my goal is. Here is my reply:

The Java™ programming language is defined by the JCP. This is a good thing as it ensures that the entire community is heard. We should also be conservative when selecting which features are added to the language. Otherwise, the language may become a mess. This was explained in Evolving the Java Language by Bracha, Hamilton, and Reinhold.

We want the main branch of javac to be stable and primarily focus on implementing approved features. On the other hand, we must experiment and have fun to get a feel for new language features. Most people seems prefer evaluating new language proposals by using them on their own code, rather than reading abstract specifications and proposals.

As you may know, I'm the Java compiler (javac) tech lead. I see a lot of proposal for enhancing the language and our team has to turn down most. So how can we experiment? James solution is the Kitchen Sink Language which will be a forum for trying out crazy (and not so crazy) ideas for language enhancements.

Over the winter holidays, Rémi Forax implemented two different proposals for language enhancements. This is exactly the kind of community James and I would like to encourage so we were happy to see Rémi's work. On the compiler-dev mailing list, Rémi's work prompted Ted Neward to suggest that we do something to coordinate future experimental features. So I felt this was the right time to implement James' idea.

Now it is time for me to create a vendor branch so we can start experimenting.

Starting tomorrow, the Java™ compiler, javac, is available under the GPLv2 from openjdk.dev.java.net/compiler.

I cannot help you interpret the license because I am not a lawyer. For the same reason, nothing on this blog is legal advice. Sun's lawyers cannot help you either because that could create a conflict of interest. Be sure to read the license carefully, the use of the Java trademark is restricted. Retain your own lawyer if you have any legal questions.

We continue to accept patches to the compiler. Before you submit a compiler patch, I recommend that you read my rules of thumb about maximizing your chances for getting a patch accepted.

All that being said, I am really excited about this and it has been real hard not to be able to talk about this. I hope to see a lot of projects prototyping new programming language constructs, and tighter integration with IDEs and other applications that use a compiler. We have already seen very cool things such as the Jackpot project and performance in GlassFish. Since both projects are open sourced, relying on closed source may have been a problem for some members of their respective communities. However, any such concerns should be history now.

Earlier this week JSR 199 passed the Final Approval Ballot. This means that the JSR is approved and we just need to do the final release (which will be part of JDK 6). Managing a JSR has not been easy but fortunately Ian and Azar stepped in and helped me with the final paperwork. The JSR would not have been possible without the support from the entire compiler team and especially Jon, Seetharam, and Igor's contributions. Similarly, the feedback from the Sun-internal reviewers, especially the NetBeans team, has been invaluable. I should not forget to mention the expert group.

We celebrated locally by drinking a glass of champagne but since our team is spread across the world, I decided to further the celebration with a new look for my blog (please let me know if you notice any problems, I have only tested the look with the Firefox and Opera web browsers).

You may have noticed that I stopped posting my bi-weekly summary of compiler fixes. I plan to pick it up again when JDK 7 development takes off.

Right now is kind of a slow period and that is nice. Later today, JSR 199 should move to the final approval ballot (FAB). This is the last stage it needs to go through before the final release which is delivered in JDK 6. All JSRs including the umbrella JSR for the release must go through this process before we can ship the release.

The only changes that we will make to JDK 6 at this time is to fix show stopper issues. This means that most of my team is looking at either improving documentation, writing sample code, making it easier to build the compiler, or JDK 7.

Personally, I'm procrastinating on writing compiler documentation by fixing bugs to go into JDK 6 update 1 and JDK 7. So far I have lined up a little over twenty fixes. I'll give a summary when I know more about how you can get them. It is most likely JDK 7 b02 but we do not have weekly builds since we are focusing our release engineering and quality resources on JDK 6. So I don't know when.

Besides pruning my bug list, I'm also helping with figuring out which language features Sun would like to see in JDK 7. Clearly, we're looking at code abstraction but there is a lot of other ideas floating around. Another thing we are investigating is if it is possible make generic types a little more convenient. For example, code like this is pretty annoying:

Map<Comparable<? extends Number>, List<Number>> x =
    new HashMap<Comparable<? extends Number>, List<Number>>();

James Gosling has returned to the Java programming language and I really like his approach. I think the only problem we have is that we have too many good ideas and just one tiny release to put them into.

If there is anything you would like to see in such a guide, please let me know within the next few days.

UPDATE: unfortunately, it only became clear to me very recently that I would have to write the guide and it was almost too late for JDK 6. I have added a minimal version that includes links to the manual pages and all the relevant API (this should make it easier to locate the Tree API docs). We hope to improve the guide in the coming month so it is ready for the first update release to JDK 6. The good news is that that gives us more time to incorporate your ideas and I have opened the comments again.

6341023: (javac) Tree API: Tree.Kind should have mapping to interface
6341177: (JSR 269) Add IDE toolability API to JSR 269
6402506: (JSR 269) Add ProcessingEnvironment.getSourceVersion
6403466: (javac) javac TaskListener should be informed when annotation processing occurs
6406212: (JSR 269) Syntax of options should be better verified
6406771: (javac) Tree API clients need access to the compiler's line number table
6407257: (javac) javac locks up when encountering cyclic inheritance

Thanks to Joe, Jon, and Wei.

Also, the JDK will be build using -target 5 from now on. Previously, we shipped with -target 6 to allow thorough testing of the new verifier. However, we always planned to ship with -target 5 in Mustang RC. The reason for this is very simple: we want people to upgrade to Mustang as soon as possible. One problem we identified in JDK 5.0 was that customers running any kind of app server will run into a problem if the app server has a built-in compiler that doesn't understand the new class file format. Fortunately, there is a very simple solution to this for Mustang: use the old class file format.

6340951: (javac) Tree API: some literals are identifiers
6365980: (JSR 269) FilerException(null) doesn't throw NPE
6395981: (javac) JSR 199: JavaCompilerTool and Tool must specify version of JLS and JVMS
6395983: (javac) JSR 199: improve documentation of JavaCompilerTool
6396397: (javac) JSR 199: JavaFileManager should extend Flushable
6397044: (javac) JCModifiers.getModifiers() returns incorrect Modifiers set.
6397097: (javac) JSR 199: JavaCompilerTool.CompilationTask is clumsy with respect to "delayed"
6397104: (javac) JSR 199: JavaFileManager.getFileForOutput should have sibling argument
6397286: (javac) TaskListener calls are not protected agains user exceptions
6397348: (javac) JSR 199: supported options
6399894: (javac) JSR 199: WrapperJavaFileObject should extend new class WrapperFileObject
6400204: (javac) JSR 199: StandardJavaFileManager.getJavaObjectsFrom* should throw exception if argument is directory
6400205: (javac) JSR 199: getClassLoader(location) returns null if getLocation(location) returns null
6400207: (javac) JSR 199: JavaFileManager.list and unset location
6400208: (javac) JSR 199: failure mode for inferBinaryName
6400225: (javac) JSR 199: StandardJavaFileManager.getEffectiveLocation
6400267: (javac) JSR 199: specify the exact requirements for relative URIs
6401107: (javac) JSR 199: all methods must return non-null values unless explicitly allowed
6401277: (javac) JSR 199: javax.tools.ToolProvider.getSystemToolClassLoader throws UnsupportedOperationException
6401906: (javac) JSR 199: rename Wrapper* to Forwarding*
6401994: (javac) JSR 199: JavaCompilerTool.CompilationTask.run should not copy Runnable spec
6406164: (JSR 269) ElementKindIterator.hasNext() throws NPE when ElementFilter Iterable methods are used.
6407066: (javac) Paths code should not discard non-existent directories
6410184: (javac) JSR 199: javax.Tools.WrapperJavaFileManager and WrapperJavaFileObject are not reflexive
6410297: (javac) JSR 199: javax.tools.SimpleJavaFileObject.isNameCompatible doesn't work or has wrong spec
6410653: (javac) REGRESSION: javac crashes if -d or -s argument is a file
6411327: (JSR 269) Some typeUtils methods can result in an IllegalArgumentException and this is not documented.

Thanks to Joe, Jon, and Scott.

6345812: (JSR 269) annotation processing breaks with assertion, in Types.isAssignable() method.
6370653: (javac) javac does not use iinc opcode for postfix decrement of local variable
6376083: (JSR 269) should check annotation processor for compatability with specified source version
6376084: (JSR 269) Annotation processor infrastructure should check for unsupported options
6391649: (JSR 269) javax.lang.model.util.Elements.getAllMembers() does not return initializers
6392818: (JSR 269) Add isDeprecated to javax.lang.model.util.Elements
6397298: (JSR 269) RoundEnvironment.getElementsAnnotatedWith(anno) returns null, no empty set
6400986: (JSR 269) RoundEnvironment.getElementsAnnotatedWith() misses some elements
6401264: (JSR 269) Link Character.isIdentifier{Start, Part} to javax.lang.model.SourceVersion.isIdentifier
6407011: (javac) javac crashes in b78 with NPE in DefaultFileManager:293

Thanks to Wei, Scott, Joe, and java.net contributor dmytro_sheyko.

A very special thanks to Tim for doing an extra integration that allowed me to fix 6407011 in less than a week.

Earlier today Xiomara discovered a nasty problem with Windows and mixed case filenames. We had to rewrite the part of javac that handles files because of an upcoming revision to JSR 199. Good news: we caught the problem before the bits were pushed to java.net. Bad news: we didn't catch it soon enough so we had to back it out in the last minute.

If we're lucky, the changes we had planned for b77 will make it to b78.

Regardless, I'll keep posting what we are putting into the various builds as soon as we have completed our part of the release cycle. I think it provides an interesting view into our daily work.

5024091: (javac) (crash) AssertionError when StringBuilder is used with cldc.jar as bootclasspath
6236704: (javac) bad jar files ignored in extension and endorsed dirs
6295519: (javac) javac throws ZipException when you have invalid files in classpath
6306967: (javac) This code compiles but gives VerifyError when run
6327885: (javac) JSR 199: open issues
6346249: (javac) javac Tree API missing functionality
6346453: (JSR 269) Types.directSupertypes implementation returns an invalid list.
6346506: (JSR 269) Elements.getTypeElement(String s) will throw NPE for a unknown class.
6346973: (JSR 269) directSupertypes(Mirror m) causes java.lang.StackOverflowError
6348193: (javac) AS8.1 UR2 BAT test failure with "javac"
6350057: (JSR 269) visitVariableAsLocalVariable is called on enum's parameter of valueOf(String name)
6358024: (javac) TaskListener should be propagated between processing rounds
6358166: (JSR 269) -verbose reports absurd times when annotation processing involved
6361619: (javac) AssertionError from ClassReader
6366196: (javac) regression in CharacterRangeTable format
6392058: (javac) rename internal class Diagnostic to JCDiagnostic
6392118: (javac) mismatch between JavacTaskImpl.context and JSR 269
6394563: (javac) REGRESSION: javac ignores -nowarn switch in 1.5.0_06 for deprecation warnings
6395269: (javac) Paths should use List<File> or List<URI> instead of List<String> to represent a path
6395974: (javac) files are parsed even after failure to find annotation processor is reported

Thanks to Joe, Jon, Neal, Nishant, Scott, and Wei. Especially a big thank you to Jon for helping me getting JSR 199 ready for b77. We pulled some later hours on that one... Unfortunately, this left the wildcard bug I mentioned two weeks ago unfixed.

For those of you using jcov, send happy thoughts to Wei for fixing the javac regressions that broke jcov.

4952629: (javac) REGRESSION: javac generates a spurious line number entry on } else {
6194785: (javac) ParameterDeclaration.getSimpleName does not return actual name from class files
6214959: (javac) Compiler fails do produce error message with ODD number of import static
6221045: (apt) apt -print option shoud not print abstract modifier on interfaces or annotations
6325201: (javac) make erasure non-destructive
6341072: (javac) printError does not result in nonzero return code for javac when processing
6341534: (javac) PackageElement.getEnclosedElements results in NullPointerException from parse(JavaCompiler.java:429)
6351767: (javac) javax.tools.JavaCompilerTool.getStandardFileManager().list() treats directories as package
6352136: (javac) TypeMirror.asElement().getEnclosedElements() throws NPE
6357331: (javac) NPE from JavacTask.getElements() after calling CompilationTask.run
6358955: (javac) javax.tools.JavaCompilerTool.getStandardFileManager().getFileForInput(directory) should throw IAE
6359313: (javac) error compiling @Annotated package-info.java with active annotation processor
6359949: (javac) @Override of static shouldn't be accepted (compiler shouldissue an error/warning)
6362067: (JSR 269) Messager methods do not print out source position information
6363481: (javac) javac usage suggests -A is valid but ignores it
6365040: (JSR 269) Support multiple processor arguments for javac -processor options
6370261: (apt) apt must pass through shared JSR 269 javac options
6372395: (javac) javax.tools.SimpleJavaFileObject.openReader() throws undocumented exception (NullPointerException)
6374357: (javac) PackageElement.getEnclosedElements() throws ClassReader$BadClassFileException
6381876: (javac) please remove PackageDef from com.sun.tools.javac.JCTree
6384539: (javac) crash: test/tools/javac/IllegalAnnotation.java
6384542: (javac) crash: test/tools/javac/versions/check.sh
6389812: (javac) com.sun.source.util.TaskListener has an unnecessary import to com.sun.tools.javac
6390467: (JSR 269) Fix typos in javax.lang.model.util.ElementScanner6 javadocs
UPDATE: 6375016: (javac) Elements.getTypeElement() sometimes returns null.
UPDATE: 6378728: (JSR 269) -proc:only generates classes if no processors found

Brought to you by Joe, Jon, Wei, and yours truly. Sometimes when we fix one thing, we break another thing. I broke wildcards in b71 and had hoped to include the fix in b75, but the fix isn't ready yet.

UPDATE: I forgot to mention that 6359949 was fixed by mondo from java.net.

4903103: Can't compile subclasses of inner classes
5009484: Compiler fails to resolve appropriate type for outer member
5073060: Package private members not found for intersection types
5086027: Inner class of generic class cannot extend Throwable
6227936: Wrong type of inherited method using specialized type parameter
6265400: Javac should be shielded from client code errors in JSR 199
6266772: crash: java.lang.AssertionError at com.sun.tools.javac.comp.Lower.access(Lower.java:958)
6343486: Change java.util.Service reference to sun.misc.Service in JavacProcessingEnvironment.
6345974: JCPrimitiveTypeTree.getPrimitiveTypeKind() inappropriately throws an AssertionError for "void"
6350124: javac -s does not have the generated source files
6357979: Clash of old "-s" and new "-s dir" options in javac
6359951: Class field usage in generic
6367459: javax.tools.JavaCompilerTool.run( (JavaFileManager) null, , , ) doesn't use default JavaFileManager
6384318: Missing files from tools.jar

Jon fixed half of these.

b65:

4941882: incorrect inference for result of lub(int[], float[])
6209839: Illegal forward reference to enum constants allowed by javac
6231847: crash: java.lang.NullPointerException at com.sun.tools.javac.comp.Attr.visitNewClass(Attr.java:1352)
6253161: Incorrect position of serial warning in anonymous class
6313120: back out changes for 4903103
6342411: Add bridge method to allow reflective access to public method in non-public class

b67:

4954546: (compound) unverifiable code for method called from ?: expression with inferred
5003431: java.lang.Object cannot be dereferenced
5034571: Wildcard capture must use the bounds of the formal
5080917: lub computes strange compound types
6213818: crash: java.lang.NullPointerException at com.sun.tools.javac.code.Types$SubstFcn.subst(Types.java:20
6214965: crash: java.lang.AssertionError at com.sun.tools.javac.jvm.ClassReader$AnnotationDeproxy.findAccessM
6215213: crash: java.lang.AssertionError at com.sun.tools.javac.code.Types.containedBy(Types.java:792)
6218229: crash: java.lang.NullPointerException at com.sun.tools.javac.comp.Check.checkCompatibleConcretes(Che
6245699: Missing bridge for final method (gives AbstractMethodError at runtime)
6268476: Arrays and super wildcards broken
6273455: crash: java.lang.StackOverflowError at com.sun.tools.javac.code.Types
6320612: Unwarranted unchecked warning for inner class of "wildcard" type
6330920: Poor error reporting situation
6358534: Cast involving raw superclass not recognized as unchecked
6358786: Doccomments are not returned from Tree API

b71:

4897301: Improve wildcards diagnostics: "capture of ?"
6207386: Undecidable type system leads to crash
6240565: Unboxing, arrays, and type variables doesn't work
6348760: crash: java.lang.AssertionError at com.sun.tools.javac.comp.Lower.abstractLval(Lower.java:1853)
6359106: Valid generics code does not compile
6365854: javac crashes when compiling against an annotated class
UPDATE: 6225935: "import static" accessibility rules for symbols different for no reason
UPDATE: 6372401: javax.tools.SimpleJavaFileObject( String ) doesn't process predefined file extenstions
UPDATE: 5049523: Inference should compute glb of type arguments

If you need one of those fixes, please download the latest version of Mustang from Java.net. You will find weekly builds of the JDK here as well as the sources. Before posting a build on Java.net we run a lot of tests, so a build will first be available a few weeks after the engineers are done. For example, today, build 67 (b67) is available and b71 will propably be available 71-67 = 4 weeks from today.

The Tree API exposes javac's abstract syntax trees through a set of public interfaces. We hope to include this in Mustang in due time.

UPDATE: The exact procedure to get access to the Tree API is as follows:

1. Go to https://jdk.dev.java.net/
2. Request the jdk.researcher role on the Membership Request page.
3. Wait for approval.

You now have access to all the projects under jdk-research including the Tree API.

interface List<E> {}

class Test {
    <T> void merge(List<T> l1, List<T> l2) {}
    void test(List<? extends Test> list) {
	merge(list, list);
    }
}

The compiler says:

../Test.java:6: <T>merge(List<T>,List<T>) in Test cannot be applied to
(List<capture of ? extends Test>,List<capture of ? extends Test>)
        merge(list, list);
        ^
1 error

This is confusing because how is List<capture of ? extends Test> different from List<capture of ? extends Test>?

I'm trying to improve this diagnostic and am considering this error message:

../Test.java:6: <T>merge(List<T>,List<T>) in Test cannot be applied to
(List<E#0 (capture of ? extends Test)>,List<E#1 (capture of ? extends Test)>)
        merge(list, list);
        ^
1 error

It should now be clear that List<E#0 (capture of ? extends Test)> is different from List<E#1 (capture of ? extends Test)>.

UPDATE: what I would really like is something along these lines:

../Test.java:6: <T>merge(List<T>,List<T>) in Test cannot be applied to (List<E#0>,List<E#1>)
  where E#0 and E#1 are type variables
    and E#0 extends Test (capture of ? extends Test)
    and E#1 extends Test (capture of ? extends Test)
        merge(list, list);
        ^
1 error

However, I think this would require too much restructuring and prefer to defer this to Dolphin. The above is a fairly simple change to the compiler and is possible for Mustang.