James Gosling: on the Java Road

« Boiling oceans | Main | Fly low, fly fast,... »

20060911 Monday September 11, 2006


The Black Hole Theory of Design

The past few weeks have been pretty entertaining: there's the final crunch to getting JDK6 out the door (with the side-thrill of coordinating with the tea-leaf-reading exercise that is Windows Vista), bringing closure to the open source debate, and the most fun: figuring out the plan for JDK7. This last is the geek version of writing up a Christmas Present wish list.

The list is mostly being put together by Danny Coward and Mark Reinhold, with a lot of kibbitzing from a cast of thousands. We do surveys of partners, fold in statistics from the bug voting system, stir up with our own opinions and then run the JCP debate.

There's an entertaining little firestorm going on over closures. Neal Gafter, Gilad Bracha, Peter Ahé and many more are fighting it out. The really heated debate is all in email and I'm incredibly far behind in reading it (by several hundred messages).

I have somewhat mixed feelings about closures: they are pretty complicated. But they're an instance of what I think of as the Black Hole Theory of Design. I have a really strong memory from years ago of Guy Steele saying roughly "Lisp is a Black Hole: if you try to design something that's not Lisp, but like Lisp, you'll find that the gravitational forces on the design will suck it into the Black Hole, and it will become Lisp". [Guy doesn't actually remember making this remark, but he does say it sounds like the kind of flip comment he would make; I could also be totally mis-remembering who said it, but I haven't found any quote like it through Google].

Closures feel to me like one of these design Black Holes. When inner classes were designed, we wanted to avoid the complexity of closures. But this brought about oddness and tension of it's own that has left me less than happy with inner classes. So, should we just give in to the force of gravity and go the rest of the way? The debate's afoot. It's going to be an entertaining year. Permalink Comments [39]

Comments:

hi james,

TYPO : "There's an entertaining little firestorm going on over clusures" -- you mean "closures" ?

I look forward to Dolphin(JDK 7) feature list callout.

Meanwhile, does anyone know why logging to the XMLFormatter's log records do not include the chain of exceptions(cause and its cause) ? (whereas if I target logs to the SimpleFormatter, all the Exception's causes print themselves in the hierarchy).

BR,
~A

Posted by anjan bacchu on September 12, 2006 at 12:22 AM PDT #

I'm neither for or against closures but what I would like to see done is for us to sit down and measure the cost/benefit of the feature.

How much more complex does the compiler/language get and how many times is the addition useful enough to justify that increase in complexity?

Personally speaking, I think Java has been moving down the line towards gaining esoterica for the sake of some corner cases or adopted language level constructs where API additions would be sufficient for 80 percent of cases.

That is not something I ever wanted for Java especially as we started out with the mantra about taking all the cruft out (unlike C++) and having a nice explicit language with no dark corners (auto-boxing anyone?).

I'd certainly like to hear your views on the tensions in inner classes. I'd also be interested in hearing about what you intended inner classes to be used for and comparing that with how they are used by the masses.

Posted by Dan Creswell on September 12, 2006 at 01:25 AM PDT #

I like inner classes compared to closures because I find inheritance of multiple methods and fields better and because it is one concept, the class. In summary:

Closure = Poor Man's Inner Class

I have suggested an RFE that shortens the syntax:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6389769

The reason for suggesting the shorter syntax is that this seems to be a large driver in the proposals for closure and alike.

Posted by Howard Lovatt on September 12, 2006 at 01:37 AM PDT #

Java is much poorer for the lack of closures, first-class functions, and higher-order functions. I say yes, they are worth the complexity - and the speed of GHC and Ocaml shows that they can be done without excessive expense.

Posted by Julian Morrison on September 12, 2006 at 03:20 AM PDT #

Closure = Poor Man's Inner Class

How can this be when inner classes have a subset of the functionality of closures?

Posted by Alex on September 12, 2006 at 03:35 AM PDT #

Closure = Poor Man's Inner Class

Actually that should should be:

Inner Class = Poor Man's Closure

I don't really think the Java community understands closures. I really hope closures are going to make it in the language. I really hope generics will get fixed.

Guys ... do something right for a change. Lets fix those oddities that make Java complex.

Posted by Alex Nedelcu on September 12, 2006 at 03:57 AM PDT #

I have nothing agains closures... neither destructors. The question is will it be Java??? In the last 10 years there were lots good ideas and I'm sure if we could restart java again it would be "different". Personally I'd like to get rid of primitives. Let compiler optimise my mutable "primitive" Integer, Boolean and so on. So we could get "fixed" generics (no primitives - no autoboxing). Where it's appropriate compiler convert this object to the primitive and then with help of escape analysis push it to stack etc, etc, etc... May be it's just a time to make "another" java? Please don't do it incrementally... It will just make a mess... Cheers.

Posted by Dim on September 12, 2006 at 04:16 AM PDT #

I'd really like to hear your views on it James, in terms of what you think the major costs/benefits are. Obviously there's a camp of people who are really for it in terms of developer productivity, and then there's other the guys who don't want the language cluttered with supposed empty wins. I think it's a lot of empty baloney that the so-called productivity camp throws at the discussion. It seems to be mostly "it just is BETTER ok? And plus, Ruby does it."

Posted by Ivan on September 12, 2006 at 04:44 AM PDT #

It seems that Java is approaching the maximum complexity that a language should have - closures and function types add to the complexity of Java to the extent that I think they perclude other potential additions.

Posted by Mike Atkinson on September 12, 2006 at 05:35 AM PDT #

It seems that Java is approaching the maximum complexity that a language should have - closures and function types add to the complexity of Java to the extent that I think they perclude other potential additions.

Posted by Mike Atkinson on September 12, 2006 at 05:36 AM PDT #

I agree with Dim: get rid of "primitive" types like Integer, Boolean, String (make it like C#). I don't know about closures, there are many other simpler things that should be fixed in the language before that (like adding multi-line verbatim string literal).

Posted by Fuerte on September 12, 2006 at 05:53 AM PDT #

Guy said something similar at a talk at Brown, maybe in 1994. Does 2006 Lisp look like 1959 Lisp? That's pretty amazing.
I'm slowly deciding that I'm against closures. The main argument for closures I've seen is that closures let us create something like anonymous inner classes more efficiently. I use anonymous inner classes as markers for prototype code that needs refactoring; eventually the anonymous inner classes get to grow up to be first-class critters. If anonymous inner classes are the larval stage of named classes, I'd treat closures as an egg stage. I don't think I need it.
My other concern is that closures may never get refactored into named classes. That'll lead to the same non-evolutionary path that Smalltalk/ObjectiveC categories led to. Categories are fine while the original development team is around, but a year or two after they're gone the dead code crushed the live code.

Posted by David Walend on September 12, 2006 at 06:14 AM PDT #

We say in Nepal that god took out every beautiful organ of all the being and made a Camel so i fear java will turn out to be a Camel in few years

Posted by javaniraj on September 12, 2006 at 06:24 AM PDT #

I came to Java from a Smalltalk background, and I have to say that I just don't see Java needing closures. For one, while they're an excellent tool for an advanced developer, they are an excellent way to hang onesself for beginners, and certainly will complicate the language (and APIs). They do make life easier on the collections methods (inject / select are fabulous), but they tend to be overused--I once had a developer make all of his "refactoring" into named closures, rather than methods! I do agree with Dim and Mike, though--there are many, many other things that would provide a much better ROI: getting rid of primitives, removing all of the cruft from the runtime, etc.

Posted by Chris Reigrut on September 12, 2006 at 07:52 AM PDT #

Forget closures. They're not really for statically typed languages anyway. I agree with the others...get rid of primitive types! They should have never been included in the first place. Finally, provide a way for the language to auto-gen getters/setters without me having to do it explicitly. IDE's make this less painfull, but I'd rather not have my code cluttered with that garbage and it's frankly a waste of time for me to do something that can be so easily handled by a machine and code.

Posted by Sam on September 12, 2006 at 08:14 AM PDT #

jesus christ java programmers are stupid. when ruby really starts taking off in 12-18 months you're all going to be screwed and you deserve it because you're stupid.

Posted by warren henning on September 12, 2006 at 08:24 AM PDT #

Another comment: Closures would be good if they could be used to get the equivalent of C# "using" construct, for example (auto execution of .close() at finally).

Posted by Fuerte on September 12, 2006 at 08:36 AM PDT #

Closures are not really for statically typed languages? The commenter above has perhaps not noticed Ocaml, ML, or Haskell.

Posted by Julian Morrison on September 12, 2006 at 08:39 AM PDT #

Well, that's a nice comment from Guy Steele, one of the creators of... Scheme!

Posted by Antonio on September 12, 2006 at 09:13 AM PDT #

Looks like you guys are going to support a Java implementation of Ruby, so why not include a full implementation of Lisp also? MS is doing that with L# and it's a good idea - Lisp is a brilliant language that lots of younger developers are waking up to, thanks to Paul Graham among others, but its major drawback is the lack of standardization and libraries. Lisp with the power of the Java libs would be a teriffic addition to the JDK, and would head .NET off at the pass, so to speak. Mindshare is important, and .NET is gaining that by supporting different languages with the CLR, Sun should do the same with Java.

Posted by Byron on September 12, 2006 at 09:54 AM PDT #

I thought that from day 1, Java didn't allow multiple superclasses because of the inherent complexity? We can argue until we're blue in the face about how interfaces are only a partial solution, and how allowing multiple superclasses would really make Java a more productive language. It's really the same thing with closures- yes, they could potentially improve the language IF they were done right (and that's a big if), but it wouldn't be java anymore. Just understanding closures requires a different way of thinking about programming than most java programmers do right now. Would java programmers be benefited by expanding their minds and learning these ideas? Yes, definitely. Should java programmers' minds be expanded by expanding the java language? No, definitely not. The paradigms of closure-oriented programming are just too different. Heck, if aspects can't be in the same language and need to go into AspectJ, then closures definitely shouldn't be in there.

It's too late for java to start introducing radically different ideas like closures. Java has (I think) done a lot for improving language design on the whole; it's time to pass the torch to the next language that will get it right the first time.

In the meantime, making things more natural (by getting rid of primitives) instead of less natural (by hacking closures, because I can't imagine closures not being a hack this late in the game) would be a good step.

Posted by Liam on September 12, 2006 at 10:09 AM PDT #

OO and functional programming do not mix well. Closures is definitely a hack, a kludge in java. I think java as OO language should continue to the true OO way, and fix problems that is has now? mentioned many times here on this thread. As for closures and functional programming. It is time for Sun to embrace new functional language with closures, higher order functions, type inference etc. And make it work seamlesly with java. In other words no FFI should be nesessary to talk from one language to another. That will make both camps, OO and FP happy. And you will not introduce unnesessary complexity into java.

Posted by Vagif Verdi on September 12, 2006 at 10:18 AM PDT #

I think adding closures is wrong focus. Who asked for it? Does it really solve a problem which can't be solved otherwise? I think it doesn't. What I think would be more important to address in the language is the support for non-null-references. Almost every day I face a NullPointerException in code coming from all sources (including mine, of course). There is currently no support to enforce that a reference may not be null in Java. This is my number one issue and hence feature request for Java. Regards, Christian Schlichtherle

Posted by Christian Schlichtherle on September 12, 2006 at 10:20 AM PDT #

Aren't Neal Gafter, Gilad Bracha, Peter Ahé all FOR closures? How are they fighting it out? Oh, I get it… someone at Sun has decided to cram closures down our throats, and all that remains to fight over is what form they take. Why isn't there a way to vote AGAINST a feature request?

Posted by Doug Erickson on September 12, 2006 at 11:11 AM PDT #

Closures are simple. They were tricky to get right in the Maclisp days, but with the implementation of Steele's Scheme compiler the techniques for making lexical closures became well-known and really, not hard to implement or conceptualize. Inner classes are an arbitrary bogosity in order to confer *some* of the power of closures, but not all, to make things easier for a certain kind of joe coder who fits Java's user profile; and this focus on ease rather than simplicity makes things more complex, and is the likely source of your "oddness and tension". Closures are an important step for Java. If the language doesn't take the training wheels off and soon, it would doubtless languish in joe-coder limbo before being completely supplanted by .NET (where closures and even lambda-expressions have been announced as a forthcoming feature for a few months now).

Posted by Jeff Read on September 12, 2006 at 12:46 PM PDT #

Instead of adding just another extension to the java language, why not enhance the way innner classes are implemented by the compiler/runtime/jit.

Inner classes add quite a lot of runtime overhead, enlarge programs, ... - could'nt this been done in an easier way?

Posted by Linuxhippy on September 12, 2006 at 02:43 PM PDT #

I should explain my:

Closure = Poor Man's Inner Class

Glib comment. I see the advantages of inner classes as:

  1. One concept, the class
  2. Can be Serializable and/or Clonable
  3. Can have multiple methods
  4. Can have fields
  5. Can inherite from a class, e.g. a partial implementation
  6. Retains single dispatch semantics

On the down side I see:

  1. Longer syntax
  2. Locals need to be final
  3. Implemented by a seperate class
  4. Need to declare checked exceptions

But if you look at the advantages of closures, these are alsmost seperate issues and are not really dependent on closures and could be added to inner classes without introducing a new concept. I.E.:

  1. Come up with some short spiffy syntax, e.g. http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6389769
  2. Auto box non local finals into a tupple, like C# 3.0 is going to do
  3. Add support for inner classes into class file format so that they don't need to be in a seperate class file.
  4. Interoduce an extension to generics to enable the declaration of multiple checked exceptions easily, e.g. varargs like: <code> interface Closure< R, Es extends Throwable... > { R call() throws Es; } </code>. Which allows Es to refer to a list of zero or more Throwables

      (That is assuming that their is broad agreement that these really are problems with inner classes. I suspect many people are quite happy with how inner classes are right now.)

      Posted by Howard Lovatt on September 12, 2006 at 05:17 PM PDT #

This is an interesting discussion, but there's one problem: the debate is largely subjective, full of opinions and short on data. Do you remember the formatting/indenting/brace wars? Who won that debate?

It's hard to objectively measure if closures are better or not. IMHO, Java (and predecessors) rose to prominence because they solved important, objective problems. Specifically, Java addressed reliability and robustness concerns while delivering excellent runtime performance. That's something everyone can agree is a good thing.

I am all for open debate, but I wonder if our time is better spent solving bigger problems. Maybe we should identify those problems first before trying to solve them.

Posted by Madhu Siddalingaiah on September 12, 2006 at 06:54 PM PDT #

Does not having closures in Java preclude other languages that run in the JVM from having them?

If not then, can the two languages still interact?

I'd like it to be easier for the JVM to work with other languages seamlessly -- including (or perhaps especially) ones that support closures.

I'd like to have the portability of running in a JVM as well as the wealth Java libraries available to me with more language choices.

Posted by Bill Robertson on September 12, 2006 at 07:00 PM PDT #

I never imagined that Java would have so many idiots.

Posted by Anonymous on September 12, 2006 at 09:24 PM PDT #

I think it's funny how people attempt to troll threads about things they dont understand at all. Agreed there are other issues to address; I'd like to see Java tackle bigger problems. A closure implementation feels highly tangential to me in Java's current state.

Posted by Ivan on September 12, 2006 at 09:55 PM PDT #

Mr. Gosling you must acknowledge ... your "black hole" is C/C++, new current and future Java features are attracted by the C/C++ "black hole" ;)

Posted by Jose M. Arranz on September 13, 2006 at 01:07 AM PDT #

There are other things that could be added to Java - value classes, true immutables, square matrices to name a few. All of these would enable Java to do things that are difficult and/or inefficient at present.

Posted by Mike on September 13, 2006 at 02:52 AM PDT #

I think adding closures to Java is a must! People are trying to do more and more dynamic style programming and the code that is being written and the frameworks written to support these ideas while workable in some instances add to a big bloat of the programs that want to use them. As for Java having high performance, etc., you need to remember that when it came out it was not highly performant. Smalltalk, a fully dynamic, heavily closure based system performed much better. To this day ST still performs very fast. It uses the Smalltalk version of closures extensively including for iteration. Closures fitting into Java has nothing to do really with mismatches between programming models. In addition to closures, I'd like to suggest that interfaces be extened or an additional construct like the mix-in idea from Lisp or Ruby be added so that I can share and mix-in implementation to a class instead of having to implement an interfacel

Posted by Sam Griffith Jr on September 13, 2006 at 04:59 AM PDT #

Sadly, I think Java must be freezed in 1.4 version, primitives removed, and the vars inference. Well, I like the foreach loop. I think the main target of Java is be simple. And, allowing other languages to run in the JVM, and interact with Java, allow we use all cool features of other languages.

Posted by Peter Parker on September 13, 2006 at 10:30 AM PDT #

You could just add better dynamic-language support to the JVM, and let other languages like Groovy and JRuby handle cases where you want those powerful language features.

That way it wouldn't clutter up Java. I agree closures are a great thing. But you could keep the Java syntax simple and when people want that power, jump to another language that still boils down to the same thing on the JVM. Java's simplicity _is_ a good thing.

If you want to change Java, I agree- fix primitives. And possibly Properties vs. getters/setters.

Posted by Tom on September 13, 2006 at 11:50 AM PDT #

For those that would like immutables to be added to Java you can vote here:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4617197

Posted by Howard Lovatt on September 13, 2006 at 09:20 PM PDT #

is the guy steele quote maybe confused with the old "Greenspun's Tenth Rule of Programming": "Any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp." (for example a program to break your captcha would require a subset of lisp!)

Posted by lb on September 14, 2006 at 04:06 PM PDT #

The quote is referenced by Paul Graham, a must-read I guess: http://www.paulgraham.com/quotes.html "Greenspun's Tenth Rule of Programming: any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp."

Posted by Bernd Eckenfels on September 14, 2006 at 09:38 PM PDT #

Post a Comment:

Comments are closed for this entry.