Java and security bits

JCP Early Draft of JSR 294 Now Available

As you have have already noticed, the JCP Early Draft specification of JSR 294 is now available for download from the JSR page.

In short, it consists of two chapters of the Java Language Specification that have been updated to cover Superpackages. There are change bars to highlight everything that has been touched. Then there are draft revisions for the Java Virtual Machine Specification. Finally, updates to the Java core reflection APIs. If you have followed the earlier explanations here or the expert group discussion, none of this should be too surprising.

If you have any comments about the draft specification, please send them to jsr-294-comments@jcp.org so that Alex and I can collect them and discuss them with the expert group. BTW, we are also working on the implementation, chiefly Jon for the javac changes. We will make the code available on the OpenJDK Modules project once it is ready.

Posted at 17:50 Nov 21, 2007 by Andreas Sterbenz in Modularity  | 

Source Code for the OpenJDK Modules Project

We just posted the first source drop of the OpenJDK Modules project on the project page. The Modules project covers the implementation of JSR 277 and JSR 294 and you now have a chance to get a really early look at it, try it out, and contribute to it.

If you want to know what it is all about and how it works, please read the docs on the project page, in particular the samples and the getting started pages. As the samples show, it is really easy to create and run a Module, even from a remote web server. It is fast too:

% time java -repository http://openjdk.java.net/projects/modules/samplerepo/ -module dep
Module 'dep' calling module 'hello'...
Hello world from module hello 1.0
0.34u 0.18s 0:00.81 64.1%

In other words, it takes less than a second to start the Java VM, download 2 minimal modules from a remote web server, and execute them [1]. (This is on a three year old Opteron 248 running Solaris; Java warm in the page cache) Note the disclaimer about untrusted code at the bottom of the samples page, though.

In general, I am really happy with the size and complexity (or lack thereof) of the JSR 277 implementation. The code that I call the core of the implementation - module initialization and class loading - is less than 1200 lines of code (see src/share/classes/sun/module/core). And that includes the big GPL copyright headers in all the files. Of course the implementation is not complete and will grow somewhat, but it already handles module initialization, including optional dependencies and reexport, versioning, dependency cycles, custom import policies, shallow validation, etc. In other words, all the core features.

Then we have the mailing lists:

• modules-dev@openjdk.java.net: Technical discussion about the implementation of the Modules project

• modules-discuss@openjdk.java.net: General discussion of the Modules project and how to use modules

If you are interested in the conversations of the developers working on the Modules project, in discussions about how to use JSR 277 and 294, or if you want to actively contribute, I encourage you to join.

Now time for a few random notes:

• this is a live development snapshot. It is not a beta. Nothing is complete and polished.

• it is source code for developers. No binaries at this time.

• not much of the JSR 294 implementation is included, because not much exists so far. This will change.

• the way to get the source is by downloading a ZIP file that includes all source code in the j2se workspace. That's why it is 68 MB, not because the Modules implementation is that large. We expect to get something more convenient - such as a Mercurial repository - before too long.

• all the new code in the Modules project is under the GPLv2 (plus classpath exception).

• regression and unit tests are included. More about that in a future post.

• before there is any confusion, the 1 second startup time does not mean we have made any fundamental changes to the JRE, that is done by another project. What the Modules sample program shows is just how fast Java is when you run small and efficient applications and systems.

Posted at 00:00 Jun 28, 2007 by Andreas Sterbenz in Modularity  |  Comments[5]

Nested Superpackages Restated

A few weeks ago I posted an entry that explained what superpackages are about. This is a continuation that deals with nested superpackages. If you have not read the first post, read it before continuing. As a reminder, the short version is: a superpackage is a language construct for information hiding [1].

With that in mind, let us explore what a nested superpackage is: a nested superpackage is a superpackage contained within another superpackage. Since a nested superpackage is still a superpackage, what I said about top level superpackages also applies to nested superpackages. In other words, a nested superpackage is a language construct for information hiding.

Let's look at what this definition says about what a nested superpackage is not: it is not a namespace construct. The Java source language does not recognize multiple types of the same fully qualified name today. Adding new access control mechanisms does not change that. Of course, the other things I mentioned which a superpackage is not also still apply: no changes to the type system or to the way you package your applications.

That states what nested superpackages are and what they are not, but it does not explain why they are useful. But that is pretty simply: they are an access control mechanism and as such allow you to separate code within one project (i.e. one top level superpackage) and to hide implementation details. A way to visualize it is to imagine a bubble representing your project and floating within it all your classes and interfaces. Often you will want to organize those classes into subsystems - a bubble within the bubble. If a subsystem is too large for a single package, you need something else. That is where nested superpackages come in. One example of a project where this structure will be very useful is the core of the JDK, as I explained in this message to the JSR 294 expert group, but there are certainly many other projects with similar needs.

The final thing to say about nested superpackages is that (as described above) they are a way to organize the internals of a component, not a way to aggregate code from multiple components. You would use a deployment module solution such as JSR 277 to do that. The guidance is that if a subsystem is tightly coupled with the rest of a project, it should be a nested superpackage. If it is effectively an independent unit and reusable, it should be its own component as a top level superpackage. Other components can import it using JSR 277. The distinction may seem arbitrary and impossible to make, but it is not much different than deciding whether to organize your code into one, two, or more packages. With a bit of experience, it comes natural to developers.

That was quite a bit of text, but the one thing to remember is that a nested superpackage is a superpackage contained within another superpackage. In other words, a nested superpackage is a language construct for information hiding. Really. Nothing more to it than that.

[1] If you don't like the term information hiding just replace it with access control in this entry. In this context it is a good approximation as information hiding is the concept and access control the mechanism to implement it.

Posted at 11:20 May 30, 2007 by Andreas Sterbenz in Modularity  | 

Superpackages Restated

There seems to be some confusion about what superpackages are about. My one sentence explanation is: a superpackage is a language construct for information hiding. Let's examine that statement.

"Information hiding" is fancy way of saying "keeping implementation details private." In Java, we have access control modifiers to achieve that: private, default (aka package private), protected and public. But sometimes "public is too public".

I also said "language construct." That means something that is part of the Java language, defined in the Java Language Specification and enforced by all Java compilers (e.g. javac). It will also be part of the Java Virtual Machine Specification so that it is enforced at runtime by the JVM.

Let's look at all the things I did not say: no changes to the type system. No changes to the compile time and runtime access control architecture (just a small extension). No changes to how you must package your apps: if you had organized your application into 6 "components" before, each with its own JAR file, you most likely will do the same thing after migrating to superpackages (see note at the bottom).

In other words, a superpackage is a language construct for information hiding. Really. Nothing more to it than that.

But if that is still too abstract, how about this version: superpackages define an extension to the Java access control mechanism: a new access level for classes, which is wider than "package private" but narrower than "public". One could call the new access level "superpackage private", but that is potentially confusing so please don't use that term.

PS: a runtime infrastructure for component based applications is also desirable, see JSR 277. But that is very much orthogonal to the information hiding objective of superpackages.

Posted at 02:13 May 08, 2007 by Andreas Sterbenz in Modularity  |  Comments[8]

Superpackage strawman and the JSR 294 mailing list

After a lot of internal discussion, Alex and I recently started the expert group discussion for JSR 294 (Improved Modularity Support in the Java Programming Language). Not an earth shattering event, but what may make this interesting to you is that we decided to make the expert group discussion publicly readable via a web archive and an observer list that anyone can subscribe to. You will not be able to post to the expert group list, but you can read all the discussions from now on. It's all out in the open.

Secondly and related to that, we also posted our strawman proposal for superpackages. The concepts will not be news to you if you read my earlier postings. Alex and I think that it is a straightforward way to address limitations in information hiding in the Java language and so far the expert group agrees.

Finally, we are presenting a session at JavaOne this May. It is called Java Language Modularity with Superpackages (abstract). Stanley will also present a session JSR 277: Java Module System plus we will all do a joint Modularity BOF that covers both JSR 277 and 294. We hope to see you all there.

Posted at 23:53 Apr 05, 2007 by Andreas Sterbenz in Modularity  |  Comments[9]

Superpackages in JSR 294

The Java language modularity concepts proposed by JSR 294 have recently attracted some attention. I will attempt to clear up a few misconceptions and clarify the relationship of JSR 294 development time superpackages with JSR 277 deployment modules.

The basic ideas are described in the JSR proposal and Gilad's blog entry. Although these pages do not go into much detail, they capture the underlying concepts very well and I encourage you to read them. I will try to explain the background and fill in some of the blanks. Let me stress that this is just the initial proposal and as always the actual specification will be the result of the expert group discussion and may end up looking differently.

Background

JSR 294 deals with information hiding. Information hiding refers to the practice of keeping implementation details concealed while allowing access to published APIs. This prevents consumers from relying on implementation internals and makes it possible to change aspects of the implementation without affecting consumers.

In Java, we achieve information hiding by declaring types (classes and interfaces), methods, and fields with non-public access modifiers. For example, a class with default access (aka package private) or a private method. The access control rules are defined in the JLS and JVMS and are enforced by javac at compile time and the virtual machine at runtime. These low level mechanisms are very reliable and also serve as the foundation of Java platform security.

All this works fine as long as a project fits into a single package. Once the project grows beyond that, you may find yourself forced to make implementation classes public in order to access them from multiple packages in the project. That loses the benefits of information hiding, which is clearly suboptimal and a long-standing complaint that often comes up in language discussions in forums such as JavaOne.

Goals

When we decided to address these issues, we had the following goals in mind:

1. extend the access control model defined by the JLS and JVMS to accomodate projects larger than a single Java package

2. the extensions to the access control mechanisms should be simple and consistent with the existing model

3. access control information should be declared in Java source files so it can be processed and enforced by javac.

4. enable the new model to serve as the basis for deployment modules, but it should also be usable on its own without requiring modifications to the deployment mechanisms.

Proposal

The solution is a new entity tentatively called superpackage. A superpackage can be thought of as a named collection of types from one or more packages. Public types can be declared as exported to make them accessible outside the superpackage. Public types in a superpackage that are not exported are accessible only to other types in the superpackage.

Unlike packages, superpackages are explicitly declared in a Java source file. They are compiled by javac into a .module file (cf. class file). At runtime, the module file is loaded by the VM, which uses it to enforce access control. It is also reified via a reflective API similar to java.lang.Class.

As a rough analogy, you could think of a superpackage as a grouping of related packages similar to how a package is a grouping of related classes. Where today classes have either public or default (package private) access, for code that uses superpackages there will be exported public, non-exported public ("superpackage private"), and default (package private) access levels.

For practical purposes, attempting to access a non-exported class from outside its defining superpackage will result in the same behavior that is observed today when attempting to access a package private class from outside its defining package.

Deployment modules

JSR 294 deals with modularity at the Java language level. Deploying a module also requires consideration of aspects such as class loading, versioning, and packaging formats. Those issues are the topic of the deployment modules defined by JSR 277. That project uses the foundations provided by superpackages to build a system with the desired runtime semantics.

The reason that the development and deployment aspects are handled by two separate JSRs is rather simple: they are dealing with two different sets of problems. For example, the Java source language should not define the JAR file format and a set of libraries should not define VM level access control rules.

There are interactions between development and deployment, but to a large extent we are talking about independent issues. A divide-and-conquer approach that enables focused expert groups to concentrate on their individual problem space is preferable to a single large project. Of course, we want to coordinate between the development and deployment solutions, which is why both expert groups have several members in common.

Q&A

Do JSR 294 superpackages require deployment via JSR 277 modules?

No. Superpackages are independent of the deployment mechanism. They can be deployed in regular JAR files loaded by an existing ClassLoader such as a URLClassLoader, as a JSR 277 module, or using other mechanisms. In all cases the superpackage access control mechanisms are enforced by the Java VM. But the advanced deployment features of JSR 277 are only available if you package and deploy the superpackage as a JSR 277 module.

Why enforce access control at the VM level rather than hide implementation classes via some filtering mechanism in the ClassLoader?

The existing Java access control mechanisms operate at the VM level. That architecture has proved to be reliable and secure. Pushing access control decisions to a ClassLoader is inconsistent with that model. It is also susceptible to incorrect ClassLoader implementations and programming errors leading to accidental disclosure of ClassLoader objects. A deployment module system might still want to hide classes at the ClassLoader level, but the fundamental access control decisions must be made by the VM.

Why does the .module file list all the member and exported types?

An authoritative membership and export list is necessary for access control enforcement. It also eliminates the reliance on packaging provisions such as package sealing that are currently required for correctness in some configurations.

Can a class be a member of more than one superpackage?

No. Just like a class can only be a member of one package, it can also only be a member of one superpackage. At deployment time, an exported class can certainly be accessed from multiple contexts by setting up the ClassLoaders appropriately, but that is unrelated to development time concepts.

What about separate compilation, which is another area mentioned in the JSR 294 submission?

That is a separate topic that we should discuss another time. Stay tuned.

PS: don't let recent history fool you into believing that I will be posting an entry every day ;-) Most times it will be more like 2 or 3 entries a month.

Posted at 15:27 Nov 08, 2006 by Andreas Sterbenz in Modularity  |  Comments[4]

First public specification of the Java Module System (JSR 277)

The early draft specification of the Java Module System (JSR 277) was posted this week. This is what I have been spending most of my time on recently, working with Stanley Ho (spec lead) on specification issues for the expert group and taking the lead on the core aspects of the alpha implementation.

The specification may seem dense, but the basic concepts are fairly straightforward. Start by reading chapter 1 and section 2.1. They also explain the main rationale for the Java Module System: the standard runtime deployment concepts of Java SE - classpath and jre/lib/ext - are insufficient for large systems that run code from many different parties within the same VM process (for example, Application Servers). People have managed to work around these limitations using the extensible features of the Java platform, in particular ClassLoaders. But that is error prone and the platform should really have a built-in standard solution.

The result are Modules, which have stable names, define their supported APIs, and their dependencies on other modules. A Module is packaged into a Java Module archive (JAM file), which is deployed into a Repository. There will be private, per-user, and per-system Repositories. At runtime, we create Module instances, automatically resolve their dependencies, and setup the appropriate ClassLoaders. This allows different Modules to coexist in the same process without interfering with each other.

The Java Module System takes advantage of JSR 294 which is called Improved Modularity Support in the Java Programming Language. It will result in language and VM specification changes that allow developers to define the module information in a Java source file, which is compiled and verified by javac. This construct will also be understood by the VM, which uses it to enforce access control.

An important aspect of the Java Module system is its pluggability. The Repository class is extensible allowing 3rd parties to deliver their own implementations. Similarly, we want to make Module itself pluggable to allow developers to use module systems not based on JSR 294 within the same framework, as long as that module system follows compatible concepts.

You should understand that the current spefication of JSR 277 is literally an Early Draft. Many details are missing and the API is tentative. The reason it was published is to gather feedback from the community. Send it to the expert group at jsr-277-comments@jcp.org.

Posted at 15:17 Oct 15, 2006 by Andreas Sterbenz in Modularity  |  Comments[7]