In my last post I talked about fragmentation in the Java ME space, and briefly mentioned one of the sources of this fragmentation: variability in specifications. Today I'll dig deeper into this fundamental issue.

In the Biblical story of the Tower of Babel, the people of earth, united and speaking a common language, built a tower that reached to the heavens. The Lord, believing that humanity was overreaching, said, "Come, let us go down and confuse their language so they will not understand each other." The people became dispersed about the earth, divided into different races and speaking different languages.

In human affairs, when groups of people who do not have a common language are brought into contact with each other (through trade, for example), a pidgin language typically emerges. Pidgins are simple "lowest common denominator" languages that borrow vocabulary and grammar from the native languages of the intermingled groups. Over time pidgins can evolve into creoles such as Bislama, the official language of Vanuatu in Melanesia whose 200,000 inhabitants speak more than 100 different languages. (The Bislama translation of the Babel story quote above is "Ale! Mi mi kotaon mi preprekem lanwis blong ol, bambae ol i nomo harem save tok blong ol.")

Similarly when two implementations of a software technology differ in the features they support, then we have an interoperability problem. For example, if one implementation supports features a b c d and e, while another supports features b c e f and g, application developers targeting both systems are likely to utilize only the features they have in common (b, c, and e), ignoring the others. Such pidgin programs, like pidgin languages, do not take advantage of the full richness of the available feature set.

Variability in Specifications

There are many ways in which specifications can permit – and sometimes even encourage – implementations to differ. The most obvious is optionality. For a variety of reasons (market pressures, difficulty of implementation, and so on), many specifications mark some features as optional rather than required. The more features that are optional, the less likely it is that two implementations of the specification will support the same feature set. (Note that just because something is optional doesn't mean that its functionality is not specified, nor that it is impossible to test – rather, the test suite must be smart enough to enable the tester to execute only the tests that apply to the implemented features.)

Optionality is only one of several dimensions of variability in specifications, all of which can – if used to excess – reduce interoperability between implementations. Others include the practice of subsetting via Profiles, Levels, or Modules, and the provision of extensibility mechanisms. For a detailed discussion of this subject, see Variability in Specifications published by the W3C QA Working Group.

As the previous posting suggested, one of the major differences between the Java SE and Java ME platforms from a conformance and compatibility perspective lies in the variability of their specifications. The Java SE platform specification is extremely "homogenous." APIs are tightly specified and very little is optional. The Java ME specifications, on the other hand, introduce a great deal of variability. Firstly, there are two "foundations" on which platforms can be built: the Connected Limited Device Configuration (CLDC), which is a small-footprint specification for low-end devices, and the Connected Device Configuration (CDC) specification, which targets more capable devices with a set of APIs that are a subset of those provided in Java SE. Since neither CLDC nor CDC is complete and rich enough to provide the full range of features required for mobile application programming, these are supplemented by a variety of profiles and "optional packages" that address different needs (networking, security, graphics, messaging, multimedia, web services, and so on.) In all, there are approximately 40 different Java ME specifications that can be mixed and matched in various ways, with another 20 or so still under development.

The resulting fragmentation of the market has made it difficult for application developers to address a broad range of devices without device-specific coding and extensive testing. "Umbrella specifications" such as the Java Technology for the Wireless Industry specification and the more recent Mobile Service Architecture specification attempt to address this problem.

Mi no Save

Finally, lest my new ME team members feel that I'm unfairly criticizing their technology, here's an example of a different kind of variability from the Java SE world. The Java SE security framework is comprehensive, flexible, and elegantly designed. It provides, among other features, support for "a wide range of cryptographic services," and it uses a pluggable architecture to enable implementations of different encryption algorithms (RSA, DSA, AES, Triple DES, and so on.) Unfortunately, the specification does not mandate that any particular algorithm be supported, so it would be quite legal for two implementations of Java SE to support different encryption algorithms and therefore to have no algorithm in common. (Of course it would also be legal for an implementation to support no algorithms at all.)

When a secure HTTP (HTTPS) connection is established between a client and a server the systems negotiate the encryption to be used. Imagine the following conversation (carried out in pidgin):

Client: "I speak French and German: how about you?"
Server: "Sorry, I don't speak either of those languages, but I do speak English and Chinese."

Result: the systems can communicate only in pidgin (HTTP) rather than in the more powerful HTTPS.

Actually, it's worse than this, since we also don't specify the names that should be used for the encryption algorithms that are supported. So, the following exchange might also occur:

Client: "I speak French and German: how about you?"
Server: "Sorry, I don't speak either of those languages, but I do speak Français and Chinese."

Oops! Sometimes elegance and flexibility come at a price.

Footnote

"Mi no save" (pronounced "me no savvy") means "I don't understand" in Bislama. Lukim yu bakagen...

I'm back. I have no excuse for such a long delay in posting other than the usual "pressure of work." As it happens, this pressure has increased recently since I'm now managing the Java ME conformance test team as well as the Java SE team. I'll have much more to say about this as I become familiar with my new responsibilities, but in the meantime here are some initial thoughts on the difference between the Java SE and Java ME worlds.

Although Java SE is deployed on more than 700 million systems (mostly PCs and laptops but also servers, since Java SE is the foundation for Java EE) the number of different implementations is limited and a significant percentage of the installed base consists of Sun's implementations (for Windows, Linux, and Solaris). These factors, coupled with the relatively homogenous nature of the specifications (more on this later) and of course the comprehensive nature of the conformance test suite (the JCK) that my team develops, ensure a high level of compatibility and interoperability between implementations.

Java ME is much more diverse. The primary market is cellphones (there are 1.2 billion Java-powered phones), but the technology is also deployed in other small and embedded devices such as TVs and set-top boxes (4 million deployments), PDAs, printers, and the new Blu-ray disk players. The Java ME portion of my team is also responsible for the conformance test suite for the Java Card smart card technology (2+ billion deployments and counting), but that's a subject for another article.

The mobile phone market presents an interesting contrast to Java SE. The market is much broader (more than 180 network operators and carriers) and much less homogenous. For a variety of reasons, the level of compatibility and interoperability between implementations of Java ME in mobile phones is lower than for Java SE. This problem is usually referred to as fragmentation.

As this article explains, there are three different types of fragmentation:

1. Platform fragmentation, caused by variability in the specifications (different versions of specs, implementation-specific extensions, optionality, etc.)
2. Implementation fragmentation (where implementations of the same APIs differ due to ambiguities in the spec, incomplete test suites, or just plain bugs).
3. Device-level fragmentation (physical differences between handsets, such as screen size and color depth, input mechanisms, or the amount of memory present).

For all these reasons, it is difficult to "write once run everywhere" in the Java mobile phone market.

This issue is being addressed by a variety of initiatives from "umbrella specifications" that define common set of component APIs that should be implemented across all devices (see the Java Technology for the Wireless Industry specification and the more recent Mobile Service Architecture specification) to the development of standardized testing tools such as the Java Device Test Suite and the promotion of broader industry-wide testing programs such as the Unified Testing Initiative.

There are enough challenges here to keep us very busy for the foreseeable future, not to mention the changes that will result from the decision to open-source all our Java technologies. Change is good...