Java and too many jars

 

One of the projects I'm helping architect is a pretty huge undertaking. It's got lots of separate functional blocks being deployed into classloaders in each Java SE JVM, and since they're being developed by lots of different teams, we're seeing the same kind of common infrastructure jars (hibernate, apache commons logging, jwsdp, etc.etc.) being loaded multiple times into some of the VMs.

Now this can cause a few problems, and I've been helping clean them up:

Referring to the same class in the same VM coming from another classloader

Having the same classes defined twice, in different classloaders, means that the code in each classloader cannot share any instances of those classes, they're seen as different class types by the VM.

The easy way to fix this is to have a shared classloader that both other classloaders inherit from. Fortunately for us, the java management container I work on using makes this trivially easy, and there are tools to analyse the classloader hierarchy and detect duplicates.

The problem with such shared classloaders is that it becomes hard to update a copy of a jar without impacting everything that depends upon it - this is a classic problem with things like app-servers that just have a single "common pool" classloader used by all apps deployed into it - it's more elegant and powerful to have a directional graph of classloaders allowing dependencies to be stated explicitly. This is what we have.

Having the same class or jar defined twice on the same class path

Things like apache commons logging and jwsdp actually don't like being on the classpath more than once, and don't like being loaded twice on different classpaths into the same jvm. apache commons logging will actually search through the classpath to find itself, and complains if it finds itself twice.

Excessive resource usage and its limits

Having so many jars loaded into the VM eats up memory and file descriptors, never a good thing.

By opening so many jars in the Java 6 VM, we've actually managed to hit a hard limit on 32-bit processes on Solaris machines - something that's got a workaround on older machines, and something that is no longer a hard limit as of Solaris 10 update 4.

For backward compatibility reasons with very old systems, the stdio libraries on 32-bit processes on Solaris machines prior to Solaris 10 update 4 use a single byte to store file descriptors. So they must have file descriptors with numbers below 256. If something else eats up all the low-numbered file descriptors, then you can no longer get a new file descriptor with 'fopen', it'll return an error.

Now Java SE 6 introduced a new performance and memory footprint feature which, instead of mapping the jar files in a classpath into memory using mmap, opens and reads the jar files instead. And if you have lots (and lots) of jar files, you eat up more file descriptors than before, and if tyou also have sockets and other things open, you are pushing your application closer to that magic 256 number. And if something calls a stdio 'fopen' call after reaching that number, it'll fail.

And this was happening to our code - a little bit of JNI code was calling fopen and was failing.

Fortunately there are various ways of working around this limitation, the best way of course is to pick up Solaris 10 update 4 where the limitation has gone away. Other ways that solve the problem for any 32-bit application, not just Java, are discussed here. I've also been working with the Java SE team to have the Java 6 VM less susceptible to this problem, this fix, which is actually a work-around to the older solaris limitation, should be in the upcoming update to Java 6 and will be in Java 7.

A note of humour

All this jars and jar-loading configuration stuff prompted Lubos to forward a highly funny and very relevant video

All the RoR hype is interesting to watch. It's very exciting to see where this is going. Convention over configuration is something I believe in strongly, and which we do every day... RoR has pushed this further. I guess we've already got Java on Rails it's just that it missed the buzzword train! :-) I could blog on just this alone... ant vs. make, maven vs. ant, mbeans vs beans vs objects, annotations, ...

( Jun 22 2007, 03:37:26 PM CEST ) Permalink