Alan Bateman

Time to update the J2SE Troubleshooting Guide!

One of the documents created for J2SE^TM 5.0 was the Troubleshooting and Diagnostic Guide. It was a late decision to create this document with the result that its scope was much narrower than might be expected. However it served a useful purpose in that it created an awareness of the troubleshooting tools that were new to J2SE 5.0. It also served as a useful place to send people when troubleshooting questions came up (like how to find the error log that is generated when a fatal error occurs).

It's now time to start thinking about updating the document for Mustang. The initial document focused on the HotSpot^TM virtual machine. The document for Mustang needs to be wider in scope and cover troubleshooting problems with client-side applications, deployment, and many other areas. Mustang brings a lot of new and updated diagnostic features. I touched on the heap dump capabilities here, and Sundar has started a series on the Object Query Language feature in jhat. There are many other useful items and these need to be written up.

If you have ideas for areas that should be covered in the updated Troubleshooting Guide please send a mail to j2se-trouble-shooting-guide@sun.com. Also, send feedback if you think that one big document isn't the right format for this - the original reason for putting everything into one document was to allow people to print it out. I don't know many trees suffered needlessly so send feedback if you think an alternative format would be better. Finally, if you would like to contribute content then please step forward. In particular if you recently diagnosed a really awkward problem with a deployed application then others might benefit from a write-up of the steps that you took to diagnose it.

Posted at 04:03AM Sep 30, 2005 by alanb in Archive  |  Comments[1]

Getting started with jconsole just got easier!

J2SE^TM 5.0 brought some great monitoring and management capabilites to the Java^TM platform. The built-in instrumentation in the Java Virtual Machine means you can monitor and manage it using JMX. And of course, if the application have been instrumented with JMX then it gets even better. As part of the monitoring and management implementation a JMX-compliant monitoring tool called jconsole was developed. I hope you have tried it. If you haven't tried it let then a good starting place is the this page on Monitoring and Management Using JMX

One of the goals during the development of jconsole was that it should be easy to get going very quickly. The JMX agent has lots of configuration properties but a new user should be able to avoid most of this and get going in a few simple steps. All you need is to start the application with a simple property on the command line. When this property is set then the JMX agent starts up in a way that allows jconsole to connect without needing to prompt for connection details. This is somethings called local monitoring because it restricts the monitoring to the local machine (you have to be the same user too). The property is com.sun.management.jmxremote so you start your application like this:

java -Dcom.sun.management.jmxremote App

Once the application is running then jconsole can connect. If you know the process-id (pid) then "jconsole <process-id>" should do it. Alternatively, if you don't know the process id then start jconsole without any parameters and select the application in the connection dialog.

It turns out that it gets even simpler in Mustang (Java SE 6.0). The reason is that jconsole has been updated so that it can connect to applications that did not start up with the JMX agent. This is rather useful as it allows jconsole to connect to applets in the browser, Java Web Start applications, and other applications where it isn't easy to fiddle with the command line.

So how does it work? jconsole uses a JMXConnector client to connect to the JMXConnectorServer in the target application. In the application isn't started with the JMX agent then there isn't a JMXConnectorServer and jconsole doesn't have anything to connect too. In that case it uses a HotSpot^TM VM specific mechanism to start the JMX agent in the target VM. Once the agent is started then jconsole connects as normal.

This sounds very cool so let us see how it looks. First, we start jconsole. I haven't use any parameters so jconsole opens the Connection Dialog. I've selected the Local Tab and jconsole shows me the applications that I have running on this machine. In this screen-shot I have clicked ona Java Web Start application called bugster:

Next I press the Connect button. Behind the scenes the JMX agent is started in the target VM and jconsole connects. (By the way, that checkbox column with the title Enabled just indictaes if the JMX agent is running in the target VM or not - this is probably not the best way to indicate this but jconsole is due a new connection dialog soon and it should look much better). Once the connection is established jconsole opens up Summary Tab which gives me a summary of the bugster application.

I'm sure you'll agree this is rather neat and makes getting started with jconsole very easy.

Posted at 09:38AM Sep 23, 2005 by alanb in Archive  |  Comments[11]

Heap dumps are back with a vengeance!

The HPROF agent has been around since 1998 and early betas of J2SE^TM 1.2. One of its more useful features, at the time, was the ability to generate a heap dump. The heap dump is a dump of all the live objects and classes. The HPROF agent also records where objects are allocated and these are written to the dump file too. Heap dumps aren't very useful without a tool to read them but a novel tool called the Heap Analysis Tool (HAT), courtesy of Bill Foote, was released at around the same time. HAT was very useful as it allowed developers to browse the heap dump, and do rudimentary queries to debug memory leak problems.

HPROF fell on hard times in J2SE 1.3 and 1.4. One consequence of this was that the heap dumps weren't always readable by HAT. The background to this is that HPROF used an experimental profiling interface called JVMPI. JVMPI was designed for the original classic VM where it worked well. An implementation of JVMPI was created for its replacement, the HotSpot^TM VM, but it was problematic. The root of the issue is that JVMPI wasn't really designed with modern virtual machines in mind. It required events from places that are highly optimized in modern virtual machines. HPROF required the OBJECT_ALLOC event when started with the heap=dump option. This was one of the more troublesome events as it inhibited many optimizations, and didn't work with all garbage collection implementations.

HPROF returned to its glory days in J2SE 5.0 thanks to a complete make over by Kelly O'Hair The catalyst for the make over (actually a complete re-write) was JSR-163 which defined a new tool interface called the JVM^TM Tool Interface. JVM TI broke from the past and didn't provide some events that one might expect in a profiling interface. In particular it doesn't provide an object allocation event - in its place the interface provides support for tools to do bytecode instrumentation. With HPROF re-implemented it meant that heap dumps were working again. HAT was back in business!

In Mustang (Java SE 6.0), HPROF gets new two new siblings which bring new ways to generate heap dumps.

• The first sibling is the built-in heap dumper. In production environments you probably don't want to run your application with an agent like HPROF as it adds significant overhead to the running application. But, if your application fails with java.lang.OutOfMemoryError then it might be useful to have a heap dump created automatically. This is where the built-in heap dumper comes in. Once you have a heap dump you can browse offline and hopefully figure out what is consuming all the memory.

  The built-in heap dumper can also be used to take a snapshot of the heap at other times. This is done using the using the jmap command line utility or jconsole monitoring and management console. This is also very useful - if you are monitoring an application with jconsole and you see the heap usage growing over time then you might want to take a snapshot of the heap to see what is going on.

• The second new sibling in the family specializes in pathology. If you are unlucky to get a crash then it might be useful to look at what is was in the heap at the time of the crash. This is where the second heap dumper comes in as it can generate a heap dump from a core file. It can also be used to obtain a heap dump from an application that is completely hung. A word of warning here - with a crash it is possible that there is some heap corruption so no guarantee that a heap dump can be obtained.

Before we meet the new heap dumpers it is important to mention that they generate simple heap dumps. That is, the dump files contain information about all the objects and classes in the heap but they do not contain information about where the objects are allocated. If you need this information then it is best to run with a JVM TI agent that records this. The NetBeans Profiler is particularly good at this.

Now let us meet the new heap dumpers ...

First, here is an example where we run an application, called ConsumeHeap, with a flag that tells the HotSpot VM to generate a generate a heap dump when we run out of memory:

$ java -XX:+HeapDumpOnOutOfMemoryError -mn256m -mx512m ConsumeHeap
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid2262.hprof ...
Heap dump file created [531535128 bytes in 14.691 secs]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at ConsumeHeap$BigObject.(ConsumeHeap.java:22)
        at ConsumeHeap.main(ConsumeHeap.java:32)
$

ConsumeHeap, as expected, fills up the java heap and runs out of memory. When java.lang.OutOfMemoryError is thrown a heap dump file is created. In this case the file is 507MB and is created as java_pid2262.hprof in the current directory. If you don't want big dump files in the application working directory then the HeapDumpPath option can be used to specify an alternative location - for example -XX:HeapDumpPath=/disk2/dumps will cause the heap dump to be generated in the /disk2/dumps directory.

[As a complete aside, notice that the java.lang.OutOfMemoryError has a stack trace - this is also new in Mustang - in J2SE 5.0 the OutOfMemoryError would have been thrown without any stack trace]

Now let's look at a second example:

C:\> jmap -dump:file=app.bin 345
Dumping heap to C:\temp\app.bin ...
Heap dump file created
C:\>

This example uses the jmap utility to generate a heap dump of the java application running as process 345. Astute readers will observe that this example was done on Microsoft Windows but the jmap utility was only included with the J2SE 5.0 releases for Solaris and Linux. This is semi-true for Mustang too but jmap.exe is included and supports a subset of the options available on the other platforms. In particular the -dump option is there on all platforms.

Now let us look at an example that generates a heap dump from a core file. As crashes are rare I've cheated a bit by getting a core file with the Solaris gcore command:

$ gcore 5831
gcore: core.5831 dumped
$ jmap -dump:file=app.bin `which java` core.5831            
Attaching to core core.5831 from executable /opt/java/bin/java, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 1.6.0-ea-b52
Dumping heap to app.bin ...
Unknown oop at 0xf14b8650
Oop's klass is 0xf14b54d0
Unknown oop at 0xf14e67a8
Oop's klass is null
Heap dump file created
$

The arguments after the -dump option are the executable and the core image file. I used `which java` which gives me the pathname of java. Needless to say these need to match. Less obvious is that you that you can only use jmap from the same JDK build as the executable too. So for example, if you have a core file from 6.0-ea-b52 then you need to use jmap from 6.0-ea-b52 to generate a heap dump from the core file.

You might notice a few warnings in the output. The messages Unknown oop at ... might be a bit off-putting but remember that the core image is taken at an arbitrary time. There is no guarantee that the heap and other data structures are in a consistent state when the crash dump is obtained.

At this point your disks are probably full of heap dumps and you are wondering how to examine them. In the introduction I mentioned the Heap Analysis Tool (HAT). HAT is now included in Mustang (since b51) as a command line utility called "jhat".

Getting started with jhat is easy - just give it the name of the heap dump file:

$ jhat java_pid2278.hprof
Started HTTP server on port 7000
Reading from java_pid2278.hprof...
Dump file created Sun Sep 18 17:18:38 BST 2005
Snapshot read, resolving...
Resolving 6162194 objects...
Chasing references, expect 12324 dots.........................................................
Eliminating duplicate references..............................................................
Snapshot resolved.
Server is ready.

At this point HAT has started a HTTP server on port 7000. Point your browser to http://localhost:7000 to connect to the HAT server.

[HAT requires a lot of memory so if you try it and it fails with OutOfMemoryError then you might need to give it a larger heap size (for example: jhat -J-mx512m java_pid2278.hprof). The memory consumption of HAT is currently being worked on - expect to see improvements in b54 or b55.]

When you connect to the HAT server you should see a list of all the classes in the heap dump. This is the All Classes query. Scroll to the bottom of the page to see the other queries. A full description of all the queries is more than I have time for today but you can read more in the HAT README.

Those that are familiar with HAT will not see many differences between HAT 1.1 and jhat in b52. However you can expect improvements in b53 (should be available on 9/23). For starters there are a number of robustness improvements - like being able to to parse incomplete and truncated heap dumps. There are also various fixes and finally HAT will be able to read heap dumps generated on 64-bit systems.

Small improvements aside, a useful (and very interesting) addition in b53 is that Sundar Athijegannathan has added a scripting interface to HAT. This make use of the new scripting API. This addition allows developers to enter arbitrary queries into browser and they aren't tied to canned queries that HAT provides. Keep an eye on Sundar's blog for updates on this topic.

To sum up: Mustang (Java SE 6.0) allows heap dumps to be obtained at out of memory time, at any time during the life time of the application, or even after an applications dies with a crash. Also with jhat included in the JDK it means that there is a simple out of the box utility to examine the heap dumps and do rudimentary memory analysis. So heap dumps are back, and back with a vengeance!

Posted at 08:31AM Sep 19, 2005 by alanb in Archive  |  Comments[40]