Weblog

Monday September 19, 2005

Heap dumps are back with a vengeance! The HPROF agent has been around since 1998 and early betas of J2SE^TM 1.2. One of its more useful features, at the time, was the ability to generate a heap dump. The heap dump is a dump of all the live objects and classes. The HPROF agent also records where objects are allocated and these are written to the dump file too. Heap dumps aren't very useful without a tool to read them but a novel tool called the Heap Analysis Tool (HAT), courtesy of Bill Foote, was released at around the same time. HAT was very useful as it allowed developers to browse the heap dump, and do rudimentary queries to debug memory leak problems.

HPROF fell on hard times in J2SE 1.3 and 1.4. One consequence of this was that the heap dumps weren't always readable by HAT. The background to this is that HPROF used an experimental profiling interface called JVMPI. JVMPI was designed for the original classic VM where it worked well. An implementation of JVMPI was created for its replacement, the HotSpot^TM VM, but it was problematic. The root of the issue is that JVMPI wasn't really designed with modern virtual machines in mind. It required events from places that are highly optimized in modern virtual machines. HPROF required the OBJECT_ALLOC event when started with the heap=dump option. This was one of the more troublesome events as it inhibited many optimizations, and didn't work with all garbage collection implementations.

HPROF returned to its glory days in J2SE 5.0 thanks to a complete make over by Kelly O'Hair The catalyst for the make over (actually a complete re-write) was JSR-163 which defined a new tool interface called the JVM^TM Tool Interface. JVM TI broke from the past and didn't provide some events that one might expect in a profiling interface. In particular it doesn't provide an object allocation event - in its place the interface provides support for tools to do bytecode instrumentation. With HPROF re-implemented it meant that heap dumps were working again. HAT was back in business!

In Mustang (Java SE 6.0), HPROF gets new two new siblings which bring new ways to generate heap dumps.

• The first sibling is the built-in heap dumper. In production environments you probably don't want to run your application with an agent like HPROF as it adds significant overhead to the running application. But, if your application fails with java.lang.OutOfMemoryError then it might be useful to have a heap dump created automatically. This is where the built-in heap dumper comes in. Once you have a heap dump you can browse offline and hopefully figure out what is consuming all the memory.

  The built-in heap dumper can also be used to take a snapshot of the heap at other times. This is done using the using the jmap command line utility or jconsole monitoring and management console. This is also very useful - if you are monitoring an application with jconsole and you see the heap usage growing over time then you might want to take a snapshot of the heap to see what is going on.

• The second new sibling in the family specializes in pathology. If you are unlucky to get a crash then it might be useful to look at what is was in the heap at the time of the crash. This is where the second heap dumper comes in as it can generate a heap dump from a core file. It can also be used to obtain a heap dump from an application that is completely hung. A word of warning here - with a crash it is possible that there is some heap corruption so no guarantee that a heap dump can be obtained.

Before we meet the new heap dumpers it is important to mention that they generate simple heap dumps. That is, the dump files contain information about all the objects and classes in the heap but they do not contain information about where the objects are allocated. If you need this information then it is best to run with a JVM TI agent that records this. The NetBeans Profiler is particularly good at this.

Now let us meet the new heap dumpers ...

First, here is an example where we run an application, called ConsumeHeap, with a flag that tells the HotSpot VM to generate a generate a heap dump when we run out of memory:

$ java -XX:+HeapDumpOnOutOfMemoryError -mn256m -mx512m ConsumeHeap
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid2262.hprof ...
Heap dump file created [531535128 bytes in 14.691 secs]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at ConsumeHeap$BigObject.(ConsumeHeap.java:22)
        at ConsumeHeap.main(ConsumeHeap.java:32)
$

ConsumeHeap, as expected, fills up the java heap and runs out of memory. When java.lang.OutOfMemoryError is thrown a heap dump file is created. In this case the file is 507MB and is created as java_pid2262.hprof in the current directory. If you don't want big dump files in the application working directory then the HeapDumpPath option can be used to specify an alternative location - for example -XX:HeapDumpPath=/disk2/dumps will cause the heap dump to be generated in the /disk2/dumps directory.

[As a complete aside, notice that the java.lang.OutOfMemoryError has a stack trace - this is also new in Mustang - in J2SE 5.0 the OutOfMemoryError would have been thrown without any stack trace]

Now let's look at a second example:

C:\> jmap -dump:file=app.bin 345
Dumping heap to C:\temp\app.bin ...
Heap dump file created
C:\>

This example uses the jmap utility to generate a heap dump of the java application running as process 345. Astute readers will observe that this example was done on Microsoft Windows but the jmap utility was only included with the J2SE 5.0 releases for Solaris and Linux. This is semi-true for Mustang too but jmap.exe is included and supports a subset of the options available on the other platforms. In particular the -dump option is there on all platforms.

Now let us look at an example that generates a heap dump from a core file. As crashes are rare I've cheated a bit by getting a core file with the Solaris gcore command:

$ gcore 5831
gcore: core.5831 dumped
$ jmap -dump:file=app.bin `which java` core.5831            
Attaching to core core.5831 from executable /opt/java/bin/java, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 1.6.0-ea-b52
Dumping heap to app.bin ...
Unknown oop at 0xf14b8650
Oop's klass is 0xf14b54d0
Unknown oop at 0xf14e67a8
Oop's klass is null
Heap dump file created
$

The arguments after the -dump option are the executable and the core image file. I used `which java` which gives me the pathname of java. Needless to say these need to match. Less obvious is that you that you can only use jmap from the same JDK build as the executable too. So for example, if you have a core file from 6.0-ea-b52 then you need to use jmap from 6.0-ea-b52 to generate a heap dump from the core file.

You might notice a few warnings in the output. The messages Unknown oop at ... might be a bit off-putting but remember that the core image is taken at an arbitrary time. There is no guarantee that the heap and other data structures are in a consistent state when the crash dump is obtained.

At this point your disks are probably full of heap dumps and you are wondering how to examine them. In the introduction I mentioned the Heap Analysis Tool (HAT). HAT is now included in Mustang (since b51) as a command line utility called "jhat".

Getting started with jhat is easy - just give it the name of the heap dump file:

$ jhat java_pid2278.hprof
Started HTTP server on port 7000
Reading from java_pid2278.hprof...
Dump file created Sun Sep 18 17:18:38 BST 2005
Snapshot read, resolving...
Resolving 6162194 objects...
Chasing references, expect 12324 dots.........................................................
Eliminating duplicate references..............................................................
Snapshot resolved.
Server is ready.

At this point HAT has started a HTTP server on port 7000. Point your browser to http://localhost:7000 to connect to the HAT server.

[HAT requires a lot of memory so if you try it and it fails with OutOfMemoryError then you might need to give it a larger heap size (for example: jhat -J-mx512m java_pid2278.hprof). The memory consumption of HAT is currently being worked on - expect to see improvements in b54 or b55.]

When you connect to the HAT server you should see a list of all the classes in the heap dump. This is the All Classes query. Scroll to the bottom of the page to see the other queries. A full description of all the queries is more than I have time for today but you can read more in the HAT README.

Those that are familiar with HAT will not see many differences between HAT 1.1 and jhat in b52. However you can expect improvements in b53 (should be available on 9/23). For starters there are a number of robustness improvements - like being able to to parse incomplete and truncated heap dumps. There are also various fixes and finally HAT will be able to read heap dumps generated on 64-bit systems.

Small improvements aside, a useful (and very interesting) addition in b53 is that Sundar Athijegannathan has added a scripting interface to HAT. This make use of the new scripting API. This addition allows developers to enter arbitrary queries into browser and they aren't tied to canned queries that HAT provides. Keep an eye on Sundar's blog for updates on this topic.

To sum up: Mustang (Java SE 6.0) allows heap dumps to be obtained at out of memory time, at any time during the life time of the application, or even after an applications dies with a crash. Also with jhat included in the JDK it means that there is a simple out of the box utility to examine the heap dumps and do rudimentary memory analysis. So heap dumps are back, and back with a vengeance! ( Sep 19 2005, 08:31:34 AM PDT ) Permalink Comments [37]