Monday May 14, 2007
Monday May 14, 2007
For those of you who came to Frank and myself's BOF session at JavaOne, you can download my "modified" jhat here jhat.jar. My modification of original jhat include:
- add exclude functionality on query results to make it easier to filter out hundreds of duplicate reference chains, this way it will be possible to find unique leaks (see Figure 1)
- add classloader browser to nicely display classloaders' parent-child relationship, as well as number of classes loaded in each classloader (see Figure 2)
To run jhat, simple do "java -Xmx512m -jar jhat.jar <heap_file_name>"
Figure 1.
Figure 2.
Update: Frank posted the presentation slides and audio recording of this BOF on his blog entry.
Posted by Gary Zhu on May 15, 2007 at 08:50 AM PDT #
This is a great addition to jhat - I haven't found any other tools that provide this information nearly so easily.
I do have a question, though - is it possible there's a bug? According to this modification, I've managed to eliminate all references to a WebAppClassLoader that wasn't getting collected on application reload in Tomcat. Unfortunately, it still doesn't get collected. :(
Anyone else run into this?
Thanks!
Posted by Cott Lang on September 24, 2007 at 08:33 AM PDT #
Hi Cott,
I am not aware of any bugs in jhat, the changes I made are on top of JDK 1.6.0_01 source code.
One possibility I can think of is that the WebAppClassLoader is actually "garbage-collectable", but it is not yet reclaimed by JVM. Garbage-collection in JVM is unpredictable and varies depending on the algorithm. One way to make sure that it is leaking in the WebAppClassLoader is to have a automated script that does the deploy/undeploy cycle of your WebApp many many times, and then use JConsole to monitor the PermGen memory space of JVM. If PermGen space usage graph increases continueously over time, then you know you are still having the Classloader leak problem.
Posted by Edward Chou on September 24, 2007 at 08:45 PM PDT #
Edward,
You are, of course, correct. I re-ran my tests and deployed and un-deployed my application a few dozen times before it finally began to garbage collect the WebAppClassLoader. It seems it does not GC until it is stressed for perm gen space.
After that, it allowed 5-15 copies of it to linger steadily through hundreds of reloads.
Thanks again for this wonderful tool - I'd sure love to see this in the JDK distribution.
Posted by Cott Lang on September 28, 2007 at 06:12 PM PDT #
Thanks! I was trying to execute a query to dicover references between objects of different classloaders, but it is taking forever (+1 day and running).
This was the query:
select o1 from instanceof java.lang.Object o1
where classof(o1).loader != heap.findObject('0x046cbf28')
&& contains(referees(o1), "classof(it).loader == heap.findObject('0x046cbf28');")
where 0x046cbf28 is a WebApp classloader of Tomcat.
Your jhat version nicely helped to find out some suspects (sorry if I'm wrong on those):
Thread.contextClassLoader:
Maybe Tomcat does not reset inmediately this member after request processing.
org.apache.tomcat.util.digester.Digester.parser:
It is holding a reference to a Xerces parser in my application lib.
Posted by Imp on January 13, 2008 at 06:38 PM PST #
Imp,
You're not wrong on Xerces - you don't want that in your application lib under Tomcat.
Posted by Cott Lang on January 13, 2008 at 08:54 PM PST #
Thanks Cott.
I think this is the wrong place to post this... :(
Xerces has been deleted from WEB-INF/lib dir. It was there to support java 1.4.2.
The application keeps throwing permgen exceptions after 20 redeploys or so.
A heap dump after the error shows several unreferenced WebappClassLoader (7). Jhat shows no livepaths to those classloaders.
select cl
from instanceof org.apache.catalina.loader.WebappClassLoader cl
where count(heap.livepaths(cl)) == 0
(This query shows 7 classloaders)
The special query of Edward Chou to find Reference Chains from Rootset (Exclude weak refs, filter enabled) also shows the CLs as unreferenced!!
I'm using JDK 6 under Windows 2000.
May somebody shed some light over the issue? Is a reference hidding somewhere? What other reason may prevent the JVM from unloading one CL?
Boy, I needed to tell someone...
Posted by Imp on January 14, 2008 at 09:39 AM PST #