Eric Lerognon's blog: Business Integration - In the field

Situations where a thread dump is helpful

There are many situations where it is very helpful to get a dump of all Java threads inside the application server. Sometimes the application server does not process messages as fast as it should be. This is caused by deadlocks, Java locks or threads waiting for resources (I/O or CPU). To identify the root cause of the issue the best way is to get a dozen of full thread dumps one after the other typically during 20 seconds. Then you can have a look at the dumps and look for patterns. From my experience it is quite straightforward to identify the threads corresponding to eGate JCDs because the JCD Java Class name is inside the stack. If you don't find anything obvious, you can try to parse the output using the Thread Dump Analyszer.

Methods to get the thread dump without JMX

With Java CAPS 5.1 running on Java SE 5 the jstack utility is only available on Solaris. On other Unix systems you should be able to send a QUIT signal to the process corresponding to the application server. But you get the resulting dump in the log file and this is somewhat difficult to extract.

On Windows, you can send a CTRL+BREAK signal inside the CMD window where the application server is running. But this requires 1/ that you launch the application manually in a CMD window 2/ that you are working on the server itself 3/ extract the dump from the log file. Practically you cannot hit CTRL+BREAK remotely and you cannot use this method when the application is running as a Windows service. One workaround is to use the SendSignal program. But this tool requires that you de-activate the -Xrs option which is enabled by default.

Method to get the thread dump with JMX

1/ Enable the JMX agent as described on my previous blog item

2/ Install the latest Java JDK 1.6

3/ Go to directory C:\Program Files\Java\jdk1.6.0_10\demo\management\FullThreadDump

4/ Type command java -jar FullThreadDump.jar localhost:9877

This is the beginning of the dump - it can contains hundreds of threads.

C:\Program Files\Java\jdk1.6.0_10\demo\management\FullThreadDump>java -jar FullThreadDump.jar localhost:9877
Connecting to localhost:9877
Full Java thread dump
"JMX server connection timeout 87" Id=87 in TIMED_WAITING on lock=[I@1536174
    at java.lang.Object.wait(Native Method)
    at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
    at java.lang.Thread.run(Thread.java:595)

"RMI TCP Connection(5)-192.168.0.3" Id=86 in RUNNABLE
    at sun.management.ThreadImpl.getThreadInfo0(Native Method)
    at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:142)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at sun.management.MXBeanSupport.invoke(MXBeanSupport.java:632)