Moazam Rajas Sun hosted weblog garbage collection II

I'm looking forward to this part..it'll actually be interesting to hear what questions are asked of the experts. Original schedule here.

MONDAY

---------------
10:30am - 1pm "Java HotSpot VM, Performance & Scalability"

    Charlie Hunt, Senior Staff Engineer, JWS Performance
    Tim Cramer, Engineering Manager, JWS Performance
    Paul Hohensee, Senior Staff Engineer, HotSpot VM
    Peter Kessler, Senior Staff Engineer, HotSpot VM

12:30pm - 3pm "Language & Core Libraries"

    Neal Gafter, Senior Staff Engineer, javac and core tools
    Joshua Bloch, Distinguished Engineer, Architect, core Java libraries

2:30pm - 5pm "Debugging & Profiling"

    Kelly O'Hair, Senior Staff Engineer, Profiling
    Jim Holmlund, JPDA (Java Platform Debugger Architecture) Technical Lead
    Robert Field, JVMTI (Java VM Tools Interface) Architect
    Alan Bateman, Staff Engineer, Profiling & Debugging
    Tim Bell, Engineer, Debugging

4:30pm - 7pm "JMX and JVM Monitoring & Management"

    Simon Vienot, JMX Engineering Manager
    Eamonn McManus, JMX Spec Lead
    Mandy Chung, VM Monitoring & Management Technical Lead
    Sanjay Radia, J2SE RAS Architect

TUESDAY

----------------
10:30am - 1pm "Networking & Security"

    Jeff Nisewanger, Security Architect
    Jessie Collet, Networking Architect

12:30pm - 3pm "Debugging & Profiling"

    Kelly O'Hair, Senior Staff Engineer, Profiling
    Jim Holmlund, JPDA (Java Platform Debugger Architecture) Technical Lead
    Robert Field, JVMTI (Java VM Tools Interface) Architect
    Alan Bateman, Staff Engineer, Profiling & Debugging
    Tim Bell, Engineer, Debugging

2:30pm - 5pm "Java HotSpot VM, Performance & Scalability"

    Tom Marble, JWS Performance Engineer
    Brian Doherty, JWS Performance Engineer
    Paul Hohensee, Senior Staff Engineer, HotSpot VM
    Steve Dever, Staff Engineer, HotSpot VM
    Ross Knippel, Senior Staff Engineer, HotSpot VM

4:30pm - 7pm "Language & Core Libraries"

    Scott Seligman, Senior Staff Engineer, apt and javadoc tools
    Joe Darcy, Engineer, apt, reflection, numerics

WEDNESDAY

---------------------
10:30am - 1pm "Language & Core Libraries"

    Neal Gafter, Senior Staff Engineer, javac and core tools
    Joshua Bloch, Distinguished Engineer, Architect, core Java libraries
    Joe Darcy, Engineer, apt, reflection, numerics

12:30pm - 3pm "Java HotSpot VM, Performance & Scalability"

    David Dagastine, Staff Engineer, JWS Performance
    Scott Oaks, Senior Staff Engineer, JWS Performance
    Jon Masamitsu, Staff Engineer, HotSpot VM
    Jon Coomes, Staff Engineer, HotSpot VM

2:30pm - 5pm "JMX and JVM Monitoring & Management"

    Simon Vienot, JMX Engineering Manager
    Eamonn McManus, JMX Spec Lead
    Mandy Chung, VM Monitoring & Management Technical Lead

4:30pm - 7pm "Tiger's Children"

    Graham Hamilton, VP & Fellow
    Mark Reinhold, J2SE Chief Engineer
    Thorston Laux, Desktop Java Strategist
    Laurie Tolson, Senior Director J2SE

Looks like I'll be at JavaOne next week, in the java.net booth. I'll post my schedule tomorrow. Oh, and I'll be at the "JavaOne Blogger Meetup" also..

Rajiv Shivane has an excellent document on his site titled "Of Thread dumps and stack traces ...". After reading his document, I ventured into some of his other writings and found the following quote here:

"The only flag I could relate to here was the -XX:+PrintCompilation. It is another thing that I have no clue how to interpret the output of this flag either! I vaguely remember reading about this in some javaOne presentation. The presenter had mentioned that it prints the methods which could not be compiled and the user can list the methods which he doesn't want to be compiled in a .hotspot_compiler file. I really wish Sun had documented these flags and their output better, instead of just mentioning "traces methods as compiled" in the VM Options document." [Rajiv Shivane - Of perf degradation with try-finallies and poor VM option docs]

Rajiv, [insert presidential voice] I feel your pain. I really do. I'm working on a document right now which will list out many JVM options and which versions (1.2.2, 1.3.1, etc.) these options are applicable to or not.

But nevertheless, lets get back to the meat of the matter, the question on -XX:+PrintCompilation. I thought that a more detailed explanation of this flags output existed somewhere on the public internet, but maybe I just can't find it today. So I'll make a small attempt to explain the output.

[read full doc here]

Rob Reyer from www.evilrob.org commented on my weblog entry about thread dump formatting and analysis. Rob mentions that he spent 6 days trying to get a thread dump right as an application was core dumping. He is not the first person I've heard this from...hell, I have the same problem.

From my outside weblog:

I can't believe that I didn't know about this before! ODB allows you to do 'Omniscient Debugging' on Java applications, with or without the original source code.

The ODB debugger allows "..developers to step backwards through the execution of a program to determine where and how programming errors occurred. By recording each state change in the target application, it allows the developer to navigate "backwards in time" to see what the values of variables and objects WERE, enormously simplifying the task of debugging programs."

One page description of the ODB
Article , Debugging Backwards in Time
AADEBUG paper on Omniscient Debugging (pdf)
ODB User Manual

I'm sure people have noticed this little idiosyncrasy in the way that garbage collection log output is handled in the JVM. But if you have not, this is a good data point when preparing to read through GC logs.

If you simply use the verbose:gc flag, you'll have GC log output sent to the stdout console. Now, if you use the -Xloggc:[filename] switch, the GC data will be sent to a log file which you can grep through later. But either way, you get the the same GC data ...right? Wrong.

The -Xloggc:[filename] switch has the additional effect of turning on the -XX:+PrintGCTimeStamps switch and hence gives your log files the added benefit of time stamps. Wierd! I'd expect both of these switches to have the same exact behavior. I might file an RFE or Bug on this tonight.

Oh, for those who want to use it, the -Xloggc switch was introduced in 1.4.0 and newer VMs.

Alexandre Rafalovitch, a BEA Backline Support Engineer emailed me about reading thread dumps and his proposal for formatting dump data. I've read through Alexandres article, and while I'm not a big fan of using XML for everything, he definitely presents an excellent view on making thread dumps easier to read and diagnose. The really cool part is that Alexandre has written an application which takes standard thread dump data, converts it into XML, then uses XSLT to convert the data into the format of your choice.

I'm actually a bit surprised that there isn't a JSR in progress to address this issue. I think one of the reasons for this is that most developers and users are not interested in the nuances of thread dump readability. Of course, as backline engineers, this topic is near and dear to myself and Alexandre. Alexandre will be presenting this in a session at JavaONE 2004, session TS-1646. I'll be sure to attend.

"What would be a perfect solution here is to be able to convert these thread-dumps into some sort of intermediate structure/language that has flexible structure, can tolerate missing fields, and can be processed by third-party tools in various ways. And, of course, if it can somehow involve a currently popular technology, so much the better.

Do we have such an intermediate language? Indeed we do — it is XML. It is somewhat human readable, has many tools written for it, and already has not one but several transformation and reporting technologies built on top of it, such as XSLT, XQuery, and XDB." [full article]

Want to look at the code of the JVM or build a test version of your own? Use the source.

And look, detailed build instructions are available too!

This has to be one of the most useful outlines of GC tuning information I have ever seen. Thanks to Pete Freitag for creating this from the original document. The original was nowhere as easy to read.
1.4.2 GC Tuning Outline by Pete Freitag

Many people have asked if the MaxPermSize value is a part of the overall -Xmx heap setting or additional to it. There is a GC document on the Sun website which is causing some confusion due to a somewhat vague explanation and an errant diagram. The more I look at this document, the more I think the original author has made a subtle mistake in describing -Xmx as it relates to the PermSize and MaxPermSize.
Read more..

Once in a while Java users and developers run into problems where their Java application simply seems to hang. No core file is generated, no IO is detected, the process just sits there waiting...for something. Usually these problems can be traced to OS and JVM level threading.
Debugging thread related hangs in the JVM

There are times when reading through megabytes of GC logs is overkill, especially when all you need is a general overview on how the JVMs GC algorithms are behaving with your application. The 'jvmstat' package helps developers and users visualize GC instead.
Visualizing Garbage Collection with jvmstat