Moazam Rajas Sun hosted weblog garbage collection II

« Previous page of month (Jun 2004) | Main | Next month (Jul 2004) »

Thursday Jun 10, 2004

Solaris 'which' command does not return proper exit code

Richard Lee over at VA Software brought this up to me today. He said that the Solaris 'which' command does not return the correct error code upon failure. Because of this he has to end up parsing stdout when he should be able to simply rely on the exit code.
Example:

# uname -a
SunOS sol10 5.10 s10_55 sun4u sparc SUNW,Sun-Blade-1500

# which ls
/usr/bin/ls
# echo $?
0

# which oogabooga
no oogabooga in /usr/sbin /usr/bin
# echo $?
0

As Richard would say, "Dirty".

Posted at 08:59PM Jun 10, 2004 by moazam in Solaris | Comments[3] | Permalink

Updating Fedora Core 1 to perfection

There are three things I do after installing Fedora Core 1 on any system.

1.) Get apt-get.

# rpm -Uvh http://atrpms.net/dist/fc1/atrpms-kickstart/atrpms-kickstart-22-1.rhfc1.at.i386.rpm

2.) Update the whole system via apt-get.

# apt-get update && apt-get dist-upgrade

3.) Install TrueType fonts from the instructions provided here.

*Steps 1 and 2 are gleaned from Jarod Wilsons MythTV Installation Guide

Posted at 03:10PM Jun 10, 2004 by moazam in Linux | Comments[3] | Permalink

Thread movement from Solaris 8, 9, and now 10

I glossed over the differences between the default Solaris 8 threads MxN behavior and the Solaris 9 1:1 behavior in my Debugging thread related hangs in the JVM writeup. Solaris 10 brings new changes in that threading has moved away from libthread and directly into libc. Phil Harman wrote a very good peice called Multithreading in the Solaris Operating Environment[PDF] which explains in great detail the switch from MxN to 1:1 threading. Phil also posted a very frank comment about the libthread changes in Solaris 10. This is great stuff not only for technical purposes, but also for Unix history buffs.

"Having been an advocate of MxN for many years (largely, because that was one of our differentiators), I was treading a very fine line. I think the paper makes it clear that I don't think 1:1 is superior as a _model_, but that our MxN _implementation_ was past its sell-by date. The switch was entirely pragmatic. I personally had to drag quite a few folk kicking and screaming to the right decision. We have never looked back.

Since then we've added other cool stuff like: compiler support for TLS, posix_spawn() and POSIX spin locks. But our crowning achievement is to get rid of libthread altogether by folding it into libc. None of this would have been possible in the same short time if we had persevered along the MxN route.

You can play with all* of this cool new stuff today via the Solaris Express program (which provides early access to Solaris 10 as we develop it). See www.sun.com for more details. Sorry, I didn't mean this to turn into a marketing opportunity :)

* currently, only the Forte SPARC compiler supports TLS, but we hope to fix that soon! Actually, with the right patches, TLS is available right back to /usr/lib/lwp in Solaris 8." [Phil Harman]

Posted at 02:17PM Jun 10, 2004 by moazam in Solaris | Comments[1] | Permalink

Debugging Backwards in Time - Omniscient Debugging

Rob Reyer from www.evilrob.org commented on my weblog entry about thread dump formatting and analysis. Rob mentions that he spent 6 days trying to get a thread dump right as an application was core dumping. He is not the first person I've heard this from...hell, I have the same problem.

From my outside weblog:

I can't believe that I didn't know about this before! ODB allows you to do 'Omniscient Debugging' on Java applications, with or without the original source code.

The ODB debugger allows "..developers to step backwards through the execution of a program to determine where and how programming errors occurred. By recording each state change in the target application, it allows the developer to navigate "backwards in time" to see what the values of variables and objects WERE, enormously simplifying the task of debugging programs."

One page description of the ODB
Article , Debugging Backwards in Time
AADEBUG paper on Omniscient Debugging (pdf)
ODB User Manual

Posted at 12:10AM Jun 10, 2004 by moazam in JVM Internals | Permalink

Wednesday Jun 09, 2004

Logging GC output to a file, the differences between Xloggc and verbose:gc

I'm sure people have noticed this little idiosyncrasy in the way that garbage collection log output is handled in the JVM. But if you have not, this is a good data point when preparing to read through GC logs.

If you simply use the verbose:gc flag, you'll have GC log output sent to the stdout console. Now, if you use the -Xloggc:[filename] switch, the GC data will be sent to a log file which you can grep through later. But either way, you get the the same GC data ...right? Wrong.

The -Xloggc:[filename] switch has the additional effect of turning on the -XX:+PrintGCTimeStamps switch and hence gives your log files the added benefit of time stamps. Wierd! I'd expect both of these switches to have the same exact behavior. I might file an RFE or Bug on this tonight.

Oh, for those who want to use it, the -Xloggc switch was introduced in 1.4.0 and newer VMs.

Posted at 03:22PM Jun 09, 2004 by moazam in JVM Internals | Comments[1] | Permalink

Tuesday Jun 08, 2004

View from the Trenches: Looking at Thread-Dumps, by Alexandre Rafalovitch

Alexandre Rafalovitch, a BEA Backline Support Engineer emailed me about reading thread dumps and his proposal for formatting dump data. I've read through Alexandres article, and while I'm not a big fan of using XML for everything, he definitely presents an excellent view on making thread dumps easier to read and diagnose. The really cool part is that Alexandre has written an application which takes standard thread dump data, converts it into XML, then uses XSLT to convert the data into the format of your choice.

I'm actually a bit surprised that there isn't a JSR in progress to address this issue. I think one of the reasons for this is that most developers and users are not interested in the nuances of thread dump readability. Of course, as backline engineers, this topic is near and dear to myself and Alexandre. Alexandre will be presenting this in a session at JavaONE 2004, session TS-1646. I'll be sure to attend.

"What would be a perfect solution here is to be able to convert these thread-dumps into some sort of intermediate structure/language that has flexible structure, can tolerate missing fields, and can be processed by third-party tools in various ways. And, of course, if it can somehow involve a currently popular technology, so much the better.

Do we have such an intermediate language? Indeed we do — it is XML. It is somewhat human readable, has many tools written for it, and already has not one but several transformation and reporting technologies built on top of it, such as XSLT, XQuery, and XDB." [full article]

Posted at 11:05PM Jun 08, 2004 by moazam in JVM Internals | Comments[3] | Permalink

Download the source code of the Java SDK

Want to look at the code of the JVM or build a test version of your own? Use the source.

And look, detailed build instructions are available too!

Posted at 10:01PM Jun 08, 2004 by moazam in JVM Internals | Permalink

Newbie SysAdmin Guide to Solaris

This is a quick cheat sheet for SysAdmins or developers who are new to Solaris. I'll update it with more stuff soon.

Posted at 09:50PM Jun 08, 2004 by moazam in Solaris | Permalink

1.4.2 Tuning Garbage Collection Outline

This has to be one of the most useful outlines of GC tuning information I have ever seen. Thanks to Pete Freitag for creating this from the original document. The original was nowhere as easy to read.
1.4.2 GC Tuning Outline by Pete Freitag

Posted at 03:39PM Jun 08, 2004 by moazam in JVM Internals | Permalink

J2SE Update Release Cycles

Many people are running on very old minor releases of J2SE. I've seen companies running 1.2.2_04 while 1.2.2_17 is out! Here is some info on how the J2SE update cycle works, and why you should upgrade to the newest update releases.
J2SE Update Release Cycles

Posted at 03:14PM Jun 08, 2004 by moazam in Java | Permalink

MaxPermSize and how it relates to the overall heap

Many people have asked if the MaxPermSize value is a part of the overall -Xmx heap setting or additional to it. There is a GC document on the Sun website which is causing some confusion due to a somewhat vague explanation and an errant diagram. The more I look at this document, the more I think the original author has made a subtle mistake in describing -Xmx as it relates to the PermSize and MaxPermSize.
Read more..

Posted at 01:45PM Jun 08, 2004 by moazam in JVM Internals | Permalink

J2SE Codenames

This little list helps to make sense of those J2SE BugIDs which reference J2SE code names.
Here, or here.

Posted at 01:43PM Jun 08, 2004 by moazam in Java | Permalink

Debugging thread related hangs in the JVM

Once in a while Java users and developers run into problems where their Java application simply seems to hang. No core file is generated, no IO is detected, the process just sits there waiting...for something. Usually these problems can be traced to OS and JVM level threading.
Debugging thread related hangs in the JVM

Posted at 09:30AM Jun 08, 2004 by moazam in JVM Internals | Permalink

Visualizing Garbage Collection

There are times when reading through megabytes of GC logs is overkill, especially when all you need is a general overview on how the JVMs GC algorithms are behaving with your application. The 'jvmstat' package helps developers and users visualize GC instead.
Visualizing Garbage Collection with jvmstat

Posted at 09:11AM Jun 08, 2004 by moazam in JVM Internals | Comments[2] | Permalink

One weblog is not enough.

About Me

My name is Moazam Raja and currently I'm a backline Java/JDK engineer at Sun Microsystems. This means that I support the Java Virtual Machine and whatever is bundled with the JDK. Most of my day is spent looking through core dumps, garbage collector logs, pstacks, and Java stack traces. While it may sound gory, it's actually alot of fun.

I maintain my personal weblog on my own site which talks mainly about Java and Java Virtual Machine internals. I will probably use this Sun hosted weblog to talk more about my experiments with Solaris 10/Express and Zones.

My interests:

Unix OS Internals
JVM Internals
Porsche 911
Cannondales...

My writings and recommendations are my own and are not necessarily those of Sun Microsystems.