Kelly O'Hair's Weblog (blogs.sun.com)

DVM: Java VM Agents and Solaris 10 DTRACE

The Solaris 10 Dynamic Tracing (DTrace) feature is an extremely powerful and popular feature, but it's only as good if you have providers of the DTrace probes you need. The Java Virtual Machine (VM) is a case in point. The current VM in development (6.0, code named "Mustang") will contain official DTrace probed, but what do people do til then on older VMs. Unless the extern interfaces of a native library and their use represent the necessary DTrace probes or events of interest, the probes need to be baked into the actual code somehow (using the macros of Solaris 10). With Java using the extern interfaces of it's libraries just won't work, since most of the interesting events are hidden, buried deep inside the VM, like class loads or garbage collects.

So a very energetic Jarod Jensen caused a bit of a stir about this (a good stir) when he created a JVM PI agent that provides some minimal DTrace probes in existing VM's, namely 1.4.2 and 5.0. The JVM PI is an old experimental VM interface that was created back in JDK 1.2 days and has never been very stable when the HotSpot VM came into existence in JDK 1.3. But JDK 5.0 (Tiger) created a new official and fully supportted VM interface that replaces JVM PI called "JVM TI" (or "tool interface" as it's often referred to, see JVM TI). The JVM PI may be removed in JDK 6.0 (Mustang) and most of the VM engineers get big smiles on their faces when they hear this, it's been a very problematic interface, yet extremely valuable.

So what exactly IS a VM agent? Quite simply, it's a shared library that gets dynamically loaded into a VM at startup. The Java debugger and the HPROF profiling tool use agents, well HPROF IS an agent. So when you see the java option -Xrun[:OPTIONS] used (e.g. -Xrunhprof:cpu=samples), that means that the VM will dlopen("libAGENTNAME.so"), and call an external inside that library passing it the OPTIONS string. The JVM PI agent libraries use the name JVM_OnLoad as the name of the external and JVMTI uses Agent_OnLoad. You can see all these gory details in the source at https://solaris10-dtrace-vm-agents.dev.java.net. Once inside the agent library (JVM_OnLoad or Agent_OnLoad) the agent library can setup for further callbacks into the agent library at selected events, and return back to the VM. The DTrace probes can be placed in the event callback functions of the agent library.

So taking Jarod's agent as a basis and a few days work, I created JVM PI and JVM TI agents called "dvmpi" and "dvmti" and placed them at: https://solaris10-dtrace-vm-agents.dev.java.net

Now this is all just a big experiment, and the source is open and available for people to tweek or play with, it has not been extensively tested so be careful. Several bugs have been fixed recently and the names have been changed for legal reasons. One of the issues with doing DTrace probes this way is that often the necessary arguments that are sent into the probes has to be generated by the agent library, these arguments (mostly strings) aren't just sitting around, there is overhead in creating the information necessary for the probes. This performance penalty should not be too severe, but it's there. Hopefully, in Mustang, it's DTrace probes will be much lower overhead, which will be a requirement since the probes will be there all the time, with these agents, they only exist if the agent was loaded (with -Xrun). The agents have various options to control what probes are available.

Mustang or JDK 6.0 should have an "attach on demand" feature so that an agent could be loaded into an already running VM. The complete details aren't available yet, and there may be some limitations as to what an agent can do when loaded into a VM late (after VM initialization), but the capability has some great potential. Keep in mind that an agent loaded in after VM initialization has a more difficult job on it's hands, it may need to redefine classes that it needs instrumented, and all the events that happened before it is loaded into a VM were lost, so it will need to adapt to that.

Posted at 01:44PM Apr 13, 2005 by kto in Java  |