JMX, SNMP, Java, etc...

JMX is a wonderful tool to monitor and troubleshoot running applications. The new JDK 6 Attach API makes it very easy to attach to a running Java process, and start a JMX agent that will expose monitoring and configuration data to JMX consoles - like JConsole. However, there are some situations where you wish to start a JMX agent on demand, explore the monitoring data or diagnose the probable cause of an observed problem, and then close your JMX agent, leaving the application just how you found it.

In this post, I will discuss a means by which you can upload and start such a remotely stoppable JMX agent. Here is how.

Starting an agent on demand

In some of my previous posts I have outlined how to use a premain agent in order to create a custom JMX RMI Connector Server. In this post, we're going to reuse much of the concepts explained before. However, we will add the ability to dynamically upload the JMX agent in the running application by creating an Agent-Main class. In fact, an Agent-Main class is nothing more than a class which has a method:

        public static void agentmain(String agentArgs);

Our agentmain method will create and start a Remotely Stoppable JMX RMI Connector Server. Later on, when we're done with monitoring our application, we will be able to stop the JMX Connector Server that was created by our agentmain, thus leaving the application in its original state (or close to it).

Custom agents started on demand cannot be stopped

The problem with custom agents created on demand is that they can't be stopped: you can't stop them at the end of the agentmain() or premain() method, because it's precisely at this point that you will start using them. And the problem with running custom agents is that they usually hold some non daemon threads alive (e.g. RMI DGC...) and thus prevent the application in which they run from exiting when all its non-daemon threads have finished working.

This behaviour is a much desirable feature in a standalone JMX application, where you can control when to start and stop your JMX Connector Server - but it leads to awkward situations in the case of JMX Agents started from premain() or agentmain().

In a previous post I have suggested a work around which consisted on creating a CleanThread daemon, which would randomly select a non-daemon thread (excluding those kept alive by the presence of a started RMI JMXConnectorServer), join it, and proceed again with the same logic until all non-daemon threads that remain are the "RMI Reaper" and the "DestroyJavaVM" threads. At that point, the CleanThread daemon would simply stop the JMXConnectorServer, which would make it possible for the RMI Reaper thread to terminate and for the application to exit normally.
This trick however presents several limitations:

• It only works with the JMX RMI/JRMP connector. There's no way to do the same thing for the JMX RMI/IIOP connector, for instance.
• It doesn't allow you to stop the connector when you no longer need it. The connector will stay started until the application exits.

In this post, we will show how to:

• Dynamically load and start a Remotely Stoppable JMX RMI Connector Server - using an Agent-Main class
• Perform some remote monitoring action through this connector - e.g. using JConsole from a remote machine.
• Then remotely stop the JMX RMI Connector Server, leaving the application running in a clean state.

If needed, the JMX Connector Server can also be re-created and re-started later on. To make things less easy, we will do all of this with a secure firewall-friendly (one single port) RMI Connector Server.

Making a Remotely Stoppable JMX Connector Server

click to enlarge picture

The trick I will be suggesting here is simply to use an MBeanServerForwarder which will hold a reference to the JMX Connector Server on which it is configured. Our MBeanServerForwarder is a simple pass-through which passes on all requests it receives to an underlying MBeanServer. The MBeanServerForwarder can be configured with a series of InvokeOperationInterceptors. The InvokeOperationInterceptors are used to intercept a given invoke operation directed at an MBean, or at a set of MBeans defined by an ObjectName pattern. In our case, we will register a single InvokeOperationInterceptor (red blob on the picture) which will intercept a stop operation directed at a "fake" MBean. When an invoke operation directed at that "fake" MBean reaches the MBeanServerForwarder, the MBeanServerForwarder diverts it to our InvokeOperationInterceptor. If the operation is stop(), the InvokeOperationInterceptor will stop the JMX Connector Server. The "fake" MBean is not returned by queryNames() or queryMBeans() since it is not registered in the underlying MBeanServer. The name of the "fake" MBean must thus be a secret shared between the program that will create and start the connector server, and the program that will remotely stop the connector server. Using such a shared secret is acceptable in our case because we're using a secure JMX Connector Server. You will also see that my implementation is a bit lax - because it stores that secret in a System property (the secret is not too heavily guarded).

Project Sources

The code source that does all of this comprises the following classes:

• StoppableAgent.java: our Agent-Main class. Can also be used as a premain agent.
• ForwardingInterceptor.java: our configurable MBeanServerForwarder, implemented as a java.lang.reflect.Proxy.
• InvokeOperationInterceptor.java a default InvokeOperationInterceptor which acts as a pass-through.
• Stopper.java: creates and configure a ForwardingInterceptor that can stop its connector server. Can also send a stop request through a given JMXConnector
• Attach.java: a command line tool. The RMI port and secret name are passed through system properties:
• start <pid>: uses the Attach API to dynamically start a StoppableAgent in the target process.
• status: creates a JMXConnector and checks whether the remote server that was started in a previous call is still alive (useful for tests).
• list: list running java processes
• stop: creates a JMXConnector to connect to the StoppableAgent and invoke stop on the "fake" MBean.
• help: prints a crude help message

You can see the example sources here in the form of a NetBeans 6 project.

Building the StoppableAgent jar

Once you have compiled all the classes, you can create an agent jar by invoking an ant target like this one in your build.xml (already done in the provided NetBeans 6 project):

    <!-- Builds dist.agent.jar -->    
    <target name="-build-agent-jar"
        description="build an agent jar that can be used either with -javaagent or the Attach API">
        <jar basedir="${build.classes.dir}" 
            jarfile="${dist.agent.jar}">
                <manifest>
                    <attribute name="Premain-Class" value="example.rmi.agent.StoppableAgent"/>
                    <attribute name="Agent-Class" value="example.rmi.agent.StoppableAgent"/>
                </manifest>
        </jar>
        <echo>To use this application with agent try:</echo>
        <echo>java &lt;SSL Properties&gt; -Dexample.rmi.agent.port=3000 -Dexample.rmi.agent.stopper=FooBar -cp:${dist.agent.jar} example.rmi.agent.Attach start &lt;pid&gt;</echo>
    </target>
                            

If you're using the provided NetBeans 6 project simply clean & build the StoppableAgent project: the agent jar (dist/jagent.jar) will be automatically generated.

Loading and Starting the agent in a Running application

The StoppableAgent class creates a secure JMX RMI Connector Server. This means that the target application in which the agent will be loaded must have an appropriate truststore and keystore. The Attach class provided in this example will forward its own default SSL configuration to the target application if the example.rmi.agent.ssl.config.send System property is set to true. In the general case, this is a very bad idea as it is profundly unsecure - and this is why the default is example.rmi.agent.ssl.config.send=false. A much better idea would be to configure SSL properly when starting the target application.

If you don't have a target application to play with, you could use this one:

    public class Test { 
        public static void main(String[] args) throws Exception {
            System.out.println("Strike Enter to exit:");
            System.in.read();
        }
    }
        

Compile it and start it as follow:

   
   java -classpath . \
        -Djavax.net.ssl.keyStore=<keystore> \
        -Djavax.net.ssl.keyStorePassword=<password> \
        -Djavax.net.ssl.trustStore=<truststore> \
        -Djavax.net.ssl.trustStorePassword=<trustword> \
        Test
        

Note: If you haven't already a keystore and trustore to play with, the Monitoring and Management guide has a section that explains how to create them.

To start the stoppable agent in a target application use the following command:

   
   java -Dexample.rmi.agent.port=<port> \
        -Dexample.rmi.agent.stopper=<secret-name> \
        -cp <agent.jar>:<jdk.home>/tools.jar \
        -Djavax.net.ssl.keyStore=<keystore> \
        -Djavax.net.ssl.keyStorePassword=<password> \
        -Djavax.net.ssl.trustStore=<truststore> \
        -Djavax.net.ssl.trustStorePassword=<trustword> \
        example.rmi.agent.Attach start <pid>
        

To check that the StoppableAgent was correctly started run the following command:

  
   java -Dexample.rmi.agent.port=<port> \
        -Dexample.rmi.agent.stopper=<secret-name> \
        -cp <agent.jar>:<jdk.home>/tools.jar \
        -Djavax.net.ssl.keyStore=<keystore> \
        -Djavax.net.ssl.keyStorePassword=<password> \
        -Djavax.net.ssl.trustStore=<truststore> \
        -Djavax.net.ssl.trustStorePassword=<trustword> \
        example.rmi.agent.Attach status
        

Connecting from remote

To connect with JConsole, start it with the following command line:

  
   jconsole -J-Djavax.net.ssl.keyStore=<keystore> \
        -J-Djavax.net.ssl.keyStorePassword=<password> \
        -J-Djavax.net.ssl.trustStore=<truststore> \
        -J-Djavax.net.ssl.trustStorePassword=<trustword>
        

Then in the JConsole connection window select Remote Connection and simply type <host>:<port> in the entry field - and that's it! JConsole will connect to your application through your JMX RMI SSL Connector.

Stopping the agent

To stop the agent, you now only need to invoke the following command:

  
   java -Dexample.rmi.agent.port=<port> \
        -Dexample.rmi.agent.stopper=<secret-name> \
        -cp <agent.jar>:<jdk.home>/tools.jar \
        -Djavax.net.ssl.keyStore=<keystore> \
        -Djavax.net.ssl.keyStorePassword=<password> \
        -Djavax.net.ssl.trustStore=<truststore> \
        -Djavax.net.ssl.trustStorePassword=<trustword> \
        example.rmi.agent.Attach stop
        

Some additional considerations

The 'start' command can only be invoked from the local machine where the application is running. This is because we use the Attach API to upload the stoppable agent.

However, once done, the agent can be accessed from remote in a secure way. Since 'status' and 'stop' use a regular JMX connection with the stoppable agent, they could also be invoked from a remote machine.

Cheers,
-- daniel