JMX, SNMP, Java, etc...

Daniel Fuchs blogs on JMX, SNMP, Java, etc...

All | Personal | Sun

tags: blogging firewall hg java jconsole jmx jvm management mbean mercurial monitoring opendmk openjdk opensource rmi snmp ssl
Table Of Contents (list all entries)

« Java Real Time video... | Main | More on premain and... »

Wednesday July 25, 2007

Connecting Through Firewall Using JMX - Without modifying the server application

In a recent comment on my blog about Troubleshooting Connection Problems in JConsole I was asked the following question:

Can you simply explain how run jconsole on a client machine to connect to an unmodifiable applicaton that run on a server. One and only one port on the server is open though firewall for managing the application?

Here is how.

As you may already know if you have been confronted with this problem, the JMX RMI connector opens two ports: one is for the RMI registry, and it's the port that you usually supply with the -Dcom.sun.management.jmxremote.port=<port> property. The other port is used to export JMX RMI connection objects. This second port is usually dynamically allocated at random. Indeed you don't need to know this port number in order to connect to the JMX agent: the only port number you need to know to connect is the RMI registry port number from which to obtain the connection objects.

This however can prove to be troublesome if your application is behind a firewall that block access to random ports. The default JVM agent will not let you specify that second port number, and you're stuck. The only way to specify that second port number is to use a JMXServiceURL, but you can't supply a JMXServiceURL to the default agent.

Fortunately, there's a work around around this, which makes use of the JDK dynamic agent loading feature. Instead of starting the default JVM agent, we will start our own custom agent, without modifying the server application.

Here is how to that:

Using a Java Agent

First, we will create a premain agent. The code below show how to create a premain agent that will start an RMI Connector server.

Update: see also my following entry on this subject.

Update 2: my post on Building a Remotely Stoppable Connector also has an example that implements a complete solution using RMI over SSL.

/*
 * CustomAgent.java
 *
 * Copyright 2007 Sun Microsystems, Inc.  All Rights Reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 *
 *   - Redistributions of source code must retain the above copyright
 *     notice, this list of conditions and the following disclaimer.
 *
 *   - Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *
 *   - Neither the name of Sun Microsystems nor the names of its
 *     contributors may be used to endorse or promote products derived
 *     from this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
 * IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 * 
 * Created on Jul 25, 2007, 11:42:49 AM
 * 
 */

package example.rmi.agent;

import java.io.IOException;
import java.lang.management.ManagementFactory;
import java.net.InetAddress;
import java.rmi.registry.LocateRegistry;
import java.util.HashMap;
import javax.management.MBeanServer;
import javax.management.remote.JMXConnectorServer;
import javax.management.remote.JMXConnectorServerFactory;
import javax.management.remote.JMXServiceURL;

/**
 * This CustomAgent will start an RMI COnnector Server using only
 * port "example.rmi.agent.port".
 *
 * @author dfuchs
 */
public class CustomAgent {

    private CustomAgent() { }

    public static void premain(String agentArgs) 
	throws IOException {

        // Ensure cryptographically strong random number generator used
        // to choose the object number - see java.rmi.server.ObjID
        //
        System.setProperty("java.rmi.server.randomIDs", "true");

        // Start an RMI registry on port specified by example.rmi.agent.port
        // (default 3000).
        //
        final int port= Integer.parseInt(
                System.getProperty("example.rmi.agent.port","3000"));
        System.out.println("Create RMI registry on port "+port);
        LocateRegistry.createRegistry(port);

        // Retrieve the PlatformMBeanServer.
        //
        System.out.println("Get the platform's MBean server");
        MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();

        // Environment map.
        //
        System.out.println("Initialize the environment map");
        HashMap<String,Object> env = new HashMap<String,Object>();
        
        // This where we would enable security - left out of this
        // for the sake of the example....
        //

        // Create an RMI connector server.
        //
        // As specified in the JMXServiceURL the RMIServer stub will be
        // registered in the RMI registry running in the local host on
        // port 3000 with the name "jmxrmi". This is the same name the
        // out-of-the-box management agent uses to register the RMIServer
        // stub too.
        //
        // The port specified in "service:jmx:rmi://"+hostname+":"+port
        // is the second port, where RMI connection objects will be exported.
        // Here we use the same port as that we choose for the RMI registry. 
        // The port for the RMI registry is specified in the second part
        // of the URL, in "rmi://"+hostname+":"+port
        //
        System.out.println("Create an RMI connector server");
        final String hostname = InetAddress.getLocalHost().getHostName();
        JMXServiceURL url =
            new JMXServiceURL("service:jmx:rmi://"+hostname+
            ":"+port+"/jndi/rmi://"+hostname+":"+port+"/jmxrmi");
        
        // Now create the server from the JMXServiceURL
        //
        JMXConnectorServer cs =
            JMXConnectorServerFactory.newJMXConnectorServer(url, env, mbs);

        // Start the RMI connector server.
        //
        System.out.println("Start the RMI connector server on port "+port);
        cs.start();
    }
}

Then we will compile this class and create a jar from it, specifying a Premain-Class attribute in its manifest. Here is how you can do this with ant/NetBeans:

    <!-- Builds dist.agent.jar -->    
    <target name="-build-agent-jar"
        description="build an agent jar that can be used with -javaagent ">
        <jar basedir="${build.classes.dir}" 
            jarfile="${dist.agent.jar}">
                <manifest>
                    <attribute name="Premain-Class" value="example.rmi.agent.CustomAgent"/>
                </manifest>
        </jar>
        <echo>To use this application with agent try:</echo>
        <echo>java -Dexample.rmi.port=3000 -javaagent:${dist.agent.jar} -classpath <application-classpath> <application-main-class></echo>
    </target>

Now you only need to start your server application with the additional magic flags:

          -Dexample.rmi.agent.port=<port> -javaagent:<agent.jar>

Adding Security

To add security to the above example, you will need to add the appropriate security parameters to the env map in the CustomAgent example above.

One way to do that is to statically code the various parameters (password file etc...) in your premain CustomAgent, as shown here.

Another way is to mimic the behaviour of the default JVM agent in your premain CustomAgent.

Also don't forget that you will also need to supply the appropriate security parameters to JConsole!

You will find all the relevant information about monitoring applications through a firewall here, in the Java SE Monitoring and Management Guide.

Update: It is also worth noting that if you put security on (in particular SSL), you might need to use two different port numbers. An easy way to do that could be to change the JMXServiceURL creation:

        // We use (port+1) to export the RMI connection objects
        JMXServiceURL url =
            new JMXServiceURL("service:jmx:rmi://"+hostname+
            ":"+(port+1)+"/jndi/rmi://"+hostname+":"+port+"/jmxrmi");

You would then need to configure your firewall to let through both <port> and <port+1> (or whatever second port number you choosed).

If you wish to use a single port number, and still use RMI over SSL, then you will need to take care to use the same RMI Socket Factories everywhere, for exporting the RMI Registry and for exporting the JMX RMI Connection objects. I have detailed these steps in a follow-up: JMX: Connecting Through Firewalls Using RMI Over SSL And a Single Port.

Hope this will help you!

Cheers,
--daniel

Tags: firewall java jconsole jmx jvm management monitoring rmi
Posted by dfuchs ( Jul 25 2007, 04:21:36 PM CEST ) Permalink Comments [37]

Trackback URL: http://blogs.sun.com/jmxetc/entry/connecting_through_firewall_using_jmx

Comments:

I am experimenting with JMX and my application running on Tomcat (5.5.23). I have succesfully got my bean to appear within JConsole, but not when the server is deployed to the production system, where it is protected by a firewall.

Do you know if the techniques you suggest here will work with Tomcat? I.e. does Tomcat object to you creating your own RMI Connector Server.

Cheers,

Posted by Des Whewell on November 06, 2007 at 02:15 PM CET #

Hi,

I don't know if Tomcat will object - but that shouldn't be too difficult to find out. If you're planning to use a pre-main agent however, you will need to use JDK 6. And if you're planning to use SSL - you will need to follow the steps described on my follow-up post.
http://blogs.sun.com/jmxetc/entry/jmx_connecting_through_firewalls_using

If Tomcat is running on JDK 5, and if you want to use RMI over SSL, then you might need to create your RMI Connector Server by hand (rather than going through the JMXConnectorServerFactory), in order to control the port# and RMI Socket Factories that are used when creating the RMI registry and when exporting the RMIServer object through the RMI registry.

Hope this helps,

-- daniel

Posted by daniel on November 06, 2007 at 02:40 PM CET #

Thanks, daniel. The 1.6 requirement is a bummer, as I'm constrained to 1.5. Ho hum.

Posted by Des on November 07, 2007 at 09:04 AM CET #

It appears that the 'premain' facility is mentioned in the 1.5 Javadocs. Is it in fact available in 1.5, or are there limitations in 1.5 that make it unsuitable in this application (ie JMX/firewall)?

Posted by Des Whewell on November 07, 2007 at 09:38 AM CET #

Hi Des,

Yes, in fact it's my mistake. Apparently it's only the static void premain(String) which is new in 1.6. If you simply add:

... public static void premain(String agentArgs, Instrumentation inst)
... ... throws IOException {
... ... premain(agentArgs);
... }

to the CustomAgent.java class it should work on 1.5 too. I've just tested it using JDK 1.5.0_13 and it works.

However if you are using SSL you will still have to go through the process of creating your connector server manually in order to use a single port...

-- daniel

Posted by daniel on November 07, 2007 at 12:37 PM CET #

Thanks, daniel :-)

Posted by 194.237.142.7 on November 07, 2007 at 01:04 PM CET #

Hi Daniel,
I'm trying to use your above technique on Glassfish, and I'm getting exceptions when I do the "asadmin start-domain":

Could not load Logmanager "com.sun.enterprise.server.logging.ServerLogManager"
java.lang.ClassNotFoundException: com.sun.enterprise.server.logging.ServerLogManager
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.util.logging.LogManager$1.run(LogManager.java:166)
at java.security.AccessController.doPrivileged(Native Method)
at java.util.logging.LogManager.<clinit>(LogManager.java:156)
at java.util.logging.Logger.getLogger(Logger.java:254)
at sun.rmi.runtime.Log$LoggerLogFactory.createLog(Log.java:154)
at sun.rmi.runtime.Log.getLog(Log.java:121)
at sun.rmi.server.Util.<clinit>(Util.java:56)
at sun.rmi.server.UnicastRef.<clinit>(UnicastRef.java:33)
at java.rmi.server.RemoteServer.<clinit>(RemoteServer.java:97)
at java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:186)
at com.sixth.rmi.agent.CustomAgent.premain(CustomAgent.java:78)
at com.sixth.rmi.agent.CustomAgent.premain(CustomAgent.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:141)
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:141)
Caused by: javax.management.JMRuntimeException: Failed to load MBeanServerBuilder class com.sun.enterprise.admin.server.core.jmx.AppServerMBeanServerBuilder: java.lang.ClassNotFoundException: com.sun.enterprise.admin.server.core.jmx.AppServerMBeanServerBuilder
at javax.management.MBeanServerFactory.checkMBeanServerBuilder(MBeanServerFactory.java:499)
at javax.management.MBeanServerFactory.getNewMBeanServerBuilder(MBeanServerFactory.java:530)
at javax.management.MBeanServerFactory.newMBeanServer(MBeanServerFactory.java:304)
at javax.management.MBeanServerFactory.createMBeanServer(MBeanServerFactory.java:219)
at javax.management.MBeanServerFactory.createMBeanServer(MBeanServerFactory.java:180)
at sun.management.ManagementFactory.createPlatformMBeanServer(ManagementFactory.java:264)
at java.lang.management.ManagementFactory.getPlatformMBeanServer(ManagementFactory.java:512)
at com.sixth.rmi.agent.CustomAgent.premain(CustomAgent.java:83)
at com.sixth.rmi.agent.CustomAgent.premain(CustomAgent.java:61)
... 5 more
Caused by: java.lang.ClassNotFoundException: com.sun.enterprise.admin.server.core.jmx.AppServerMBeanServerBuilder
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at javax.management.MBeanServerFactory.loadBuilderClass(MBeanServerFactory.java:441)
at javax.management.MBeanServerFactory.checkMBeanServerBuilder(MBeanServerFactory.java:484)
... 13 more
Exception in thread "main"

Is the premain technique you describe above compatible with GF (I'm running v2-b58c). Are there any domain.xml settings I need to tweak for class loading, etc?

Thanks.

Posted by Curt Tudor on November 08, 2007 at 11:53 PM CET #

Hi Curt,
I'm not sure you can use a java agent to start your connector with GF - from the stack trace you can see that there is a class loading issue there, so it looks as if your agent comes too early in the game, or is not loaded by the appropriate ClassLoader.
If you want to start an alternate JMX Connector Server in glassfish, I assume that there are other better means to do that.
However I haven't tried to experiment with GF yet.

Posted by daniel on November 09, 2007 at 11:05 AM CET #

Hi,

I'm trying to monitor a java app on a remote machine, that is behind a firewall, and I only have SSH access.

I'm using the agent described above. When I run jconsole on the remote server (with X11 forwarding enabled), it works fine, except that the performance is horrible. Changing a tab takes about 30 seconds! I suspect that the combination of my bandwith and the amounts of data X11 forwarding produces is the problem here.

So I was thinking of trying to run jconsole locally, and let it connect to the remote app through an SSH tunnel (hoping that this generates less network traffic). I've set up the tunnel, and I can connect to the port using telnet. However when I start jconsole, and let it connect, it times out and fails!. I can't figure out why this doesn't work...

I've enabled detailed logging for jconsole:

jconsole -J-Djava.util.logging.config.file=logging.properties localhost:9999
Feb 26, 2008 1:29:04 PM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi] connecting...
Feb 26, 2008 1:29:04 PM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi] finding stub...
Feb 26, 2008 1:30:19 PM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi] connecting stub...
Feb 26, 2008 1:30:19 PM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi] getting connection...
Feb 26, 2008 1:31:34 PM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi] failed to connect: java.rmi.ConnectException: Connection refused to host: 127.0.1.1; nested exception is:
java.net.ConnectException: Operation timed out

Thanks for any help!

Tom

Posted by Tom van den Berge on February 26, 2008 at 01:45 PM CET #

Hi Tom,

I haven't experimented much with ssh tunelling, but have you tried to start your application with a -Djava.rmi.server.hostname=<hostname> property?

The message you get is "Connection refused to host: 127.0.1.1" (BTW: isn't that 127.0.0.1?) let me think that the stub you're retrieving from the remote machine doesn't have the correct address embedded.

This sometimes happens on linux boxes - where localhost might resolve to the loopback address only.

Hope this helps,

-- daniel

Posted by daniel on February 26, 2008 at 02:18 PM CET #

Hi Daniel,

Oh, you're the best!! I've set
-Djava.rmi.server.hostname=localhost on the remote machine, and it works. When I start jconsole locally, I can see that it connects to "localhost" instead of 127.0.1.1.

The other good thing is that the performance of this setup (jconnect through an ssh tunnel) is a thousand times better than running jconnect on the remote server with X11 Forwarding.

Thanks a million Daniel.

Tom

Posted by Tom van den Berge on February 26, 2008 at 02:48 PM CET #

Hi Daniel!

First of all thanks for all your great posts about JMX! I am currently trying to manage a webapp running in Tomcat 6 using JMX and reused your code above to create a connector server during the start of the web application and shutting it down when the app is stopped. This is working great. What's not working is, to connect to the application once it is deployed on a server that I can only access through an SSH tunnel.

My code uses two ports that I can specify, one for the exported connections (8117) and one for the registry (8116). I created a tunnel to both of them and set the hostname of the application to "localhost". When I try to connect to the app with JConsole using "localhost:8116", JConsole hangs for about 30 secs and comes up with "error during JRMP connection establishment" because of a read timeout. I tried a myriad of settings (premain agent, using a single port instead of two, specifying the JMXServiceUrl in JConsole), but to no avail. Do you have any further hints for me in which direction to look? BTW the SSH tunnels seem to work, because if I do not create them I get an immediate exception "connection refused"...

Thanks in advance

Sven

Posted by Sven on March 06, 2008 at 10:04 AM CET #

Hi Sven,

The behaviour you describe let me suspect a hostname resolution issue. Could you try to
specify -Djava.rmi.server.hostname=localhost
on the server command line?

I believe that when tunneling through SSH the
stub returned by the server needs to have its
hostname set to 'localhost'.

see:
http://www.javaranch.com/journal/2003/11/rmi-ssh_p2.html

-- daniel

Posted by daniel on March 06, 2008 at 10:57 AM CET #

Hi Daniel,

I already set the hostname to localhost, but that didn't help. I created a small Java app that tried to connect and here is what I get:

Mar 6, 2008 11:08:48 AM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi://localhost:8117/jndi/rmi://localhost:8116/jmxrmi] connecting...
Mar 6, 2008 11:08:48 AM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi://localhost:8117/jndi/rmi://localhost:8116/jmxrmi] finding stub...

It hangs for about 30 secs at that state before it fails with this

Mar 6, 2008 11:09:48 AM RMIConnector connect
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi://localhost:8117/jndi/rmi://localhost:8116/jmxrmi] Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
java.net.SocketTimeoutException: Read timed out]
Exception in thread "main" java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
java.net.SocketTimeoutException: Read timed out]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:323)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:207)
at net.netm.me.JmxTest.main(JmxTest.java:19)
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
java.net.SocketTimeoutException: Read timed out]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1871)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1841)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257)
... 3 more
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
java.net.SocketTimeoutException: Read timed out
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:286)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
... 8 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
at java.io.DataInputStream.readByte(DataInputStream.java:239)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:228)
... 12 more

Does that make any sense to you?

Thanks

Sven

Posted by 62.214.145.200 on March 06, 2008 at 11:12 AM CET #

Hi Sven,

Are you using JConsole on the client side?
If so try to simply specify
localhost:8116
in the connection window - and not the full JMXServiceURL...
And check that 8116 is really the port number where the RMI registry is listening.

-- daniel

Posted by daniel on March 06, 2008 at 11:21 AM CET #

Hi Daniel,

the stacktrace as well as the log messages where produced by a simple Java App (not JConsole). When I use JConsole I always tested with localhost:8116, since specifying the JMXServiceUrl immediately resulted in a connection failure. Interestingly enough I do not get any logging output from JConsole, even if I specify -J-Djava.util.logging.config.file and pointing it to an existing file with FINEST log level, so I cannot show an output of JConsole here...

Thanks

Sven

Posted by Sven on March 06, 2008 at 11:34 AM CET #

Hi Sven,

Sorry for the late answer. Not sure whether it will make a difference, but can you try
using these JMXServiceURL on the client side?

service:jmx:rmi:///jndi/rmi://localhost:8116/jmxrmi
service:jmx:rmi:///jndi/rmi://127.0.0.1:8116/jmxrmi
service:jmx:rmi:///jndi/rmi://<servername>:8116/jmxrmi

Another things that comes to my mind: is your RMI registry secured with SSL? If so you will need to specify keystore & truststore etc... on the client side and also provide an RMI SSL socket factory to the client connection map:
env.put("com.sun.jndi.rmi.factory.socket", csf);
(requires Java SE 6)

It is strange that you do not get any logging output from JConsole. Did you try the logging.properties file from here:
http://blogs.sun.com/jmxetc/entry/troubleshooting_connection_problems_in_jconsole

Also you could try passing the -debug option to jconsole...

Hope this helps,
-- daniel

Posted by daniel on March 21, 2008 at 11:01 AM CET #

Hi,
I have couple of questions related Jmx
1) how can I connect with the Jconsol without the command like -Dcom.sun.management.jmxremote SimpleAgent . actualy I have to give the funtionality in which one application can monitore the other so where to write this command
So is there any way I don’t need to used command and connect to JConsol.

2) I have couple of different application which I want to monitor and they are on different servers. So where I have to register my mbean or where I can put the servlet in which I registered the Mbean. So that every application I have to monitor.

Thanks in Advance
Jon

Posted by Ankita on April 03, 2008 at 11:05 PM CEST #

Hi Jon,

1) You can connect JConsole without the command line, but only if the agent runs on top of JDK 6. If it runs on JDK 5, then you need the special command line option. However, in both cases - to connect through the Attach API, JConsole must run on the *same* machine than the managed process.

When the process you want to manage is remote, then it needs to have remote management enabled.
If remote management is enabled (e.g. using the -Dcom.sun.management.jmxremote.port=port flag), then JConsole will be able to connect from a remote machine (unless your firewall prevents
it).
The post below shows how remote management can be dynamically enabled - provided that you can rlogin on the machine where the server runs in order to enable it.
http://blogs.sun.com/jmxetc/entry/building_a_remotely_stoppable_connector

2) I don't understand the question completely: Each server should have its own MBeans exposing monitoring data. If you need a single point of entry you could possibly use something like cascading, but I am not sure this is what you're looking for. Is it?

Hope this helps,
-- daniel

Posted by daniel on April 04, 2008 at 04:40 PM CEST #

Thanks Daniel,

I am using sun java application server, and building a jmx component and my management resource is in remote. So how can they see the Jconsol that I am not understanding. (meance how the JConsol will start for them)
please give me some document which describes that if possible.
Is jdk 6 necessory?

I am going to build a jmx component which is monitoring 4-5 application which is on remote place so how should I approch them.

Thanks in Advance

Posted by Jon on April 04, 2008 at 06:59 PM CEST #

Hi,

I am using Sun App Server and want to monitor one application. my application and Mbean both are on the same matchine rightnow. and its working fine when I start the Jconsole for the local. but when change the code and try to connect with the Jconsole it not able to connect and gives me the connection failed error my code is like this

JMXServiceURL url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://localhost:9999/server");
JMXConnectorServer cs = JMXConnectorServerFactory.newJMXConnectorServer(url, null, mbs);
cs.start();

Please suggest something.
thanks,
Ankita

Posted by Ankita on April 10, 2008 at 07:13 AM CEST #

Hi Ankita,

You're showing the server code. Are you sure the JMXConnectorServer started properly? Or does cs.start() throws an exeption?

You do know that you have to start an RMI registry before starting the RMI JMXConnectorServer - right?

Hope this helps,

-- daniel

Posted by daniel on April 10, 2008 at 03:12 PM CEST #

Hi Daniel. Great Job you and Luis.
It seems that JMX info is "spread" and is very hard to get a "all-in-one" responses.

I'm not a programmer, so, but incredible that it seems... the Ant file u put to complie in Jar is saying errors.

I copied the first one as CustomAgent.java
the ant one i renamed build.xml

and the error is:
Buildfile: build.xml

BUILD FAILED
C:\Documents and Settings\services\Desktop\build.xml:3: Unexpected element "{}target" {antlib:org.apache.tools
.ant}target

Total time: 0 seconds

jdk version: 1.5.0_15

Tranks!

Posted by Carlos on July 11, 2008 at 12:27 PM CEST #

Hi Carlos,

The target shown in this blog is not a 'stand-alone' build.xml file.

You're supposed to copy & paste these few line in a build.xml file generated by the NetBeans IDE.

Also you will probably need to define the value of ${dist.agent.jar} and to add a reference to the -build-agent-jar target in the dependencies of the -post-jar target.

What I suggest you do is:

Create a new J2SE project with NetBeans IDE.
From the IDE create a new java package named
"example.rmi.agent".

Copy the file CustomAgent.java in the folder .\src\example\rmi\agent\ that NetBeans just created.

From NetBeans edit the build.xml file and copy the
following targets inside:
<target name="-post-jar" depends="-init-agent-jar,-build-agent-jar"/>
<target name="-post-clean" depends="-init-agent-jar,-clean-agent-jar"/>

<target name="-init-agent-jar" if="dist.jar.dir">
<property location="${dist.jar}" name="dist.jar.resolved"/>
<property location="${dist.jar.dir}/jagent.jar" name="dist.agent.jar"/>
</target>

<target name="-clean-agent-jar" if="dist.agent.jar"
description="clean the jagent jar">
<delete file="${dist.agent.jar}"/>
</target>

<target name="-build-agent-jar" if="dist.agent.jar"
description="build a jagent jar">
<jar basedir="${build.classes.dir}"
jarfile="${dist.agent.jar}">
<manifest>
<attribute name="Premain-Class" value="example.rmi.agent.CustomAgent"/>
</manifest>
</jar>
<echo>To use this application with agent try:</echo>
<echo>agent="java -javaagent:${dist.agent.jar} -cp ${dist.jar.resolved} example.rmi.agent.Test"</echo>
</target>

Then build your project in NetBeans.
You should find your jagent.jar in the .\dist\ folder.

Hope this helps,

-- daniel

Posted by daniel on July 11, 2008 at 03:03 PM CEST #

Tryed and Worked (at least the Netbeans compiling)
I have 2 .jar files, agent and "try" (the name of the proyect) Does it matter ?

Gonna try it really fast!

THANKS daniel. Will update :)

Posted by Carlos on July 14, 2008 at 02:23 PM CEST #

Hi Carlos,

You need to pass the jagent.jar file using the -javaagent: option to your server process. You can ignore the other jar.

The other jar is generated by default - and doesn't have the necessary attributes in its manifest file.

Cheers,

-- daniel

Posted by daniel on July 15, 2008 at 10:14 AM CEST #

Hi Daniel,

This blog have really very good and important info. Thanks....!!!

Actually I am having problem accessing my application whivh is behind the NAT device.

I am having following code snippet:

----------------------------------------------
env.put( Context.INITIAL_CONTEXT_FACTORY, JBOSS_CONTEXT_FACTORY );
env.put( Context.URL_PKG_PREFIXES, JBOSS_URL_PKG_PREFIXES );
env.put( Context.PROVIDER_URL, host + ":" + port );

// Disable Discovery
env.put( NamingContext.JNP_DISABLE_DISCOVERY, "true" );

// Socket timeout, set to one minute.
env.put( TimedSocketFactory.JNP_SO_TIMEOUT, "60000" );

// Initial connect timeout, set to one minute.
env.put( TimedSocketFactory.JNP_TIMEOUT, "60000" );

Context context = new InitialContext( env );
adaptor_ = ( RMIAdaptor )context.lookup( "jmx/invoker/RMIAdaptor" );
host_ = host;
port_ = port;
}
catch( NamingException ne )
{
cleanup(); // clear security context in case of error

throw new JmxRemoteException( ne );
}
--------------------------------------------------

And I am getting the following exception and stack trace:

-------------------------------------------------
FINE: Creating JMXRemoteException for javax.naming.CommunicationException [Root exception is java.rmi.ConnectException: Connection refused to host: 11.22.33.444; nested exception is:
java.net.ConnectException: Connection timed out: connect]
<R [RSMScheduler_Worker-132],07/24/08 04:14:50 UTC,STDERR> Jul 24, 2008 4:14:50 AM com.bmc.patrol.patsdk.lib.jmx.JmxRemoteException JmxRemoteException
FINE: Creating JMXRemoteException for java.rmi.ConnectException: Connection refused to host: 11.22.33.444; nested exception is:
java.net.ConnectException: Connection timed out: connect
<R [RSMScheduler_Worker-132],07/24/08 04:14:50 UTC,STDERR> Jul 24, 2008 4:14:50 AM com.bmc.patrol.patsdk.lib.jmx.JmxRemoteException JmxRemoteException
FINE: Creating JMXRemoteException for java.net.ConnectException: Connection timed out: connect
<R [RSMScheduler_Worker-132],07/24/08 04:14:50 UTC,STDERR> Jul 24, 2008 4:14:50 AM com.bmc.patrol.patsdk.solutions.jmx.JmxParamlet doExecute()
FINEST: Jmx remote exception:
com.bmc.patrol.patsdk.lib.jmx.JmxRemoteException: javax.naming.CommunicationException : null
at com.bmc.patrol.patsdk.lib.jmx.JBossJmxClient.connect(JBossJmxClient.java:101)
at com.bmc.patrol.patsdk.solutions.jmx.JmxParamlet.doExecute(JmxParamlet.java:271)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)
Caused by: com.bmc.patrol.patsdk.lib.jmx.JmxRemoteException: java.rmi.ConnectException : Connection refused to host: 11.22.33.444; nested exception is:
java.net.ConnectException: Connection timed out: connect
... 9 more
Caused by: com.bmc.patrol.patsdk.lib.jmx.JmxRemoteException: java.net.ConnectException : Connection timed out: connect
----------------------------------------------------

172.87.65.145 is the NAT address and 11.22.33.444 is the real IP address of the host that is behind the NAT device.
There is need to use NAT'ted addresses is a must in my environment. As a result, I can not make use of the real IP address. Only the NAT address is available.

As I have specified NAT address in the connection URL but in trace logs its giving error for real IP. Please suggest. How can I overcome this situation? What are the code changes i need to do for my requirement. Please suggest, its really urgent for me.

Thanks.
Nilesh

Posted by Nilesh on September 06, 2008 at 01:44 PM CEST #

Hi,

This sounds like exactly what I need but i'm failing to get it to work, can yo supply the working .jar so i can take out a compilation error in the jar from the equation?

The error is below if that helps?

Failed to find Premain-Class manifest attribute in ./rmi.jar
Error occurred during initialization of VM
agent library failed to init: instrument

Thanks

Tom

Posted by Tom Grange on January 16, 2009 at 04:28 PM CET #

Hi,

The jar needs to contain a manifest which specifies a Premain-Class. See the ant/NetBeans target in the post, which shows how to create a jar with a manifest.

Hope this helps,

-- daniel

Posted by daniel on January 16, 2009 at 05:11 PM CET #

Hi,

Thanks for your help with this it resolves a major issue of getting the connection through a firewall.

The only problem I have now is that I am using this to monitor Tomcat 6 which is run in a JAVA service wrapper JSVC, unfortunately javaagent is not supported by JSVC. Have you any ideas on how to implement this without using javaagent?

Thanks for your help with this.

Tom

Posted by Tom Grange on January 19, 2009 at 03:38 PM CET #

You'll get a "java.rmi.server.ExportException: internal error: ObjID already in use" if CustomAgent uses the standard system property "com.sun.management.jmxremote.port" instead of something else like "example.rmi.agent.port".

I guess that's because the out-of-the-box JMXConnectorServer also starts, and you can't have more than one in the same JVM.

Posted by J. David Beutel on February 13, 2009 at 12:52 AM CET #

Jstatd uses an RMI registry, so I assume it has the same problem with firewalls. Will this technique work on it too? Or, does it not use JMX?

Posted by J. David Beutel on March 04, 2009 at 04:59 AM CET #

I am using Sun Java Web Server and want to monitor a web application. My application and Mbean both are on the same matchine right now. I registered the MBean with the MBean Platform Server in ContextListener class.
In my server.xml, I added the following in my JVM options:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8004
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
When I start the Jconsole I can not able to viw my MBean. Please let me know if I am missing something here...

Posted by Slyth on March 25, 2009 at 08:14 AM CET #

All works well, I use the agent, I open a tunnel and I can connect with jconsole or jvisualvm.

For some reasons I need to choose the IP the customagent listen on. But I can only choose the port. The customagent always listen on 0.0.0.0:Port

How can I choose the IP ?

(Thx for this good tutorial, it helps me a lot)

Posted by F. CANTIN on March 26, 2009 at 04:31 PM CET #

I'm experiencing a similar problem. I'm trying to load balance Tomcat servers on Amazon EC2. The load balancer queries the JMX servers on the app servers to get their current load. I am not able to connect to them however.

1. I set all the java options at the time the JVM starts. This includes the hostname property, but I set it to the external IP of the EC2 instances.

2. There is a listened running on the port I had selected on the instance.

3. I cannot connect to the instances using jconsole or programmatically. I need to specify which ports should be given access in EC2. Is this the reason? That there is port to which access is required other than the one i specified and its chosen at random?

Posted by Siddharth on April 12, 2009 at 09:51 AM CEST #

I have used the above solution and it is almost working. I am using it with glassfish and I have changed asadmin to set the correct parameters. When asadmin exits (leaving glassfish running). My listener on port 3000 stops responding and my jconsole drops connection. Any ideas?

Posted by TroyB on September 15, 2009 at 08:10 PM CEST #

I have what I believe is the same question as F. CANTIN. Our security concerns dictate that we listen only on the loopback interface.

For example, after starting my agent (which uses port 3344), I can do the following:

$ netstat -an | grep 3344
tcp 0 0 0.0.0.0:3344 0.0.0.0:* LISTEN

Clearly we're listening for packets from any address. How can I specify a LISTEN only on 127.0.0.1?

Posted by Chad Showalter on September 23, 2009 at 07:15 PM CEST #

Name:
E-Mail:
URL:
	Notify me by email of new comments
	Remember Information?

JMX, SNMP, Java, etc...

Daniel Fuchs blogs on JMX, SNMP, Java, etc...

Using a Java Agent

Adding Security

Calendar

RSS Feeds

DFuchs on DZone

Search

Links

Lookup RFC

Planet JMX

From Grenoble

Navigation

Referers

Sun	Mon	Tue	Wed	Thu	Fri	Sat
« November 2009
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30

Today