JMX, SNMP, Java, etc...

Note: if you are using JConsole - you might also want to try the Java VisualVM. Java VisualVM comes with the JDK since JDK 6 update 7. It can be used to troubleshoot, monitor, and improve applications performance. It makes it possible to generate and analyse heap dumps, track down memory leaks, browse the platform's MBeans and perform operations on those MBeans, perform and monitor garbage collection, and perform lightweight memory and CPU profiling. It is also an open source project and can be downloaded as a stand alone tool from visualvm.dev.java.net.

If you have read the JConsole FAQ but are still experiencing difficulties, here are a few additional tips:

Processes not displayed in JDK 6 JConsole connection window

This may be due to weird permissions in the TMP dir. See this post for a more detailed description.

See also this post where David explains how your TMP dir settings can prevent the tutorial examples from working under windows systems - and how to solve it. On the JMX Forum, Thomas provides another hint:

My problem was a little bit different : I could see processes PID but could not 
connect to them (nor see the Main class)

The source of the problem ? : hsperfdata was named "hsperfdata_Username" where my 
login is "username".
After closing all VM I deleted the dir and re-launched java new dir was created 
with the good name "hsperfdata_username" and everything went OK.

Security

The most common troubles which prevent JConsole to connect to a remote application are linked to SSL/Security configurations. This does not apply if you connect to a local process using its PID, but will most certainly occur if you try to connect to a remote process where remote management was enabled using system properties.

When remote management is enabled using system properties, it is secure by default. This means that there are a minimum configuration steps to perform when you connect to a secure connector using JConsole. These steps are described in details here and there.

If possible, I'd advise to first start the application and JConsole without security enabled. If this works, then try again with security on.
The properties you need to pass in order to disable security are these:

-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false 

security issues are further explained here.

Firewall and RMI

If your application is behind a NAT or a Firewall, the firewall could be blocking JConsole connection. Some answers to common problems can be obtained here and there.

Update: A common problem with RMI and firewall is that the JMX default agent will not let you specify which port to use to export the server's RMI stub. To control the port on which the RMI stub is exported you will need to create your own JMXConnectorServer instead of using the connector used by the default agent.
Luis-Miguel Alventosa has just contributed two excellent blog entries which will teach you how to mimick the out-of-the-box JMX management agent and work with SSL/TLS-based RMI Socket Factories.
The first entry will teach you how to get full control over the JMX RMI Connector.
The second blog entry is not directly linked to JMX but may unveil some of the mysteries that surround the fine points of using SSL/TLS with RMI.

Update 2: The JDK 6 Monitoring and Management Guide has been updated with a section about Monitoring Applications through a Firewall. In particular it explains how to configure the two port numbers used by a JMX RMI Connector Server.

Update 3: I have also written a new entry explaining How To Connect Through Firewall Using JMX - Without modifying your server application.

Update 4: If you are trying to access a server which is behind a NAT - you will most probably have to start your server with the option

      -Djava.rmi.server.hostname=<public/NAT address>
  

so that the RMI stubs sent to the client contain the server's public address allowing it to be reached by the clients from the outside.
This was nicely summarized by DongWoo Lee in a very nice picture.

Switching on JConsole debug traces

JConsole has a -debug option that can provide useful information when trying to understand why it fails to connect. If these traces are not sufficient to make a diagnosis, you can also try to switch on the JMX and/or security traces as explained in the next section.

Switching on JMX and Security traces

Finally - here is probably the simplest way to understand what is going on: switch on the traces.

You can switch on JMX traces on the client side by calling jconsole with the following flags:

jconsole -J-Djava.util.logging.config.file=logging.properties

where logging.properties is your logging.properties file. This will activate both JMX and JMX remote traces on the client side (jconsole)

A few times ago I was blogging about how to switch on the JMX traces and gave a description of the JMX loggers and logging properties. Both JMX and JMX Remote traces can be activated in this way.

You can use the logging.properties file I have shown there. If you're trying to debug connection problems I'd recommend you switch the javax.management.remote logger to FINEST - and not to FINER as shown in the file.

If it's a security related problem (using SSL) you may also want to use:

-J-Djava.security.debug=all

or if you want to reduce the amount of security traces you can get a list of debug options by typing:

$ java -Djava.security.debug=help foo

all           turn on all debugging
access        print all checkPermission results
combiner      SubjectDomainCombiner debugging
jar           jar verification
logincontext  login context results
policy        loading and granting
provider      security provider debugging
scl           permissions SecureClassLoader assigns

The following can be used with access:

stack     include stack trace
domain    dumps all domains in context
failure   before throwing exception, dump stack
          and domain that didn't have permission

Note: Separate multiple options with a comma

See the Security Tutorial or Andrew's blog on Fine granularity diagnosis on security for more info on java.security.debug.

JConsole and BEA Weblogic Server 9.0

This article describes how to connect JConsole to Weblogic MBeanServer. Thanks Prashant for pointing this out!
The article however has one typo: the correct form of the JMXServiceURL to use on the client side is in fact:

   service:jmx:iiop:///jndi/iiop://host:port/weblogic.management.mbeanservers.runtime

Linux

If you're running on Linux, or if you're experimenting with SSH tunneling you might encounter issues with the way hostnames are resolved. See this comment and watch out for indications that RMI might use the address "127.0.1.1". This problem has also been described by Philipp Reichart (if you can read German). To solve it you might have to set the -Djava.rmi.server.hostname=<hostname or localhost or ip> property. If you're in local simply try with -Djava.rmi.server.hostname=localhost or -Djava.rmi.server.hostname=127.0.0.1. If you are running jconsole on a different host then have a look at the FAQ about JConsole on Linux.

Diagnosing what's wrong

Dustin Marx has also written a very good cookbook that might help you diagnose what's wrong by looking at the JMX Remote Exception stack trace.

Conclusion

As a conclusion, don't forget to check the FAQ which contains important information about common problems on Windows, Linux, and more common security configuration tricks.

Many thanks to all of you who are actively answering to posts in the Java and JMX forums, and contributing to the liveliness of the Java and JMX communities!