Weblog

All | Archive | General | Java
20070428 Saturday April 28, 2007

Multicasting with NIO

A long standing issue for many developers is that the java.nio.channels package has lacked support for Internet Protocol (IP) multicasting. In the NIO.2 early review draft specification you will see that we have addressed this issue by adding multicast support to DatagramChannel. Here's a small example that opens a DatagramChannel, binds the channel's socket to a local port, sets the network interface for multicast datagrams sent via the channel, and then joins a multicast group on the same interface: 

NetworkInterface interf = NetworkInterface.getByName("eth0");
InetAddress group = InetAddress.getByName("225.0.0.100");

DatagramChannel dc = DatagramChannel.open(ProtocolFamily.INET)
    .setOption(SocketOption.SO_REUSEADDR, true)
    .bind(new InetSocketAddress(5000))
    .setOption(SocketOption.IP_MULTICAST_IF, interf);

MembershipKey key = dc.join(group, interf);

Once the channel has joined the group then it reads or receives multicast datagrams in the same manner that it reads or receives unicast datagrams.

The most significant thing in this example is the static factory method to open the channel specifies a protocol family. In this example, the channel is to an IPv4 socket. In the existing APIs, the protocol family is transparent and all sockets created by the java.net or java.nio.channels packages are either all IPv4 or all IPv6. For IP multicasting it is important that the protocol family corresponds to the address type of the multicast groups that the socket joins; otherwise it is highly operating system specific if the socket can join the group, configure options, or receive multicast datagrams. Legacy java.net.MulticastSocket has suffered greatly from problems in this area. Another interesting thing to point out is that the channel's socket is bound and socket options are configured directly. The awkward and counterintuitive socket adaptor isn't required so it isn't necessary to mix java.net socket APIs when configuring the channel's socket.

In the example, the MembershipKey that is returned by the join method is a token to represent membership of the group. It defines methods to query information about the membership and defines the "drop" method to drop membership of the group.

Developers tracking multicast standards will know that the RFCs in this area have been updated in recent years to add source filtering and this is now supported by almost all modern operating systems. In the NIO.2 draft specification we have included basic support for source filtering. The following code fragment shows a channel joining a multicast group to only receive multicast datagrams sent by a specific IP source address (otherwise known as "include-mode" filtering): 

MembershipKey key = dc.join(group, interf, source);

"Exclusive-mode" filtering is where a group is joined to receive all multicast datagrams except those from specific IP source address: 

MembershipKey key = dc.join(group, interf).block(source1).block(source2);

So that's a brief introduction to the multicast support that we propose to add to the java.nio.channels package. There's a lot more detail in the draft specification for those interested in this topic. ( Apr 28 2007, 01:46:02 PM PDT ) Permalink Comments [6]

20070427 Friday April 27, 2007

Accessing file permissions

One of the many issues addressed by NIO.2 is that the Java SE platform doesn't have a complete set of APIs to access file permissions and other security related file attributes. It's not rare to encounter code that has to resort to using Runtime.exec to execute a command such as chmod(1).

In the NIO.2 early review draft specification you will see that we address the issue of file attributes by organizing related attributes into groups. A FileAttributeView is then defined to provide a read-only or updatable view of the attributes in the group. The specification defines a number of attribute views, of which only BasicFileAttributeView is required to be supported by a file system implementation (BasicFileAttributeView provides access to a small set of attributes such as file size and last modified time - essentially the basic attributes that are common to many file systems).

Aside from BasicFileAttributeView, a file system implementation can support other file attribute views. The specification defines PosixFileAttributeView for access to file attribtues commonly found on platforms that implement the POSIX family of standards. Here's a small example that uses an instance of this attribute view to print the owners and permissions of a file called "foo". It then changes the permissions to deny others access to the file: 

PosixFileAttributeView view = PathReference.from("foo")
    .newFileAttributeView(PosixFileAttributeView.class);

// bulk read
PosixFileAttributes attrs = view.readAttributes();

int perms = attrs.getPermissions();

// prints "rw-r--r-- alice bandits"
System.out.format("%s\t%s\t%s%n", 
    PosixFilePermission.toString(perms),
    attrs.getOwner(),
    attrs.getGroup());    

// deny others
perms &= ~OTHERS_READ & ~OTHERS_WRITE & ~OTHERS_EXECUTE;
view.updatePermissions(perms);

The only code that might need explanation here is the code that obtains an instance of PosixFileAttributeView. FileAttributeViews are selected by type-token and this code selects an instance of PosixFileAttributeView that is bound to the file "foo".

The comment hints that foo is owned by the "bandits" group and we can fix this by changing the group: 

UserPrincipal cops = view.lookupPrincipalByGroupName("cops");
view.updateOwners(null, cops);

The other part to file permissions is that sometimes you need to create a file with initial permissions (umask applies of course). Here's an example that creates a new file called "securefile" with initial permissions, opening the file for random access: 

PathReference file = PathReference.from("securefile");
PosixFileAttributeView view = ...

Attribute<Integer> attr = view.newPermissionsAttribute()
    .setValue(OWNER_READ | OWNER_WRITE | GROUP_READ | GROUP_WRITE);

SeekableByteChannel sbc = file.newSeekableByteChannel(CREATE_NEW | READWRITE, attr);

So that's a brief tour of PosixFileAttributeView. The other file attribute view in the specification that provides access to security related attributes is AclFileAttributView. This provides access to Access Control Lists (ACLs) based on the NFSv4 ACL model. That will be interesting to those on file systems that support ACLs. When I get time I'll write up a few words on how this attribute view is used. ( Apr 27 2007, 07:52:04 AM PDT ) Permalink Comments [2]

20070412 Thursday April 12, 2007

NIO.2: The Return From The Lost Planet

We (the JSR-203 expert group) recently submitted the Early Draft Review (EDR) specification for JSR-203. Otherwise known as NIO.2, this JSR is tasked with developing the second phase of the New I/O APIs for the Java SE platform. Specifically, it is tasked with developing a new file system interface, asynchronous I/O APIs, and to complete the socket-channel API carried over from the first NIO JSR (JSR-51). The plan is to put JSR-203 forward as a candidate component JSR to include in Java SE 7.

The draft specification that we have submitted can be downloaded from: http://jcp.org/aboutJava/communityprocess/edr/jsr203/index.html. We've setup a mailing list to discuss this JSR and anyone can subscribe by going to: http://groups.google.com/group/jsr203-interest/subscribe. Comments can also be sent to: jsr-203-comments@jcp.org. The most important goal with releasing this draft is to get feedback that we are addressing the right problems. In the file system area at least, this JSR is about solving some basic problems and adding support for functionality that is sadly missing from the platform today. A second goal is to poke holes in the API. As I'm sure all developers agree, API design benefits greatly from feedback and it can often take several iterations to get an API right.

For those interested in the topics in this JSR I have uploaded a few slides to get you started. In coming weeks I'll follow up on specific topics covered in this JSR. ( Apr 12 2007, 04:06:26 PM PDT ) Permalink Comments [3]

20060526 Friday May 26, 2006

HeapDumpOnOutOfMemoryError option in 5.0u7 and 1.4.2_12.

5.0 Update 7 was released this week. Among the changes is the backport of the HeapDumpOnOutOfMemoryError option from Mustang. This VM option tells the HotSpot VM to generate a heap dump when OutOfMemoryError is thrown because the java heap or the permanent generation is full. A heap dump is useful in production systems where you need to diagnose an unexpected failure. Lots of developers sent mail asking to make this option available in the shipping releases. For 1.4.2 it will be available shortly in 1.4.2_12.

So how do you analyze these dumps? The legacy Heap Analysis Tool (HAT) is one option but it's old, provides only limited queries, and it can't import heap dumps generated by 64-bit VMs. A better choice is jhat which fixes many issues with HAT, provides new queries, and most important, it provides a query interface called Object Query Language to write your own queries. Sundar has some great blog entries to introduce OQL and get you started:

The jhat utility is included in Mustang (which you can download from the binaries snapshot site). It will happily munch on dumps produced by 5.0u7 and 1.4.2_12.

Another tool to analyze the heap dump is the YourKit Java Profiler. Anton Katilin and his colleagues in YourKit recently released version 5.5 of the product and this includes an "Import HPROF Snapshot" option to import the heap dump and convert it to the format used by the profiler.

Two other useful things to know are: (i) the heap dumps are platform independent and so you don't need to analyze the dump on the same system that produced it, and (ii) running with -XX:+HeapDumpOnOutOfMemoryError does not impact performance - it is simply a flag to indicate that a heap dump should be generated when the first thread throws OutOfMemoryError.

[Update 08/04/06: In the original entry I mentioned a potential issue with the low-pause collector. That turns out not to be a concern.] ( May 26 2006, 04:26:23 AM PDT ) Permalink Comments [18]

20060104 Wednesday January 04, 2006

Another piece of the tool puzzle. One of the updates in Mustang b65 was the addition of the getAgentProperties method to the Attach API. This method gives tools access to a set of properties maintained in the Java virtual machine on behalf of agents. This may sound obscure but it gets interesting when you know that the JMX agent will create an agent property when it starts a local JMX connector server. A local JMX connector server is started when you start an application with -Dcom.sun.management.jmxremote or you start the management agent in a running application using the Attach API.

MemViewer.java is a simple "one page" example to demonstrate how a tool might use this method. MemViewer takes one argument to identify the target application and for this you use the process-id (or pid). MemViewer attempts to attach to the specified application and looks for the agent property named "com.sun.management.jmxremote.localConnectorAddress". If this property is set then it is the address of the local JMX connector server. If the property is not set then it means a local JMX agent is not running so MemViewer starts the management agent by loading a java agent named "management-agent.jar" into the target application. Once started, it re-obtains the agent properties to get the value of the agent property. Once MemViewer has the address of the connector server then it connects and prints information about each of the memory pools. Here is an example where MemViewer is used to print information about the memory used by a Java Web Start application. 

C:\> jps -l
2304 sun.tools.jps.Jps
2560 com.sun.javaws.Main

C:\> java -classpath .;%JDK_HOME%\lib\tools.jar MemViewer 2560
                Pool         Used    Committed          Max
Perm Gen [shared-ro]      5279360      8388608      8388608
          Code Cache      4901696      4980736     33554432
          Eden Space       313968      1638400      4194304
Perm Gen [shared-rw]      7021848     12582912     12582912
         Tenured Gen     14466928     23662592     61997056
            Perm Gen     10631352     12582912     67108864
      Survivor Space       196608       196608       458752

If you examine MemViewer then you will see that the code is relatively simple. VirtualMachine.attach is used to attach to the target Java virtual machine. Once attached the getAgentProperties method is used to read the agent properties. If the management agent is not running then it constructs the name of the management agent and then uses the loadAgent method to load it into the target. The Attach API is a simple tool API for bootstraping agents into a running application and ships in the JDK in tools.jar. This means that tools.jar must be on the classpath when compiling or running. The JAR file "management-agent.jar" is the java agent with the Agent-Class attribute in the main manifest to specify the class name of the agent. The agent property com.sun.management.jmxremote.localConnectorAddress is set by the agent when the local JMX connector has started. This property will be documented along with the other properties on the Monitoring and Management with JMX page.

Finally, in the example output you will see that I used jps utility to get a list of the Java virtual machines that I was running. An alternative approach would have been to use the VirtualMachine.list method and prompt the user to select from a list. ( Jan 04 2006, 08:14:04 AM PST ) Permalink Comments [3]

20051213 Tuesday December 13, 2005

Late binding agents and fun times for the tool maker! A few months ago, I highlighted an update to jconsole that allows it connect to applications that weren't started with special command line options. This ability to connect to a running application isn't unique to jconsole. When the NetBeans Profiler first appeared (as JFluid) it included a similar feature that allowed it to attach, apply instrumentation, and profile an application without needing to restart it with the profiler agent. At the time JFluid ran on a customized release of J2SE 1.4.x. The underlying implementation was very different, but the general idea is the same. There are many improvements in Mustang that mean that this ability to attach, start monitoring agents or instrumentation-based agents, is available to anyone developing tools. Today, I thought I might write up a few words on this topic.

First, some background. When I use the term "tool agent" I'm talking about the piece of the tool that runs in-process in the same VM as the application. Typically, tools are architected so that there is a "front-end" that the user interacts with, and a "back-end" that runs in the VM with the application. We see this general architecture with many of the profiling tools today. You'll also see essentially the same thing with the debugger architecture, or with JMX tools that connect to the JMX agent running in the target application. Agents doing exact and sample based profiling, space profiling, coverage analysis and so on are often developed as native agents (in C/C++) and make use of the JVM Tool Interface. Agents doing fine-grain monitoring, execution time profiling, journaling, and other tasks that make use of bytecode instrumentation are developed in the Java Language and make use of the instrumentation support provided by java.lang.instrument.

So how does a tool load its agent into a running application? In Mustang we have included the Attach API. This is a very simple API to bootstrap agents into a running application. The API is a Sun-specific API and is included in the JDK for use by tools (it can of course attach to applications that are deployed with the JRE). The VirtualMachine class is used to attach to an application. Once attached, the loadAgent or loadAgentLibrary methods can be used to load and start an agent. There is also a list method which can be used to enumerate the user's Java virtual machines (useful if you want to prompt a user to select the application to be monitored or profiled).

So what can an agent do once it is loaded in a running application? In the case of native agents using the JVM Tool Interface then it depends on the capabilities that the VM is able to provide in the live phase. Many of the capabilities needed by a fully featured debugger for example, are only available in the onload phase because they require special initialization or code generation that is only done at startup. Profilers starting in a running application will typically require the capabilities that will allow it instrument classes that are already loaded, instrument new classes as they are loaded, and perhaps tag objects and classes. In Mustang, these capabilities are always available in the HotSpot Server VM. Most are available in the HotSpot Client VM too except that the ability to instrument classes that are loaded from the shared archive (a short term implementation detail that will get resovled in time). For Java Lanaguage agents doing instrumentation then the story is the same - they can add ClassFileTransformers to instrument classes that are subsequently loaded, or retransform and instrument already loaded classes. Another aspect to instrumentation is that agents will often need to instrument classes so that they invoke methods on supporting classes provided by the tool. When an agent is loaded into a running application these supporting classes may not be available but Mustang has methods to make the supporting classes available for the bootstrap class loader or the system class loader. Many Java Language agents (like the JMX agent) won't be concerned with byetcode instrumentation - instead they will (for example) make use of java.lang.management package so that the tool can obtain telemetry information and manage the VM.

So now that we know what the agent can do you might wonder about security. This is very important and we've taken the position that there shouldn't be any suprises. That means that a tool shouldn't be able to interact with an application that won't normally be allowed by the security on the operation system. For UNIX environments this means that the tool must have the same effective uid/gid as the target application. On Windows, the implementation can only be used to attach to applications that you have privilege to open and interact with. So if Alice is running a memory profiler then she will only be able to attach to her own applications. She won't be able to attach her profiler to Bob's application that is churning out the winning numbers for next week's lottery.

So that's a brief overview of the late-binding agent support in Mustang. As you can see, all the major pieces to allow tools attach, observe, and profile a running application are in place. Fun times for the tool maker. ( Dec 13 2005, 02:50:31 AM PST ) Permalink Comments [1]

20051128 Monday November 28, 2005

OutOfMemoryError looks a bit better! OutOfMemoryError has always been a confusing error. For a long time the HotSpot Virtual Machine threw this error without a detail message or stack trace so typically the thread throwing the error would terminate with this: 

  Exception in thread "main" java.lang.OutOfMemoryError

That was confusing. Is my java heap full or does it mean something else? Those familiar with the heap layout that the HotSpot VM uses will know that the "something else" might mean the "permanent generation". This is place where reflective data such as class and method objects are allocated. It is also the place where interned strings are stored. If you've got an application that loads a huge number of classes or interns millions of strings then it's possible that the OutOfMemoryError is because the permanent generation is full rather than the java heap.

In 5.0 the error is less confusing as there is a detail message. This means you will see something like this: 

  Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
or: 
  Exception in thread "main" java.lang.OutOfMemoryError: PermGen full

In Mustang, there has been further changes to the way that OutOfMemoryError is reported. One obvious one is that the HotSpot VM will attempt to include a stack trace. This means you should see something like this: 

  Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at ConsumeHeap$BigObject.(ConsumeHeap.java:22)
        at ConsumeHeap.main(ConsumeHeap.java:47)

In this example we see the stack trace where the allocation failed. If the OutOfMemoryError is because the perm gen it full then you might see String.intern or ClassLoader.defineClass near the top of the stack.

Is a stack trace useful? In some limited cases it can be. For example, suppose you've got a thread looping and in the loop it is allocating objects and putting them into a collection. In that case the stack trace might direct you to a good starting place. More generally, the stack trace is not likely to be useful. If you've got a busy application and the heap is nearly full, then OutOfMemoryError will be likely be thrown in some random place by some random mutator.

One useful improvement is the -XX:+HeapDumpOnOutOfMemoryError option which tells the HotSpot VM to generate a heap dump when an allocation from the java heap or the permanent generation cannot be satisfied. There isn't any overhead to running with this option so it should be useful for production systems where OutOfMemoryError takes weeks (or longer) to surface. The heap dump is in HPROF binary format so it can be analyzed using any tools that can import this format. If you read Sundar's blog then you'll know that Mustang includes a useful tool called jhat which can be used to do rudimentary analysis of the dump. jhat supports Object Query Language so you can easily create your own queries and mine the heap dump.

The next improvement is visible when the system is almost out of swap space and an allocation from the native heap fails in the VM. In that case the VM aborts and you get a one-line error such as the following: 

  Exception java.lang.OutOfMemoryError: requested 16 bytes for CHeapObj-new. Out of swap space?

This message has been known to confuse a lot of developers. At first glance it might it might look like that the java heap is full. Of course if you increase the size of the heap with the -mx option it might make the problem worse as the larger java heap means there is less native memory available.

In Mustang this condition will trigger the VM to invoke the fatal error handling mechanism. This means you should get a fatal error log as you would get with a normal (abnormal?) crash. The fatal error log is named hs_err_<pid>.log and contains useful information about the thread, process, and system at the time of the crash. In the case of native heap exhaustion then the heap memory and memory map information in the log can be useful. The exact format is version and platform specific and you will get more information in the J2SE Troubleshooting Guide.

Hopefully, developers will find these improvements useful. It should make OutOfMemoryError a little less confusing and it means the error is no longer a one-line wonder. ( Nov 28 2005, 06:05:37 AM PST ) Permalink Comments [16]

20051121 Monday November 21, 2005

Two fine demos

I was in New York last week, and attended a Java SE User Group meeting. After the scheduled talks there was open discussion on a wide range of topics. One of those topics related to tutorials, documentation, and sample code. It is always interesting to hear how people learn new APIs. Some people start with the API documentation, some google for sample code, and books continue to be popular. The JDK "demo" directory was mentioned and a number of people remarked that they hadn't checked it out in years because the demo directory appeared to be stale. There are indeed some old demos in there but there are a number of more recent examples too. Today I thought I might highlight two of the more useful examples.

The first is JTop. You'll find it in %JDK_HOME%\demo\management\JTop (or $JDK_HOME/demo/management/JTop) where JDK_HOME is C:\jdk1.6 or wherever you have installed the JDK. Mandy Chung added this demo in Mustang b59 and it will work on JDK5.0 too. There's a README.txt file in the directory which explains what it does. In brief, it monitors the CPU usage of the threads in a remote application. If you've ever had to deal with a looping process and you tried to figure out which thread is looping then you will relate to this.

In API terms, this example is a demonstration of the monitoring and management package. If the Java virtual machine is able to measure thread CPU time then getThreadCpuTime method can be used to obtain the CPU time for a thread.

To use JTop, we start the application with the JMX agent configured for remote management. For demonstration purposes both authentication and SSL are disabled (which you won't want to do in production). 

   java -Dcom.sun.management.jmxremote.port=7000
        -Dcom.sun.management.jmxremote.authenticate=false
        -Dcom.sun.management.jmxremote.ssl=false 
        -jar Application.jar

JTop is connected by passing it the hostname and port number so that it can connect to the JMX agent: 

   java -jar %JDK_HOME%\demo\management\JTop\JTop.jar localhost:7000

JTop uses a TimerTask to refresh the thread list every 2 seconds. It gets the CPU time for each thread and then sorts the list by CPU time. You should see something like this:

It is primitive but it serves as a useful demonstration of the APIs. The source is in the src directory and with a little bit of effort it could be turned into a nice tool. For example, it currently just sorts the threads by total CPU usage and it might be nicer to highlight the threads that were busy in the proceeding interval. Also, it might be interesting to sample the stack traces of the busy threads to get an idea of the code that is executing.

The second example is heapViewer. You'll find it in %JDK_HOME%\demo\jvmti\heapViewer (or $JDK_HOME/demo/jvmti/heapViewer). This example has been included since JDK5.0. It's a native agent that uses the JVM Tool Interface to do some basic memory analysis. It uses IterateOverHeap to do a linear iteration over all objects in the heap. It then prints a summary in the form of a class-wise histogram to show which objects are taking up space in the heap.

The agent is an in-process agent that is started using the -agentlib or -agentpath command-line options. Here's one example: 

java -agentpath:%JDK_HOME%\demo\jvmti\heapViewer\lib\heapViewer.dll
     -jar Application.jar

When the VM exits or the agent gets a DataDumpRequest event then it will print a histogram to summarize the objects in the heap. A DataDumpRequest is triggered by a Ctrl-Break on Windows, or Ctrl-\ (or SIGQUIT) on Solaris/Linux. Here is some sample output: 

Heap View, Total of 123199 objects found.

Space      Count      Class Signature
---------- ---------- ----------------------
  19597872       7421 [I
   1420264      13037 [C
   1217872       2681 [B
   1194256       2381 Ljava/lang/Class;
    985624       3619 [S
    266688      11112 Ljava/lang/String;
    173576       4682 [Ljava/lang/Object;
    164800        824 Lsun/java2d/SunGraphics2D;
    123888       5162 Ljava/util/Hashtable$Entry;
    106560       4440 Ljava/awt/Rectangle;
     79552       2486 Ljavax/swing/text/html/parser/ContentModel;
     70816        818 [Ljava/util/Hashtable$Entry;
     67440        843 Ljavax/swing/text/html/InlineView;
     63504       2646 Ljavax/swing/SizeRequirements;
     57984        906 Ljava/awt/geom/AffineTransform;
     56600       1415 Ljava/util/WeakHashMap$Entry;
     55560       2315 Ljava/util/HashMap$Entry;
     52864        112 Ljavax/swing/plaf/metal/MetalScrollButton;
     46880       2930 Ljavax/swing/event/EventListenerList;
     45056        352 Ljavax/swing/text/html/LineView;
     44808       1867 Ljava/awt/Insets;
---------- ---------- ----------------------

In this example, there were 123199 objects in the heap, and integer arrays are taking up the most space. There are of course other ways to generate a heap histogram like this but this one is useful as it makes use of the standard APIs. The source for this example is in the src directory.

I hope you agree these are useful examples. There are a number of others in the demo/management and demo/jvmti directories which are worth checking out too. ( Nov 21 2005, 05:00:45 AM PST ) Permalink

20051103 Thursday November 03, 2005

To poll or epoll: that is the question: One of the updates in build 59 of Mustang (JavaTM SE 6) is that the New I/O Selector implementation will use the epoll event notification facility when running on the Linux 2.6 kernel. The epoll event mechanism is much more scalable than the traditional poll when there are thousands of file descriptors in the interest set. The work done by poll depends on the size of the interest set whereas with epoll (like Solaris /dev/poll) the registration of interest is separated from the retrieval of the events. A lot has been written on the topic. The C10K problem has been documenting I/O frameworks and strategies for several years. One relatively recent paper on Comparing and Evaluating epoll, select, and poll Event Mechanisms makes it clear the workloads where epoll performs a lot better than poll.

This isn't the first NIO Selector implementation to use epoll. The Blackdown folks added epoll support in their 1.4.x release. On Solaris, the /dev/poll based Selector has been default on Solaris 8 (and newer) since the original implementation of New I/O in J2SETM 1.4.

So if you are running on a Linux 2.6 system with an application that handles lots of simultaneous connections you might want to give b59 a test-run. The weekly builds have been appearing like clockwork on the binary snapshot release site so b59 should be available tomorrow (November 4). Will you see a difference? It depends on the workload. If you've registered lots of SelectableChannels with a Selector and you notice a lot of time spent in the kernel due to poll then you should see a difference. If you are doing test runs and you want to do a direct comparison with poll then you can set the java.nio.channels.spi.SelectorProvider system property to sun.nio.ch.PollSelectorProvider. This will select the poll-based Selector that will continue to be the default on 2.4 kernels. There is an epoll patch for 2.4 kernels but at this time anyway, the NIO implementation doesn't attempt to detect this.

[Update 08/04/2006: The epoll SelectorProvider will be included in 5.0 update 9. To enable it requires setting the system property java.nio.channels.spi.SelectorProvider to the value sun.nio.ch.EPollSelectorProvider. Sorry, it won't be the default in 5.0 but we must be very careful when changing anything in an update release.] ( Nov 03 2005, 04:19:35 AM PST ) Permalink Comments [10]

20051021 Friday October 21, 2005

java.io.Console is finally here! One of the most popular feature requests for J2SETM in recent times has been the request to improve console support and provide a way to enter passwords with echo disabled. Developers know this feature 4050435 as it has been skulking in the Top 25 RFEs list for some time.

The good news is that the feature has made it into Mustang thanks to Xueming Shen. It went into b57 and should show up be on the download site later today. The feature adds java.io.Console which provides methods to read lines and passwords from the console. It also provides useful methods to write formatted strings to the console too. Here's a little taster that prompts user to enter a password that is at least 8 characters in length. The password is not echoed to the console as it is entered. 


static final int MIN_PASSWORD_LENGTH = 8;

char[] password;
do {
    password = System.console().readPassword(
      "Enter password (minimum of %d characters): ", MIN_PASSWORD_LENGTH);
} while (password.length < MIN_PASSWORD_LENGTH);

System.console() is used to obtain the unique Console for the Java virtual machine. There may not be a console of course - it depends on the platform, and also on how the Java virtual machine was started. If there isn't a console then the console() method returns null (the above code fragment doesn't check for this).

The readPassword method writes the prompt and reads the password. The prompt is provided as a format string and an argument list. If you've used the formatted printing support that was added in J2SE 5.0 then you'll recognize this.

Another thing about this code fragment is that it leaves you with a password in a character array. As with anything sensitive you don't want to have this in memory for a long time so it's good to zero the array as soon as you can.

So if you develop applications that need to access a character based console then you should find java.io.Console very useful (and very simple to use). ( Oct 21 2005, 10:17:28 AM PDT ) Permalink Comments [39]

20051014 Friday October 14, 2005

jstack One of the useful troubleshooting utilities in J2SETM 5.0 is jstack. It prints the stack traces of all java threads for a given process or core dump. The -m option will print a mixed-mode stack so that you can see native frames in addition to the java frames. jstack was originally created for troubleshooting problems involving hangs and crashes but developers have found it a useful utility to look at running applications too.

jstack isn't the only way to obtain a thread dump of a running application. Developers have long been accustomed to using Ctrl-\ (or Ctrl-Break if you are on Windows) to get a thread dump of an application that is running interactively. An alternative to the key sequence on Solaris (or Linux) is to send a QUIT signal to the target process. In that case the target processs prints a thread dump to its standard output (which can be pain if you don't know where the log file is).

If you are lucky to be on Solaris 10, then yet another way to obtain a stack trace is using the the pstack utility. This has been updated in Solaris 10 so that it prints java method names for interpreted, compiled and inlined java methods. pstack prints a mixed-mode stack and the output is similar to jstack -m.

Mustang (Java SE 6.0) brings more improvements. For starters the default mode for jstack is to work a bit like a remote Ctrl-\. This means that the output of the thread dump has changed a bit but there is more information on deadlocks and JNI Global References in additional to the thread stacks. If the VM is hung then a thread dump can still be forced using the -F option.

The second improvement is that jstack is included on Windows. It doesn't have all the features yet of the jstack utility on Solaris but it does solve a long standing requirement from developers to get a thread dump of an applications that run in the background as a Windows Service.

The third improvement is the "-l" option to print information about java.util.concurrent.locks. More specifically, it instructs the utility to look for ownable synchronizers in the heap. This should be useful for applications that make use of the concurrency APIs added in J2SE 5.0. Without the "-l" option the thread dump only includes information on monitors. One word of warning though, this option can be expensive if the heap is large. ( Oct 14 2005, 10:00:56 AM PDT ) Permalink Comments [13]

20051003 Monday October 03, 2005

One password to rule them all! A few weeks ago I talked about how easy it is to get started with jconsole. The focus of that discussion was local monitoring and management where jconsole connects to applications running as the same user on the same machine. It is a simple scenario with simple configuration and jconsole doesn't need to prompt for username and password.

Now let us move onto the more complicated (and arguably more realistic) scenario where an application is deployed over a number of machines. In these cases the configuration of the JMX agent gets more complicated as it involves setting up SSL, a password file, and choosing a port number for the agent. This is all described on the Monitoring and Management with JMX page. One bugaboo with all this is that the password and access files create yet another place for user administration and yet another password to remember. If the application environment is distributed across the globe then it means sharing files in a secure manner or maybe a password file per machine. SSL client certificates provide a better alternative but in many environments they are a missing piece in the overall security infrastructure. Assuming you aren't using SSL client certs then another alternative to password authentication is LDAP. In many companies there is already directory server infrastructure and many of your applications may be using LDAP already. So today I want to talk about configuring the JMX agent to use LDAP instead of the file-based authentication.

Let's start by talking about the JMXAuthenticator implementation in the JMX agent. It is based on Java Authentication and Authorization Service (JAAS). Authentication is performed by passing the user credentials to a JAAS LoginModule. By default it uses a LoginModule that authenticates using the password and access files. An alternative LoginModule (and hence an alternative authentication mechanism), is specified using the com.sun.management.jmxremote.login.config property. The property is set to the name of a JAAS login configuration entry. For LDAP that means we need to set it to the entry name for an LDAP-based LoginModule.

J2SETM 5.0 didn't ship with an LDAP based LoginModule but there is one in a Mustang (Java SE 6.0) thanks to Vincent Ryan. The LoginModule is com.sun.security.auth.module.LdapLoginModule. To use it we need to create a JAAS configuration entry. I've created the configuration file ldap.config and configured the entry to work within Sun. Here's how it looks: 

SunConfig {
    com.sun.security.auth.module.LdapLoginModule REQUIRED
        userProvider="ldap://sun-ds/ou=people,dc=sun,dc=com"
        userFilter="(&(uid={USERNAME})(objectClass=inetOrgPerson))"
        authzIdentity=monitorRole
        useSSL=false;
    };

If you aren't familar with the syntax of the configuration file then the Configuration class details the information needed for configuration, and there are examples available.

The specification for the LoginModule details all the options. In this example I've named the entry "SunConfig". The flag "REQUIRED" means that authentication is required to succeed. The other items are module options. The userProvider option identifies the LDAP directory. It's a LDAP URL (or a list of URLs). The URL identifies the LDAP server to use and the position in the directory tree where user entries are located. The userFilter module option is another piece of LDAP configuration. It specifies the search filter to use to locate a user entry in the LDAP directory. The token "{USERNAME}" is replaced with the username before the filter is used to search the directory. If all this looks like gobbledegook then have a chat with your LDAP administrator to get the magic settings for your environment.

The authzIdentity takes a bit of explaining and we'll come back to it shortly. The final module option in this confguration is useSSL which indictaes if the connection to the LDAP server uses SSL or not. The default is true but I've tried to keep this example simple and so it is disabled.

So now we have a JAAS configuration entry for the LDAP LoginModule. The next piece is to configure two system properties:

Here's a sample command line with all the properties set: 

java -Dcom.sun.management.jmxremote.port=5000
     -Dcom.sun.management.jmxremote.ssl=false
     -Dcom.sun.management.jmxremote.login.config=SunConfig
     -Djava.security.auth.login.config=ldap.config
     -jar MyApplication.jar

I've snuck in two other properties here. The com.sun.management.jmxremote.port property configures the TCP port number, and com.sun.management.jmxremote.ssl specifies if SSL is enabled or disabled (I've chosen to disable it to keep the configuration simple but you'll want to enable it in production environments).

At this point we have our configuration so we start the application. The JMX agent starts with the application so we now try to connect using jconsole. For example I'll assume the application is on "myserver" and we're "John Doe" (username "jdoe"). To connect we start jconsole, switch to the Remote tab of the Connection Dialog, and enter the connection details:

When we press the Connect button then jconsole will connect to the JMX agent, and the JMX agent will attempt to authenticate "jdoe".

When you initially setup your configuration it can be useful to add the debug opton to the JAAS configuration entry. If you add debug=true then the LdapLoginModule will print trace information to standard output. Here's an example for when jdoe does a successful login: 

[LdapLoginModule] search-first mode; SSL disabled
[LdapLoginModule] user provider: ldap://sun-ds/ou=people,dc=sun,dc=com
[LdapLoginModule] searching for entry belonging to user: jdoe
[LdapLoginModule] found entry: cn=John Doe,ou=people,dc=sun,dc=com
[LdapLoginModule] attempting to authenticate user: jdoe
[LdapLoginModule] authentication succeeded
[LdapLoginModule] added X500Principal "CN=John Doe,OU=people, DC=sun, DC=com" to Subject
[LdapLoginModule] added UserPrincipal "jdoe" to Subject
[LdapLoginModule] added UserPrincipal "monitorRole" to Subject

This log is saying that the LoginModule searched for the user entry "jdoe". It was found and then the password was authenticated.

If the password was entered incorrectly then you'll see something like this: 

[LdapLoginModule] search-first mode; SSL disabled
[LdapLoginModule] user provider: ldap://sun-ds/ou=people,dc=sun,dc=com
[LdapLoginModule] searching for entry belonging to user: jdoe
[LdapLoginModule] found entry: cn=John Doe,ou=people,dc=sun,dc=com
[LdapLoginModule] attempting to authenticate user: jdoe
[LdapLoginModule] authentication failed
[LdapLoginModule] aborted authentication

Now let's get back to the authzIdentity module option that I skipped over earilier. In the example I used authzIdentity=monitorRole which means that all authenticated users get read-only access to the managed VM. In technical terms the Principle named monitorRole is added to each authenticated user (or Subject). The name monitorRole corresponds to an entry in the JMX access file. The JMX access file is usually ${JRE_HOME}/lib/management/jmxremote.access but you can use the com.sun.management.jmxremote.access.file property to specify an alternative file if you wish. By default the file contains two roles, namely monitorRole with readonly access, and controlRole with readwrite access. In some environments it might make sense to allow all users access to the managed VM but more realistically there will only be a small number of technical staff that will be monitoring and managing the applications. In that case we need to update the configuration as follows:

So, assuming we want our "John Doe" to have read-write access then we create an entry in the JMX access file as follows: 

jdoe    readwrite

So what does this mean? When John Doe connects he will be authenticated using his normal LDAP password. Once authenticated he has readwrite access to the managed application (by virtue of the entry in the access file).

In summary, I think you will agree that the ability for the JMX agent to authenticate users using LDAP is very nice. It eliminates the need to manage additional password files and for those managing applications it means there is only one password to remember. There is some configuration required and specific users still need to be setup in the access file but overall it is a whole lot better than having to mantain password files. ( Oct 03 2005, 12:07:43 PM PDT ) Permalink Comments [9]

20050930 Friday September 30, 2005

Time to update the J2SE Troubleshooting Guide! One of the documents created for J2SETM 5.0 was the Troubleshooting and Diagnostic Guide. It was a late decision to create this document with the result that its scope was much narrower than might be expected. However it served a useful purpose in that it created an awareness of the troubleshooting tools that were new to J2SE 5.0. It also served as a useful place to send people when troubleshooting questions came up (like how to find the error log that is generated when a fatal error occurs).

It's now time to start thinking about updating the document for Mustang. The initial document focused on the HotSpotTM virtual machine. The document for Mustang needs to be wider in scope and cover troubleshooting problems with client-side applications, deployment, and many other areas. Mustang brings a lot of new and updated diagnostic features. I touched on the heap dump capabilities here, and Sundar has started a series on the Object Query Language feature in jhat. There are many other useful items and these need to be written up.

If you have ideas for areas that should be covered in the updated Troubleshooting Guide please send a mail to j2se-trouble-shooting-guide@sun.com. Also, send feedback if you think that one big document isn't the right format for this - the original reason for putting everything into one document was to allow people to print it out. I don't know many trees suffered needlessly so send feedback if you think an alternative format would be better. Finally, if you would like to contribute content then please step forward. In particular if you recently diagnosed a really awkward problem with a deployed application then others might benefit from a write-up of the steps that you took to diagnose it. ( Sep 30 2005, 04:03:07 AM PDT ) Permalink Comments [1]

20050923 Friday September 23, 2005

Getting started with jconsole just got easier! J2SETM 5.0 brought some great monitoring and management capabilites to the JavaTM platform. The built-in instrumentation in the Java Virtual Machine means you can monitor and manage it using JMX. And of course, if the application have been instrumented with JMX then it gets even better. As part of the monitoring and management implementation a JMX-compliant monitoring tool called jconsole was developed. I hope you have tried it. If you haven't tried it let then a good starting place is the this page on Monitoring and Management Using JMX

One of the goals during the development of jconsole was that it should be easy to get going very quickly. The JMX agent has lots of configuration properties but a new user should be able to avoid most of this and get going in a few simple steps. All you need is to start the application with a simple property on the command line. When this property is set then the JMX agent starts up in a way that allows jconsole to connect without needing to prompt for connection details. This is somethings called local monitoring because it restricts the monitoring to the local machine (you have to be the same user too). The property is com.sun.management.jmxremote so you start your application like this:

java -Dcom.sun.management.jmxremote App

Once the application is running then jconsole can connect. If you know the process-id (pid) then "jconsole <process-id>" should do it. Alternatively, if you don't know the process id then start jconsole without any parameters and select the application in the connection dialog.

It turns out that it gets even simpler in Mustang (Java SE 6.0). The reason is that jconsole has been updated so that it can connect to applications that did not start up with the JMX agent. This is rather useful as it allows jconsole to connect to applets in the browser, Java Web Start applications, and other applications where it isn't easy to fiddle with the command line.

So how does it work? jconsole uses a JMXConnector client to connect to the JMXConnectorServer in the target application. In the application isn't started with the JMX agent then there isn't a JMXConnectorServer and jconsole doesn't have anything to connect too. In that case it uses a HotSpotTM VM specific mechanism to start the JMX agent in the target VM. Once the agent is started then jconsole connects as normal.

This sounds very cool so let us see how it looks. First, we start jconsole. I haven't use any parameters so jconsole opens the Connection Dialog. I've selected the Local Tab and jconsole shows me the applications that I have running on this machine. In this screen-shot I have clicked ona Java Web Start application called bugster:

Next I press the Connect button. Behind the scenes the JMX agent is started in the target VM and jconsole connects. (By the way, that checkbox column with the title Enabled just indictaes if the JMX agent is running in the target VM or not - this is probably not the best way to indicate this but jconsole is due a new connection dialog soon and it should look much better). Once the connection is established jconsole opens up Summary Tab which gives me a summary of the bugster application.

I'm sure you'll agree this is rather neat and makes getting started with jconsole very easy. ( Sep 23 2005, 09:38:27 AM PDT ) Permalink Comments [11]

20050919 Monday September 19, 2005

Heap dumps are back with a vengeance! The HPROF agent has been around since 1998 and early betas of J2SETM 1.2. One of its more useful features, at the time, was the ability to generate a heap dump. The heap dump is a dump of all the live objects and classes. The HPROF agent also records where objects are allocated and these are written to the dump file too. Heap dumps aren't very useful without a tool to read them but a novel tool called the Heap Analysis Tool (HAT), courtesy of Bill Foote, was released at around the same time. HAT was very useful as it allowed developers to browse the heap dump, and do rudimentary queries to debug memory leak problems.

HPROF fell on hard times in J2SE 1.3 and 1.4. One consequence of this was that the heap dumps weren't always readable by HAT. The background to this is that HPROF used an experimental profiling interface called JVMPI. JVMPI was designed for the original classic VM where it worked well. An implementation of JVMPI was created for its replacement, the HotSpotTM VM, but it was problematic. The root of the issue is that JVMPI wasn't really designed with modern virtual machines in mind. It required events from places that are highly optimized in modern virtual machines. HPROF required the OBJECT_ALLOC event when started with the heap=dump option. This was one of the more troublesome events as it inhibited many optimizations, and didn't work with all garbage collection implementations.

HPROF returned to its glory days in J2SE 5.0 thanks to a complete make over by Kelly O'Hair The catalyst for the make over (actually a complete re-write) was JSR-163 which defined a new tool interface called the JVMTM Tool Interface. JVM TI broke from the past and didn't provide some events that one might expect in a profiling interface. In particular it doesn't provide an object allocation event - in its place the interface provides support for tools to do bytecode instrumentation. With HPROF re-implemented it meant that heap dumps were working again. HAT was back in business!

In Mustang (Java SE 6.0), HPROF gets new two new siblings which bring new ways to generate heap dumps.

Before we meet the new heap dumpers it is important to mention that they generate simple heap dumps. That is, the dump files contain information about all the objects and classes in the heap but they do not contain information about where the objects are allocated. If you need this information then it is best to run with a JVM TI agent that records this. The NetBeans Profiler is particularly good at this.

Now let us meet the new heap dumpers ...

First, here is an example where we run an application, called ConsumeHeap, with a flag that tells the HotSpot VM to generate a generate a heap dump when we run out of memory: 

$ java -XX:+HeapDumpOnOutOfMemoryError -mn256m -mx512m ConsumeHeap
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid2262.hprof ...
Heap dump file created [531535128 bytes in 14.691 secs]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at ConsumeHeap$BigObject.(ConsumeHeap.java:22)
        at ConsumeHeap.main(ConsumeHeap.java:32)
$

ConsumeHeap, as expected, fills up the java heap and runs out of memory. When java.lang.OutOfMemoryError is thrown a heap dump file is created. In this case the file is 507MB and is created as java_pid2262.hprof in the current directory. If you don't want big dump files in the application working directory then the HeapDumpPath option can be used to specify an alternative location - for example -XX:HeapDumpPath=/disk2/dumps will cause the heap dump to be generated in the /disk2/dumps directory.

[As a complete aside, notice that the java.lang.OutOfMemoryError has a stack trace - this is also new in Mustang - in J2SE 5.0 the OutOfMemoryError would have been thrown without any stack trace]

Now let's look at a second example: 

C:\> jmap -dump:file=app.bin 345
Dumping heap to C:\temp\app.bin ...
Heap dump file created
C:\>

This example uses the jmap utility to generate a heap dump of the java application running as process 345. Astute readers will observe that this example was done on Microsoft Windows but the jmap utility was only included with the J2SE 5.0 releases for Solaris and Linux. This is semi-true for Mustang too but jmap.exe is included and supports a subset of the options available on the other platforms. In particular the -dump option is there on all platforms.

Now let us look at an example that generates a heap dump from a core file. As crashes are rare I've cheated a bit by getting a core file with the Solaris gcore command: 

$ gcore 5831
gcore: core.5831 dumped
$ jmap -dump:file=app.bin `which java` core.5831            
Attaching to core core.5831 from executable /opt/java/bin/java, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 1.6.0-ea-b52
Dumping heap to app.bin ...
Unknown oop at 0xf14b8650
Oop's klass is 0xf14b54d0
Unknown oop at 0xf14e67a8
Oop's klass is null
Heap dump file created
$

The arguments after the -dump option are the executable and the core image file. I used `which java` which gives me the pathname of java. Needless to say these need to match. Less obvious is that you that you can only use jmap from the same JDK build as the executable too. So for example, if you have a core file from 6.0-ea-b52 then you need to use jmap from 6.0-ea-b52 to generate a heap dump from the core file.

You might notice a few warnings in the output. The messages Unknown oop at ... might be a bit off-putting but remember that the core image is taken at an arbitrary time. There is no guarantee that the heap and other data structures are in a consistent state when the crash dump is obtained.

At this point your disks are probably full of heap dumps and you are wondering how to examine them. In the introduction I mentioned the Heap Analysis Tool (HAT). HAT is now included in Mustang (since b51) as a command line utility called "jhat".

Getting started with jhat is easy - just give it the name of the heap dump file: 

$ jhat java_pid2278.hprof
Started HTTP server on port 7000
Reading from java_pid2278.hprof...
Dump file created Sun Sep 18 17:18:38 BST 2005
Snapshot read, resolving...
Resolving 6162194 objects...
Chasing references, expect 12324 dots.........................................................
Eliminating duplicate references..............................................................
Snapshot resolved.
Server is ready.

At this point HAT has started a HTTP server on port 7000. Point your browser to http://localhost:7000 to connect to the HAT server.

[HAT requires a lot of memory so if you try it and it fails with OutOfMemoryError then you might need to give it a larger heap size (for example: jhat -J-mx512m java_pid2278.hprof). The memory consumption of HAT is currently being worked on - expect to see improvements in b54 or b55.]

When you connect to the HAT server you should see a list of all the classes in the heap dump. This is the All Classes query. Scroll to the bottom of the page to see the other queries. A full description of all the queries is more than I have time for today but you can read more in the HAT README.

Those that are familiar with HAT will not see many differences between HAT 1.1 and jhat in b52. However you can expect improvements in b53 (should be available on 9/23). For starters there are a number of robustness improvements - like being able to to parse incomplete and truncated heap dumps. There are also various fixes and finally HAT will be able to read heap dumps generated on 64-bit systems.

Small improvements aside, a useful (and very interesting) addition in b53 is that Sundar Athijegannathan has added a scripting interface to HAT. This make use of the new scripting API. This addition allows developers to enter arbitrary queries into browser and they aren't tied to canned queries that HAT provides. Keep an eye on Sundar's blog for updates on this topic.

To sum up: Mustang (Java SE 6.0) allows heap dumps to be obtained at out of memory time, at any time during the life time of the application, or even after an applications dies with a crash. Also with jhat included in the JDK it means that there is a simple out of the box utility to examine the heap dumps and do rudimentary memory analysis. So heap dumps are back, and back with a vengeance! ( Sep 19 2005, 08:31:34 AM PDT ) Permalink Comments [40]