And another thing... Tim Quinn's Blog

If you see redeployment or undeployment errors on Windows, with GlassFish saying it cannot rename or delete files, you can now help diagnose the problem yourself on your system. This will save you from having to send us your app and it will save us from installing and trying your app, all of which will make for a faster resolution. Plus you don't have to share your app, which is a problem for many people.

Background

When GlassFish deploys an app, it creates an internal directory to hold the files contained in the app. Do not depend on this convention, but currently by default GlassFish places an enterprise app's files in $install-dir/domains/$domain-name/applications/j2ee-apps/$app-name. As it redeploys an app, GlassFish starts by renaming the existing internal app directory to ${app-name}_old. This allows GlassFish to restore the original app in case of errors during the redeployment. And, naturally, during undeployment GlassFish removes the app's directory and its contents.

One aspect of Windows (you decide if it's a feature or a bug!) is that an open file cannot be deleted or overwritten nor can its containing directory be renamed.

Put these facts together and you can see there can be trouble on Windows if, for some reason, a file in the application's directory is open when you try to redeploy or undeploy the application. Perhaps in another post I'll talk about why files might be open. Suffice it to say that this does happen from time to time. The result is failed redeployments or undeployments. Typically restarting the server clears up the problem, but that's a real nuisance especially for developers in the edit/build/deploy/test/repeat cycle.

It's important to find out which file or files are open and why. It could be due to behavior in GlassFish, in the application code, in Java itself, or in a third-party library. The process for solving the underlying problem is different in each case. (I should point out that we continue to work on ways of letting GlassFish tolerate such problems without causing deployment or undeployment failures, regardless of the underlying problem. But it's still good to discover the root cause.)

A tool to help

If it's a JAR that's open and causing the trouble, I can now offer a tool that helps identify what code opens the JAR.

Disclaimer: This tool is unsupported!

It's also a bit rough around the edges but it gives information that's extremely valuable in tracking down where JAR files are opened. (A link to the JAR for the tool is below; I plan to post the source soon.)

How the tool works

The basic idea is this: Whenever a JAR file is opened, a new instance of the java.util.jar.JarFile class is created. JarFile extends java.util.zip.ZipFile which has some very useful information about the file. The tool uses the Java Debugger Interface (JDI) (JDI API spec, intro page for the overall Java Platform Debugger Architecture or JPDA) to connect to the GlassFish JVM. It sets a breakpoint in the ZipFile constructor (which is invoked whenever a zip file is opened) and another in its close method. When the tool detects that the GlassFish JVM reaches the constructor breakpoint it collects some information about the current ZipFile object: the current stack trace, the file path, and the object's hash code. The tool saves this information locally and then lets the GlassFish JVM continue. When the GlassFish JVM reaches the breakpoint for the close method, the tool finds the object's hash code and discards the previously-saved information about that ZipFile instance.

So at any given moment the tool has a record of all zip files (JAR files) that are opened but not yet closed by the GlassFish JVM. This could be a lot of JARs, so the tool allows you to select which files to monitor using a filter.

The tool also responds to two commands (“show” and “exit”). The “show” command will display the recorded information about every currently-open zip file in the GlassFish JVM that matches the file path filter. This output includes the path of the file and the stack trace for each instantiation of ZipFile corresponding to that same file.

The net effect of all this is that if you issue the “show” command immediately after a redeployment or undeployment has failed, you should see stack traces that lead you to the code that opened – but has not closed – the locked JAR file.

Using the tool

Download the tool

At least for the moment, get the tool from here. It's a single, small JAR file. Save it somewhere where you can find it again.

Set up GlassFish for debugging

Because the utility uses the Java debugger API to connect to GlassFish, you'll need to configure your GlassFish instance to accept debugger attaches. If you have not done this before the following instructions may seem long but it's not that bad a process, really. You can either use the admin GUI :

• Choose Application Server from the left-hand panel.

• Choose the JVM Settings tab.

• Make sure the Debug check box is checked.

• Look for the debug options and notice what port number is in the "address" setting or set it to something you want.

or edit the domain.xml file manually (and carefully!) to accomplish the same things:

• The <java-config> element has a debug-enabled attribute; set it to "true"

• The same element has a debug-options attribute where the port number is set. Note its value or set it to something you want.

Start GlassFish

• Start GlassFish and wait for it to settle down. (The admin start-domain command will return before the app server is completely started.)

Start the utility

• Start the utility in a command window using
  
  

java -classpath <path-to-the-utility-jar>;%JAVA_HOME%/lib/tools.jar util.ZipFileMonitor -host yourhost -port 9009 -filter xyz

Replace 9009 if you changed the port number that was already set. Replace “yourhost” with the host where you started GlassFish.


Replace “xyz” with a filter expression that will match the JARs in your application. A good choice is the name of the application itself, since all JARs in an app will appear within a subdirectory tree named after the module name. The default module name for a deployed app is the name of the archive file – foobar for the file foobar.ear, for example.



• From another command window or from the admin GUI deploy the application in question.

• Redeploy or undeploy the application in question, repeating if needed until you see the error.

And when it fails...?

In the command window for the utility enter the “show” command and capture the output. If the output indicates that your application code opened the JAR then decide where the application should close the file and change the logic. If it looks like something else opened the JAR, see if someone has already reported a similar problem in the GlassFish forum. If not, post your output with a brief description of the problem and the community will take a look.

A little more about the utility

The full set of command-line options are:

During deployment and redeployment you'll see quite a few messages displayed on the utility's System.out. This is not the real information we're after but the messages are reassuring that the files you want to monitor (and only those) are in fact being monitored.

Once you have started the utility it accepts two commands: show and exit. The show command will list out every JAR file that is still open and will display the stack trace from which it was opened. Note that a single jar file may be opened from more than one place, so you may see multiple traces for a given JAR. Note also that if you deploy the app and then use the show command you will see a large number of stack traces. This is normal, because those JARs are legitimately still open while the app is deployed. What will be of interest is the set of stack traces after a redeployment or undeployment fails. You can issue the show command as many times as you want.

The exit command detaches from the debugger in the GlassFish JVM and exits.

As I said, there are some rough edges. For example, if you stop GlassFish the utility will report this but does not automatically exit. When you then give it the exit command it throws an exception because it tries to disconnect from the GlassFish JVM even though the connection is no longer in place.

Feedback

For comments on the utility itself, please feel free to add a comment to this posting or send me an e-mail message.

If you encounter a locking problem and you use the tool to help diagnose it, please first search the GlassFish forum to see if someone has already seen a similar problem. If there is none, please create a new topic there. This way the community can cooperate in understanding and solving the problem and you will help other users who may run into the same problem later.

GlassFish JPDA