Kelly O'Hair's Weblog (blogs.sun.com)

Main | Next page »

Monday Jun 15, 2009

Warning Hunt: OpenJDK, NetBeans, Warnings, and FindBugs

Did you see Bill's FindBugs slides from JavaONE 2009? You should create some step by step directions on getting started with NetBeans, FindBugs and the OpenJDK. We need to get developers working on this.
Humm, Ok, I'll look into that.
Don't just "look into it", do it!
Ok ok already, I'll "do it".
And try and talk about how to fix warnings, and especially the FindBugs errors, maybe talk about some kind of best practices.
Yeah yeah, and I'll take care of the world peace problem too.
Smart ass, how did you ever get this job in the first place.
Everything else I wanted to do was illegal, creating software seemed like the most criminal thing I could do that wouldn't put be in jail.
Oh geez, why does this seem like a Dilbert comic? Get to work!
Aye aye Captain! Batton down the hatches, full speed ahead, we be looking for the White Whale...
"Drink, ye harpooners! drink and swear, ye men that man the deathful whaleboat's bow -- Death to Moby Dick!"
Oh for heavens sake, maybe you should look into a mental institution.

Ok folks here we go...

My sincere apologies to the Whales and Whale lovers out there, just remember Moby Dick is just a fictional story and I am in no way condoning the slaughter of those innocent and fantastic animals.

NetBeans, FindBugs, and Hunting Moby Dick

For a developer that wants to actually change the source code, the very best way to work with a tool like FindBugs is through an IDE.

Get .
Download and install NetBeans 6.5.1, I used the smallest "Java SE" version but any one of the "Java" versions will do. It installs quickly and easily. I would normally recommend the newer NetBeans 6.7, and I am trying 6.7 myself right now, but so far I have been unable to get the other steps below to work with 6.7, so I'd stick with 6.5.1 for now.
Turbo charge it .
Configure the NetBeans memory settings for the best performance, this is optional. See my NetBeans performance blog. Effectively we want to boost the memory sizes to improve the interactive experience. Of course, if you have less than 1Gb RAM, you may want to avoid this completely or adjust the -Xmx number down:
```
Mac<1> mkdir -p ${HOME}/.netbeans/6.5/etc
Mac<2> cat >> ${HOME}/.netbeans/6.5/etc/netbeans.conf
netbeans_default_options="-J-Xms256m -J-Xmx768m -J-XX:PermSize=32m -J-XX:MaxPermSize=160m
-J-Xverify:none -J-Dapple.laf.useScreenMenuBar=true -J-XX:+UseConcMarkSweepGC
-J-XX:+CMSClassUnloadingEnabled -J-XX:+CMSPermGenSweepingEnabled"
```
NOTE: Keep in mind that if you run FindBugs inside NetBeans as I am suggesting here, you may need to adjust the above -Xmx number even higher depending on what you are running FindBugs on (e.g. how many class files or the size of the jar file). Running FindBugs as a separate process allows you to avoid using NetBeans heap space, but you won't get the good integration with the IDE that I'm advertising here, and what I think you need to actively work on fixing the bugs. FindBugs does use up some heap space, just make sure you have enough.
Get FindBugs .
The NetBeans FindBugs plugin is a bit hard to find. You need to tell NetBeans where the UpdateCenter center is that has this plugin. The NetBeans plugin is not maintained by the FindBugs developers. To do this you need to start up NetBeans 6.5.1 and do the following:
1. Select "Tools->Plugins".
2. Select "Settings".
3. Select "Add"
4. Enter any name you want in the "Name:" field, and for the "URL:" field use:
  https://sqe.dev.java.net/updatecenters/sqe/updates.xml
  then select "OK".
5. Select "Available Plugins", select the "SQE Java" plugin, select "Install", and go through the plugin install process.
You should now see several new icons in NetBeans, specifically the icons for the SQE site , the FindBugs tool , and the PMD tool . My experience with FindBugs has been very positive and I trust this tool completely, rarely has it been wrong or given bad advice. On PMD, you will note that the PMD tool has the image of a gun in it's icon, for a good reason, be very careful with PMD, shooting yourself in the foot is pretty easy to do.
Find a NetBeans project to play with. My first example is jmake, a small Java project available from kenai.com. Get yourself a clone with:
hg clone https://kenai.com/hg/jmake~mercurial ${HOME}/jmake
Then use the NetBeans "Open Project..." to open up this directory ${HOME}/jmake as a NetBeans project. It should load up fine, build cleanly, and you can try using FindBugs to look at and even fix some of the bugs.
Note: You need to be a developer member of the jmake project to push changes back.
Or you can get sources, and either find a NetBeans project or create one. I chose to create one and picked the jdwpgen tool in the jdk/make/tools/ directory. So I did this:
1. Cloned the OpenJDK sources (just a partial forest):
  hg clone http://hg.openjdk.java.net/jdk7/jdk7 ${HOME}/myjdk7
  cd ${HOME}/myjdk7
  hg clone http://hg.openjdk.java.net/jdk7/jdk7/jdk
2. Tell NetBeans to create a "New Project", pick "Java Project with Existing Sources", naming the project "jdwpgen", and I'll locate this throw away project at ${HOME}/jdwpgen, then select "Next".
3. Then I tell it to use the source directory ${HOME}/myjdk7/jdk/make/tools/src/, and select "Next".
4. Then I specify build/tools/jdwpgen/** in the "Includes", and select "Finish" to complete the project definition. This process should work for any clean subset of Java sources in the OpenJDK, but I can't attest to that fact, yet.
5. Try a build (should compile 50 files or so), run FindBugs, try and fix something.
  Note: You will need to be a member of the OpenJDK project and be able to create correct changesets to push changes back, this normally requires a BugID, and will require that certain changeset comment and whitespace rules are followed. (Yeah yeah, I know we are working on this, someday I'll be able to describe this better and it will all be open and completely logical ;^). Anyone outside of Sun that has tried to push changes into OpenJDK will confirm how difficult this currently is, but we are working on it.

That's it, play with it, try fixing FindBugs errors or compiler warning errors. Aye, we didn't manage to kill Moby Dick, but we sure sunk in a few good harpoons.

Command Line Usage

The FindBugs and PMD tools can be run from the command line to create all kinds of reports. You will need to download and install FindBugs and/or PMD of course.

Some sample FindBugs command lines:

findbugs -maxHeap 512 -textui -effort:default -low -exitcode ${HOME}/jdwpgen/dist/jdwpgen.jar
findbugs -maxHeap 512 -textui -effort:default -low -exitcode -xml:withMessages -output ${HOME}/jdwpgen/build/findbugs.xml ${HOME}/jdwpgen/dist/jdwpgen.jar

Some sample PMD command lines:

pmd.sh ${HOME}/myjdk7/jdk/make/tools/src text basic,imports,unusedcode
pmd.sh ${HOME}/myjdk7/jdk/make/tools/src xml basic,imports,unusedcode > ${HOME}/jdwpgen/build/pmd_out.xml

I use the command line versions when I just want the data for statistic purposes or as part of a batch build process, which I'll talk about later.

Mac, NetBeans/Ant, and PATHs

FYI... anyone that works with Ant scripts (all NetBeans users) has found out that the best way to get the Ant <exec> to work well is to set PATH properly, but controlling PATH from a GUI launch is tricky. For the Mac see my previous post on NetBeans, Mac and PATHs, which explains the Mac tricks needed to set the PATH for NetBeans.

For Developers, Hudson IS The Answer...

Have you used Hudson? If you haven't, you should, I'll blog about it more very soon, but to get yourself started, do this:

Download Hudson (hudson.war)
In a separate shell window, run:
java -jar hudson.war
Go to http://localhost:8080/ and play with your very own Hudson, create a new job, browse the many plugins available,...

I'll talk about Hudson and how to connect it up to FindBugs, PMD, Warning Messages, etc. in a future blog...

-kto

Posted at 04:37PM Jun 15, 2009 by kto in Java | Comments[0]

Tuesday May 19, 2009

Windows Visual Studio Stupid Pet Trick of the Day

UPDATE (5/19/2009): Added more env vars, trying to get it to work with older 64bit SDKs.
UPDATE (5/3/2009): Added some automatic detection of csh or sh shell type for a better default. Added MSVCDIR and DEVENVDIR as exported env variables.
UPDATE (4/23/2009): Fixed a few problems with paths that do not exist, no longer adding them to PATH. Added -help and -verbose options. Also discovered that these vcvars*.bat files rely on PATH being set (to run 'reg query' to find the SDK), so I had to remove my clearing out of PATH before running it.
Also, a warning. It appears that some of the vcvars*.bat files are adding double quoted strings into PATH, not very nice. They may also have some errors with regards to the Frameworks path handling. The end result is that you get some strange PATH values that don't parse very well or have been truncated or just don't exist. If you are curious, get the latest version and run 'vsvars -v' to see what warnings you get about non-existent directories.

I've blogged about vcvars32.bat and ant before, but I decided to take it a step further.

I'm constantly having to deal with different versions of Visual Studio and am always in need of having to run those silly vcvars32.bat files so the appropriate environment variables get set, but to run them you need to be in a Windows command shell, which I never use. So this is my Windows Visual Studio Stupid Pet Trick of the Day, for you CYGWIN Visual Studio Users, the vsvars.sh script.

The vsvars.sh script will exec a Windows command shell, have it spit out the environment variables it sets, and then it echos out shell environment variable setting commands. And here it is:

Running "vsvars -s" will get you something like:

Nasty looking stuff, but guaranteed to be what the official Visual Studio vcvars32.bat (or vsvars32.bat) files do to the Windows command shell environment.

The nice thing is that if you use it like so:

csh/tcsh: eval `vcvars.sh -c` sh/bash/ksh: eval `vsvars.sh -s`

By default it should figure out what kind of shell you are using and do the right thing, but that logic was a little tricky, so if you know what kind of shell you are using, use the -c or -s options.

It works pretty slick... although I can't possibly test it in all situations.

This will probably be hooked into the JDK builds at some point, or provided with those sources.

-kto

Posted at 04:40PM May 19, 2009 by kto in Java | Comments[4]

Monday May 11, 2009

Fedora 9 and Mac VMware Fusion 2.0.4

UPDATE: (5/11/2009) Patches for openjdk6 were deleted from this blog, not needed any more if using the latest OpenJDK 6 sources.

Thought I would provide some notes on setting up a Fedora 9 VMware image on my Mac laptop using VMWare Fusion 2.0.4. I used the same steps for both Fedora 9 32bit (i386) and 64bit (x86_64), however I had some trouble with installing x86_64, even seemed to trigger a MacOS panic at one point after doing the yum update, not sure what that was all about, only happened once. This was strictly a local Fedora install, so I didn't need to deal with any of the networking issues of setting up a real physical machine.

I'll try and re-create the order of things as best I can:

Create VMware Virtual Machine from Fedora 9 install iso image. I set it up to have a 20GB disk (You cannot change this disk size afterwards!). I'm using 768Mb RAM (512Mb caused slow builds) and during the install I asked for the "Software Development" packages.

Update your system and make sure you have all you need. Logged in as root:

yum install kernel kernel-headers kernel-devel
yum install hg ksh tcsh csh cups cups-devel freetype freetype-devel lesstif-devel
yum groupinstall "X Software Development" "Development Tools" "Java Development"
yum update

This will take a while. A reboot after you are all done would be a good idea.

Install VMware tools. Once you extract out the VMware tools folder vmware-tools-distrib, once again logged in as root do the following:
```
cd vmware-tools-distro
./vmware-install.pl
```
The list of questions to answer is long and convoluted, mostly the default answer works fine, but in some cases it seems to think you are using a remote login and you have to say "yes" to continue the installation.
Mouse problems: For some reason all my single clicks were being treated as double clicks, which drove me nuts. I found this posting which solved the problem, I use option 2 and edited the file /etc/X11/xorg.conf and added the following lines, logged in as root:
```
Section "ServerFlags"
        Option      "AutoAddDevices" "false"
EndSection
```
A reboot of your virtual machine is necessary to fix this.
The default limit on file descriptors is very low, to allow for a larger limit the following addition to the file /etc/security/limits.conf will increase that limit, again logged in as root:
```
################################################
* soft nofile 64000
* hard nofile 64000
################################################
```
You need to logout and back in for these new limits to be available.
Recently it was discovered that the upgraded kernel-headers package has trimmed down the files it delivers to /usr/include/linux/ (i.e. dirent.h) and although this doesn't impact OpenJDK building, it could impact builds of parts of the Sun JDK (plugin). So to avoid this missing include file problem, you have to do this last step because the above steps need the latest and matching kernel-headers files. To get the older kernel-headers package run:
```
yum remove kernel-headers glibc-headers
yum install kernel-headers-2.6.25 glibc-headers
```
Bugs have been filed on the Sun JDK to see if we can break this dependency on the /usr/include/linux/ files.

That's the basic system setup. In addition I also setup my own home directory with the following so I can build the OpenJDK:

Get webrev tool:

mkdir -p ${HOME}/bin
cd ${HOME}/bin
wget http://blogs.sun.com/jcc/resource/webrev
chmod a+x webrev

Get latest ant:

mkdir -p ${HOME}/import/ant_home
cd ${HOME}/import/ant_home
wget http://www.apache.org/dist/ant/binaries/apache-ant-1.7.1-bin.tar.gz
tar -xzf apache*.tar.gz
mv apache-ant-1.7.1/* .

Get forest extension:

mkdir -p ${HOME}/hgrepos
cd ${HOME}/hgrepos
hg clone http://bitbucket.org/pmezard/hgforest-crew hgforest

Setup your ${HOME}/.hgrc file:

cat > ${HOME}/.hgrc <<EOF
[ui]
username = ${USER}
ssh = ssh -C
[trusted]
groups = wheel
[extensions]
fetch=
purge=
mq=
forest=${HOME}/hgrepos/hgforest/forest.py
[defaults]
clone = --pull
fclone = --pull
fetch = -m Merge
ffetch = -m Merge
EOF

Get OpenJDK7 sources (jdk7 build source forest):

mkdir -p ${HOME}/hgrepos/jdk7
cd ${HOME}/hgrepos/jdk7
hg fclone http://hg.openjdk.java.net/jdk7/build jdk7-build

Get OpenJDK6 sources (jdk6 master source forest):

mkdir -p ${HOME}/hgrepos/jdk6
cd ${HOME}/hgrepos/jdk6
hg fclone http://hg.openjdk.java.net/jdk6/jdk6 jdk6-master

Now to see if I can build both OpenJDK7 and OpenJDK6:

# To get rid of a few sanity errors unset JAVA_HOME LANG=C export LANG # My own private copy of ant ANT_HOME=${HOME}/import/ant_home export ANT_HOME # Use the JDK that is part of Fedora 9 ALT_BOOTDIR=/etc/alternatives/java_sdk_1.6.0 export ALT_BOOTDIR # Add java and ant to the PATH PATH="${ALT_BOOTDIR}/bin:${ANT_HOME}/bin:/usr/local/bin:/usr/ccs/bin:/usr/ccs/lib:/usr/bin:/bin:/usr/bin/X11:/usr/sbin:/sbin" export PATH # Go to the root of the jdk7 source forest cd ${HOME}/hgrepos/jdk7/jdk7-build # Build jdk7 # Don't run javadoc, too slow, needs 1024Mb RAM minimum make NO_DOCS=true # Go to the root of the jdk6 source forest cd ${HOME}/hgrepos/jdk6/jdk6-master # Build jdk6 # Don't run javadoc, too slow, needs 1024Mb RAM minimum make NO_DOCS=true

SUCCESS! They both build.

-kto

Posted at 06:29PM May 11, 2009 by kto in Java |

Thursday May 07, 2009

Painful Ant Bite: A generous CLASSPATH and ant.bat out of hell

Painful ant bytes... ;^) Starting to number them...

When ant runs, it sometimes uses temporary files using the java property java.io.tmpdir as the root of the temporary file area for the system. Unfortunately, the temp file used by ant isn't unique enough to prevent two processes running ant from bumping into each other. We had problems using the same machine to build both Solaris 32bit and Solaris 64bit at the same time. OUCH!
Jonathan came up with a good solution specific to the JDK in the langtools Makefile that runs ant. He defines the java.io.tmpdir property on the ant command line to be unique to this build area and platform:
ant -Djava.io.tmpdir='$(ABS_OUTPUTDIR)/build/ant-tmp' ...

The basic idea is to redefine the java.io.tmpdir (the root of the temporary file area) to something more unique to the circumstances, and ideally in a location that will get cleaned up at the right time.
Had a lovely time (<- sarcasm) tracking down a problem on Windows with JDK ant builds. Apparently on Windows, if you manage to get the ant.bat startup instead of the shell script version of the startup, commas are not allowed on the command line. So you cannot do this:
ant -Djavac.debuglevel=source,lines,vars OUCH!
This problem was reported here but nothing was done. Seems like a simple note in the User Manual was a minimum here.
So I downloaded the cpptasks for ant from the ant-contrib site. And it builds from the command line just fine with the latest ant, but it won't build using the ant that comes with NetBeans, and it won't build when loaded into NetBeans. So why is that? The build requires xercesImpl.jar, but the cpptasks build.xml file doesn't explicitly say that, so how did it find it in one case and not in another?
Turns out that the default behavior for the ant <javac> task is to include all the ant runtime classpath in your java compilation. Yipes! That seems like a horrible default if you ask me, depending on what ant decides to use in its runtime classpath, you get it all? :^( OUCH!
Seems like every ant installation could potentially behave differently, depending on how ant is configured.
So I'm thinking I want to change all <javac> uses to <javac includeAntRuntime="false">
This was also talked about on a JavaLobby forum, with some helpful comments.

-kto

Posted at 11:40AM May 07, 2009 by kto in Java | Comments[6]

Monday Apr 13, 2009

JavaOne 2009 is coming soon...

Don't forget to sign up for JavaOne 2009! You can browse the Online Catalog and see the huge list of talks and sessions this year.

Just a few technical sessions on my list:

-kto

Posted at 06:20PM Apr 13, 2009 by kto in Java |

Source Repository Rules

A few commandments for dealing with Software repositories (source bases) and the build results of those repositories.

There shalt not be binary files in the repository.
Binary files (executables, native libraries, zip files, jar files, etc.) are NOT source and should not be managed in a source repository.

Keep thy path names simple.
Directory names and filenames in the repositories should never contain blanks or non-printing characters. Certain characters such as '$' should also be avoided.

There shall be one newline convention.
The contents of all source files should follow the standard unix conventions on newlines (no ^M's).

Generated source files shall not be added as managed files.
Source files generated during the build process should not be managed files in a repository.

All output from the build shall be kept separated from the source.
All files generated during the build should land in a well defined output only directory such as build/ or dist/. The src/ directory should never get written to during the build process.

Of course the commandments will change from time to time. Reminds me of this scene in Mel Brooks "History of the World Part II" movie, loved the ending "These 15 Commandments... oops... 10 Commandments!". ;^)

-kto

Posted at 09:58AM Apr 13, 2009 by kto in Java | Comments[7]

Saturday Feb 28, 2009

Ant and Importing

Just how many copies of junit.jar have been added to source repositories on the planet? Quite a few I imagine, seems like a waste of repository data space and well, just wrong. Not junit, which is a fantastic product, just the fact that we have so many copies. Granted you have gained a pretty stable tool by freezing the version you have, and you have guaranteed having a copy at all times, but is it a good idea to add all these binary fines to your repository data? As the list of tools like this grows and grows, does the "just add it to the repository" solution continue to scale? And each time you need a new version, you end up adding even more binary data to your repository.

Some projects have taken to doing a kind of "tools bootstrap" by downloading all the open source tools the first time you setup a repository, making the files immune from normal 'ant clean' actions. Ant has a a task called <get> which can allow you to download tool bundles and it works quite well, but there are some catches to doing it this way. Expecting all the download sites to be up and available 24/7 is not realistic. And predictability is really important so you want to make sure you always download the same version of the tools, keeping a record of what versions of the tools you use.

So what we did in the openjfx-compiler setup repository, was to create an import/ directory to hold the downloaded tools, automate the population of that area with the <get> task, and also allowed for quick population of import/ with a large import zip bundle. The initial version of the repository had a very similar mechanism, so this idea should be credited to the original authors on the OpenJFX Compiler Project.

This logic is contained in the file build-importer.xml of the setup repository and for each tool NAME downloaded, a set of properties is defined (import.NAME.*), and 2 ant tasks import-get-NAME and import-NAME. Probably best to look at the bottom of this file first. As before quite a few macrodefs were used to make this all work.
Your browser does not support iframes.

The ant build script then just uses ${import.junit.jar} to get a junit.jar file.

You can actually try this out yourself pretty easily if you have Mercurial (hg) and ant by doing this:

hg clone https://kenai.com/hg/openjfx-compiler~marina-setup setup cd setup ant import

Of course I'll predict that it fails the first time for 50% or more people, this kind of downloading is just not that reliable when depending on all these sites. So you may have to run ant import a few times.

-kto

Posted at 04:36PM Feb 28, 2009 by kto in Java | Comments[5]

What Continuous Build Systems Got Wrong

Pearls Before Swine had a great comic recently, not that I have any issue with people that don't agree with me:

I have always been a big fan of regular automated build and testing during development. With every project I have ever been on, if there wasn't some kind of nightly build and test of the team sources, or my own sources, I set one up. Anyone working in software development knows or should know the importance of having this constant monitoring of the product status. So don't get me wrong, I think that all the continuous build systems out there are great to have.

BUT... They all seem to have grown up around the single central source repository model, and I think they have developed some bad habits. The ideal to me is that a changeset or change to the source repository is built and tested BEFORE it ever sees a shared team area, and the continuous build of the team area should be only a final verification. The build should never be broken, we should be able to have golden (or more golden?) changesets all the time. With a Distributed SCM like Mercurial, this is actually quite possible. With a Distributed SCM a changeset can be created, then the repository can be merged with the parent repository, and everything can be fully built and tested before the changeset is ever seen by other developers.

So what we need is a continuous verification build system.

Clint comments about Git and Continuous Integration and also refers to another article on Agile Continuous Integration with Git.. These comments apply to Mercurial too.

I know that the ElectricCloud ElectricCommander product supposedly has a 'trial run' type of feature I was told about, which is a step in the right direction. And I have heard that the Hudson system may have some 'test build' plugin someday.

-kto

Posted at 01:00PM Feb 28, 2009 by kto in Java | Comments[8]

Monday Feb 23, 2009

Ant and Platform Specific macrodefs

Ant works great for any pure Java project, very simple to deal with, might get a little tricky when dealing with jar manifests, but not bad, and very efficient in terms of limiting the Java VM startup overhead. But what about platform specific tasks? I myself find the "<exec>" ant task so painful to use that I avoid it at all costs, or at least isolate each use to a "<macrodef>". And this macrodef isolation actually works pretty well when you are dealing with many different platforms that you need to build on.

There are many solutions to the issue of platform specific builds, including the ant cpptasks and I am sure many more. So what I am saying here is not new and not the end all to this issue. Just some ideas for people to consider when up against this problem. Please, add your comments if you have some good references and ideas. It's very obvious to me that I am no where near an ant expert, so take all this with a grain of salt. I also want to give credit to the many JavaFX teams and individuals you wrote the various ant scripts in all the repositories, most of this is a consolidation of other peoples ideas and techniques.

So how did the JavaFX SDK deal with multiple platform issues in ant? This project was composed of many sub repositories, each with different system needs and often using slightly different techniques for building. The top repository (or setup repository or root repository) we have allows for this independence as much as possible, but at the same time trys to create some kind of structure to the build process. From the Mercurial file view of the openjfx-compiler setup repository, I will try and explain what is happening.

Basic OS arch detection is done in the file build-os-arch.xml:
Your browser does not support iframes.
People unfamiliar with xml or ant might find the syntax a bit convoluted, it takes time to get used to it. Key here is the property os_name (which will contain one of: solaris, windows, linux, or macosx), and will be used in the build-defs.xml file to import the right platform specific file build-${os_name}-defs.xml. Keep in mind this is unique to this project, but the basics should work for any multi-platform build project.
The platform specific macrodefs are in the build-${os_name}-defs.xml files, customized for each OS, and each defines the -init-platform-defs task. Consider the macosx file:
Your browser does not support iframes.
Special to this file is the ability to run the xcodebuild utility, this macrodef should probably be turned into some kind of generic do-project-build macrodef, someday. Note that with the JavaFX SDK project we have allowed teams to work in sparse Mercurial forests, this repository we are looking at is the top repository but it could have many sub repositories. Depending on the sub repositories present, there are different needs. We try and check them in these build-${os_name}-defs.xml files, via the -init-platform-defs target.
Pull it all together with the file build-defs.xml which we ask all sub repositories to import early in their own build.xml
Your browser does not support iframes.
This file is imported by each subrepository. Note that this file imports build-os-arch.xml, build-${os_name}-defs.xml, and many other files to provide lots of macrodefs and property settings for a sub repository.
Then we established an ant target contract between each sub repository and the top repository by requiring certain jfx-* targets to be available in the sub repository, for example in the openjfx-compiler repository build.xml file (somewhere in the first 100 lines you should see jfx-* targets defined):
Your browser does not support iframes.
Note that it imports in ../build-defs.xml and has defined a set of jfx-* targets for use by the top repository build.xml file.
The forest build script then uses some macrodefs to cycle through the various sub repositories (sometimes called components) in the file build-components.xml, look for the do-all-* targets:
Your browser does not support iframes.
Which you will see imported in the top level build.xml file:
Your browser does not support iframes.
We have a cached area where a previous SDK build is used in the case of a partial forest, allowing a developer to concentrate on the work in a single repository and not have to build the entire forest. The OpenJDK builds uses a similar concept with the Import JDK, where the pieces you aren't building can come from. Effectively, we will cycle over the sub repositories present, in a particular order, and request each one to perform a certain action as defined by the jfx-* target contract. Look for the use of the do-all-components macrodef for where we will cycle over the sub repositories. (It's a shame that ant doesn't have some kind of applyant task, but you use the tools you are given.)
You can read more about all this in the JavaFX SDK Build README.

The JavaFX SDK project is a bit unique, and the techniques used in it's build process may not suit many projects, but I thought some of this might be of interest to anyone considering putting their hand into any large ant nest someday. :^)

Hope someone has found this helpful, as always, comments on better ideas is always welcome.

-kto

Posted at 10:55AM Feb 23, 2009 by kto in Java | Comments[1]

Saturday Feb 21, 2009

Are you a House Elf?

For those of you who haven't seen the Harry Potter movies or read the books, a House Elf is an obedient servant who must obey the orders of their masters. They have their own special magical powers but are bound to their master until freed with a present of clothes (most of them wear old pillowcases). When they do go against orders, they must punish themselves. They pride themselves in their work and are very loyal servants.

In the Harry Potter stories, Dobby (in the picture above) had to punish himself many times for going against his master, but he did it to protect Harry Potter, and because his master was just plain evil. Dobby was an unusual House Elf, he stood up for what he thought was right, even though it was painful.

Are you a House Elf? Are we all House Elves? Most people I know and work with take great pride in what they do. I would like to think that when the time comes, and we are asked to do something we feel is wrong, that we all behave like Dobby, and do the right thing.

-kto

Posted at 09:16PM Feb 21, 2009 by kto in General |

Monday Feb 16, 2009

The pain of vcvars32.bat, and an ant solution

On Windows, and using Ant to do native compilations with Visual Studio requires certain environment variables to be set. So to be able to say "just run ant", you need to require the user to have set these properties before they start ant. The standard way to set them is with the script vcvars32.bat or vsvars32.bat. So how do you just make this work inside ant? You can't just call vcvars32.bat from your ant script, or can you?

Turns out that you can dip into a cmd.exe environment just long enough to run vcvars32.bat, print out the settings as a properties file and pop back out. As an example, see the file genVSproperties.bat. Of course, the \ characters need to be changed to /. This results in something like:


############################################################
# DO NOT EDIT: This is a generated file.
windows.vs.vsvars32.bat=C:/PROGRA~1/MICROS~2.NET//Common7/Tools/vsvars32.bat
windows.vs.VS71COMNTOOLS=C:/PROGRA~1/MICROS~2.NET//Common7/Tools/
windows.vs.DEVENVDIR=C:/Program Files/Microsoft Visual Studio .NET 2003/Common7/IDE
windows.vs.VCINSTALLDIR=C:/Program Files/Microsoft Visual Studio .NET 2003
windows.vs.VSINSTALLDIR=C:/Program Files/Microsoft Visual Studio .NET 2003/Common7/IDE
windows.vs.MSVCDIR=C:/Program Files/Microsoft Visual Studio .NET 2003/VC7
windows.vs.INCLUDE=C:/Program Files/Microsoft Visual Studio .NET 2003/VC7/ATLMFC/INCLUDE;C:/Program Files/Microsoft Visual Studio .NET 2003/VC7/INCLUDE;C:/Program Files/Microsoft Visual Studio .NET 2003/VC7/PlatformSDK/include/prerelease;C:/Program Files/Microsoft Visual Studio .NET 2003/VC7/PlatformSDK/include;C:/Program Files/Microsoft Visual Studio .NET 2003/SDK/v1.1/include;
windows.vs.LIB=C:/Program Files/Microsoft Visual Studio .NET 2003/VC7/ATLMFC/LIB;C:/Program Files/Microsoft Visual Studio .NET 2003/VC7/LIB;C:/Program Files/Microsoft Visual Studio .NET 2003/VC7/PlatformSDK/lib/prerelease;C:/Program Files/Microsoft Visual Studio .NET 2003/VC7/PlatformSDK/lib;C:/Program Files/Microsoft Visual Studio .NET 2003/SDK/v1.1/lib;
windows.vs.LIBPATH=
windows.vs.PATH=C:/Program Files/Microsoft Visual Studio .NET 2003/Common7/IDE;C:/Program Files/Microsoft Visual Studio .NET 2003/VC7/BIN;C:/Program Files/Microsoft Visual Studio .NET 2003/Common7/Tools;C:/Program Files/Microsoft Visual Studio .NET 2003/Common7/Tools/bin/prerelease;C:/Program Files/Microsoft Visual Studio .NET 2003/Common7/Tools/bin;C:/Program Files/Microsoft Visual Studio .NET 2003/SDK/v1.1/bin;C:/WINDOWS/Microsoft.NET/Framework/v1.1.4322;"C:/cygwin/bin;C:/cygwin/;c:/PROGRA~1/MICROS~2.NET/Common7/IDE;c:/PROGRA~1/MICROS~2.NET/Vc7/bin;c:/PROGRA~1/MICROS~2.NET/Common7/Tools;c:/PROGRA~1/MICROS~2.NET/Common7/Tools/Bin/prerelease;c:/PROGRA~1/MICROS~2.NET/Common7/Tools/Bin;c:/PROGRA~1/MICROS~2.NET/SDK/v1.1/bin;c:/PROGRA~1/Java/JDK16~1.0_1/bin;C:/cygwin/home/ohair/import/APACHE~1.1/bin;C:/cygwin/usr/local/bin;C:/cygwin/bin;C:/cygwin/bin;C:/cygwin/usr/X11R6/bin;C:/cygwin/usr/sbin;C:/cygwin/sbin;C:/cygwin/home/ohair/hgws/home_bin/bin;C:/cygwin/home/ohair/hgws/home_bin/bin/win32;C:/cygwin/home/ohair/bin;c:/WINDOWS/System32;c:/WINDOWS;c:/WINDOWS/System32/Wbem;C:/cygwin/lib/lapack";
############################################################

So in your ant build script you can effectively do this (on Windows only of course :^):


<target name="generate-vs-properties">
    <exec dir="." executable="cmd" output="vs.properties">
        <arg value="/q"/>
        <arg value="/c"/>
        <arg value="genVSproperties.bat"/>
    </exec>
    <replace file="vs.properties" token="\" value="/"/>
    <property file="vs.properties"/>
</target>

So now what? The last piece of the puzzle is that to need to set the environment variables when you run Visual Studio, e.g.


<property file="vs.properties"/>
<exec dir="." executable="${windows.vs.VS71COMNTOOLS}/../../Vc7/bin/cl.exe" failonerror="true">
  <env key="VS71COMNTOOLS" value="${windows.vs.VS71COMNTOOLS}"/>
  <env key="INCLUDE" value="${windows.vs.INCLUDE}"/>
  <env key="LIB" value="${windows.vs.LIB}"/>
  <env key="LIBPATH" value="${windows.vs.LIBPATH}"/>
  <env key="PATH" value="${windows.vs.PATH}"/>
  <arg line="${compile_options}"/>
</exec>

Hope this is helpful to someone.

-kto

Posted at 01:28PM Feb 16, 2009 by kto in Java | Comments[4]

JavaFX Compiler Setup Files

So what happened to my Java and OpenJDK/JDK blogs for the past few months?

Yup, JavaFX bit me. I've been busy working on a special build project for JavaFX, including learning all about the Hudson continuous build system, Ant build script capabilities, and dealing with another forest of Mercurial repositories. Along the way I discovered some new build tricks which I will try and share in separate postings.

The OpenJFX Compiler Project is now sporting a new Mercurial repository (previously was in SubVersion), and a new setup repository that has taken up a great deal of my time. This OpenJFX Compiler project is the "open source" part of JavaFX right now.

The OpenJFX Compiler Setup Files is an open Mercurial repository that allows for building of the entire JavaFX product from a forest of repositories, not all open of course.

Now before you post a question asking why JavaFX isn't all open source, you would be asking the wrong person, I don't know and I have very little influence over this, see the open source statement here. Also see the OpenJFX Data Site for more information on what is visible in the OpenJFX Compiler project.

Some of the issues I tried to tackle with this new setup may relate to many other projects:

Dealing ant scripts on a very large project full of hundreds of ant scripts.
Dealing with ant and native compilation, including using GNU make, Mac xcodebuild, Visual Studio devenv, and the always painful Visual Studio vcvars32.bat environment variable setup on Windows.
Builds of a Mercurial forest, allowing maximum independence but sharing what makes sense to share.
Hudson setups and benefits, and the limitations of continuous build system with a distributed source code management system (Mercurial).
Ant tricks and limitations I found.
Source repository rules, what you should and should not put into your repository.

Sun	Mon	Tue	Wed	Thu	Fri	Sat
« June 2009
	1	2	3	4	5	6
7	8	9	10	11	12	13
14	15	16	17	18	19	20
21	22	23	24	25	26	27
28	29	30

Today