Alan Burlison's Work Related Ramblings

Tuesday June 27, 2006

Building Win32 JNI code using NetBeans and MinGW

This entry is copied from my personal blog - I'm going to revamp that site in a while, so I've put a copy of it over here with the rest of my Java and NetBeans-related stuff. I've updated it to refer to the latest versions of both the JDK and NetBeans.

I use a GPS mapping application called OziExplorer, and just for laughs I started fiddling around with encapsulating the API for the application so that I could use it from Java. Access to the API is via a DLL, which meant delving into the scary world of both JNI and building DLLs on Win32. Whenever I've tried to do this in the past, it always seemed to require loading vast amounts of unwanted crap on my machine so that I got a compiler that worked, arcane spells involving mapfiles, sacrifical slaughter of various wildlife and so much pain in general that I'd always given up in disgust. However, I'm a glutton for punishment so I thought I'd have another crack at it. I couldn't find anything on the web that told you everything you needed to know all in one place, so I thought I'd document it since I'd got it all working.

Step 1

If you don't already have it, download and install the latest version of the Java Development Kit. I used version 1.5.0_07, the instructions below should work with other versions... Do this first to make you life easy - the NetBeans installer will pick up the latest version of the JDK that you have installed.

Step 2

If you don't already have it, download Netbeans and install it. I used version 5.0, that or anything later should be fine.

Step 3

Install the Ant cpptasks module. This adds support to Ant for building C code. The only bit of this you need is cpptasks.jar, and you should copy it to the lib subdirectory of the new version of Ant you installed in step 3.

Step 4

Install a copy of MinGW. This is a Win32 version of the well-know gcc compiler. Unfortunately the MinGW documentation is as clear as mud on exactly how to do this - if you go to the download page you get a huge shopping list of bits, with no clear directions as to which bits you actually need - the documentation is clearly written with the assumption that you already know. The easiest way to sort the mess out is to download the "MinGW" installer - look for the section entitled "MinGW" and you'll see a file called MinGW-<version>.exe, in my case <version> was 4.1.0. Download and run that and it will ask you for an installation location (I used C:\Program Files\MinGW, then deselect everything except The minimal set of packages required to build C/C++. Leave everything on the next screen selected and start the install. The installer will then pull the bits you need from the web. You can probably prune the list, but the whole thing is only 60Mb so it didn't seem worth the effort.

Step 5

Set up your environment. In order for Ant to be able to run the compiler, it will need to be able to find it, and the same goes for any DLLs that you will use or generate. Windows particularly sucks in this area, so to save spraying DLLs all over my C drive I created the directory C:\DLLs to dump everything in. With that done, right click on My Computer, click on Advanced then Environment Variables. Select Path from the list, then add ;C:\Program Files\MinGW\bin;c:\DLLs to the end, or wherever you installed the compiler, and wherever you intend to install your DLLs. If it isn't already there, add the path to the Java bin directory as well - in my case this is ;C:\Program Files\Java\jdk1.5.0_04\bin. I don't know if it is necessary, but at this point I logged out/into Windows to make sure my environment changes took effect.

Step 6

Add the appropriate JNI code to your project - I don't intend to cover the process for doing that here, instead see the JNI documentation. The tricky bits are getting everything to link together, the problem being caused by the awful Win32 name decoration mess. Basically, DLLs usually use the stdcall calling convention, and when you build them with gcc by default it 'decorates' the symbol names by appending '@' and a number (the number of bytes that will be popped off the stack by the function) - follow the links for details. There are two main issues:

• In the case of the DLL I was wrapping, the functions used the stdcall convention but didn't have the name decorations expected by gcc. The solution was to use a def file in concert with dlltool to generate a gcc library file - then instead of linking directly with the DLL, you link against the library file. The def file itself is just a list of the decorated names that gcc is expecting to see in the DLL - you can get these by looking at the linker output, when gcc can't find the decorated names when it is linking it helpfully prints out the missing symbol names. Beware - if you do build without using stdcall your code will sort-of work, but because the stack isn't being cleaned up properly you'll get all sorts of odd crashes.
• In the case of the DLL I was generating (containing the JNI code), by default gcc adds the name decoration to the symbols but the Java runtime doesn't look for the decorated symbols, so you get run-time linker errors from Java when you try to load the DLL. The solution here is quite simple - add the --kill-at flag to the gcc command line and gcc will generate your DLL without the name decorations, and Java will be happy.

Step 7

Javap can be used to output the method and field signatures that will be needed in the JNI C code. Unfortunately there isn't an Ant target to do this, so it's necessary to roll our own. The first thing is to put the following 2-line Windows batch script somewhere - I called mine javap-s.bat and put it under the nbproject subdirectory of my NetBeans project. See the following section for the explanation of how to hook it into Ant.

@echo off
javap -s -private -classpath %~dps1 %~n1 > %~s2

Step 8

The next tricky bit is adding the necessary goop to your project.xml file to get Ant to build your code bearing in mind the issues I described above. This isn't helped by the fact that the cpptasks add-on has a bug that means it doesn't work properly with MinGW. Here's what works for me. Firstly, add the following to the nbproject/project.properties file:

lib.dir=lib
jni.dir=jni
signatures.dir=signatures

Then add the following to the build.xml file:

    <!-- Load the cpptasks task -->
    <taskdef resource="cpptasks.tasks"/>
    <typedef resource="cpptasks.types"/>

    <!-- Compile the JNI code into a DLL -->
    <target name="-post-compile">
        <!-- Make sure the output directories exists -->
        <mkdir dir="${jni.dir}"/>
        <mkdir dir="${signatures.dir}"/>

        <!-- Run javah to produce a header file for the JNI functions -->
        <javah verbose="yes" classpath="${build.classes.dir}"
         destdir="${jni.dir}">
            <class name="com.oziexplorer.OziAPI"/>
        </javah>

        <!-- Run javap to produce files containing the JNI signatures -->
        <apply executable="nbproject/javap-s.bat" dest="${signatures.dir}"
          resolveexecutable="true" failonerror="true" ignoremissing="false">
            <fileset dir="${build.classes.dir}/com/oziexplorer"
              includes="*.class"/>
            <mapper type="glob" from="*.class" to="*.txt"/>
            <srcfile/>
            <targetfile/>
        </apply>

        <!-- Check the library definition file is up to date -->
        <apply executable="dlltool" dest="${jni.dir}" failonerror="true"
          ignoremissing="false">
            <filelist dir="${src.dir}/com/oziexplorer" files="OziAPI.def"/>
            <mapper type="glob" from="*.def" to="lib*.a"/>
            <arg value="-k"/>
            <arg value="--dllname"/>
            <arg value="OziAPI.dll"/>
            <arg value="--def"/>
            <srcfile/>
            <arg value="--output-lib"/>
            <targetfile/>
        </apply>

        <!-- Compile the C code -->
        <cc link="shared" outtype="shared" multithreaded="true" optimize="speed"
          objdir="${jni.dir}" outfile="${jni.dir}/OziAPI">
            <compilerarg value="-Wall"/>
            <compilerarg value="-D_JNI_IMPLEMENTATION_"/>
            <compilerarg value="-fno-strict-aliasing"/>
            <linker name="gcc">
                <linkerarg value="--kill-at"/>
                <linkerarg value="-oOziAPIJava.dll"/>
            </linker>
            <sysincludepath location="${java.home}/../include"/>
            <sysincludepath location="${java.home}/../include/win32"/>
            <fileset dir="${src.dir}/com/oziexplorer" includes="OziAPI.c"/>
            <libset dir="${jni.dir}" libs="OziAPI"/>
        </cc>
    </target>

    <!-- Copy the stripped DLL and OziAPI.DLL to the dist directory -->
    <target name="-post-jar">
        <apply executable="strip" dest="${dist.dir}" failonerror="true"
          ignoremissing="false">
            <filelist dir="${jni.dir}" files="OziAPIJava.dll"/>
            <mapper type="glob" from="*.dll" to="*.dll"/>
            <arg value="-s"/>
            <srcfile/>
            <arg value="-o"/>
            <targetfile/>
        </apply>
        <copy todir="${dist.dir}">
            <fileset dir="${lib.dir}" includes="*.dll"/>
        </copy>
    </target>

Let's go through it bit at a time:

• The typedef statements pull in the cpptasks extension.
• The mkdir makes sure the output directory exists - this rule runs before the JAR file is generated, so we need to manually create the output directory.
• The javah section runs javah to generate a C header file from the OziAPI.java file, which contains the native function definitions. This is useful to make sure you get the function prototypes correct.
• The next section runs the batch file created in step 8 over each of the generated class files to produce the corresponding text files containing the signatures of the methods and fields. These are placed in the source directory, but that can be changed to suit.
• There also isn't any obvious way of running dlltool from within the cpptasks framework, so it's necessay to manually check the dependencies between the def file and the resulting library, and rebuild if necessary. That's done by the second apply section.
• The cc section has a few things that need clarification:
  • The linkerarg with the -o is to work around the bug I mentioned above - without it the cc task thinks that it is running on Unix rather than Win32 and generates libOziAPIJava.so instead of OziAPIJava.dll.
  • --kill-at stops the linker putting the @ decoration on the generated code and means that the functions can be linked to correctly.
• The -post-jar section strips and copies the resulting DLL to the directory established in step 6, along with any other required DLLs that have been put in the ${lib.dir} directory.

Although it took a bit of fiddling to get all the bits to play together, the process of building DLLs is far easier than it was in the past - the work that the gcc and MinGW crowd have done to do all the dirty work for you is really very impressive.

Posted by alanbur ( Jun 27 2006, 04:37:33 PM BST ) Permalink Comments [0]