Thursday Aug 27, 2009

Controlling the order of JUnit tests

Unit tests are small and independent, but today I was working on some functional tests that run though a series of steps (some using htmlunit which worked very well!).. most of the time the tests ran in the order they appeared in my code, but once I got enough tests a couple showed up out of order.  I didn't find a way to control the order individual @Test methods were run, but I did find that the JUnit Suite class provides control over the order separate test classes are run.  Example:

import org.junit.runners.Suite;
import mypkg.MyTests.*;

@RunWith(Suite.class)
@Suite.SuiteClasses({Group1.class, Group2.class, Group3.class})
public class MyTests {
  public class Group1 {
    //...
  }
  public class Group2 {
    //...
  }
//...
}

The order of tests within each group is not predictable, but the groups run in order.  To avoid listing all the inner classes in the @Suite.SuiteClasses annotation, I added this:

    public class SortedSuite extends Suite {
	public SortedSuite(Class<?> klass, RunnerBuilder builder) throws InitializationError {
	    super(builder, klass, getSortedClasses(klass));
	}

	private static Class<?>[] getSortedClasses(Class<?> klass) throws InitializationError {
	    TreeSet<Class<?>> list = new TreeSet<Class<?>>(new Comparator<Class<?>>() {
		public int compare(Class<?> o1, Class<?> o2) {
		    return o1.getSimpleName().compareTo(o2.getSimpleName());
		}
	    });
	    for (Class<?> innerclass : klass.getClasses()) {
		if (Object.class.equals(innerclass.getSuperclass())) {
		    list.add(innerclass);
		}
	    }
	    return list.toArray(new Class<?>[0]);
	}
    }

This automatically finds the inner classes and sorts them by name.  It only adds classes that extend Object, to avoid adding some other classes I had used in my tests.  Now all it takes is @RunWith(SortedSuite.class).

Posted at 09:47PM Aug 27, 2009 by mindless in Sun  |  Comments[1]

Thursday Mar 12, 2009

HTTPS Login for Hudson deployed in Sun Java System Webserver 7

I helped setup Hudson for my group's automated builds.  We want to use HTTPS for secure transmission of passwords when logging in, but plain HTTP is fine for use of the application itself.  Here is how we set this up in Sun Java System Webserver 7:

  1. Install a certificate to be used for HTTPS into SJSWS.
  2. Add a second http-listener in the SJSWS configuration, so it now listens on both port 80 for http and port 443 for https.
  3. Navigate to the Virtual Server config, Content Handling tab, URL Redirects subtab.
    Add the following two redirects:
    • Condition: $url =~ '^http://[^/]*/login'
      TargetURL: https://{server-hostname}
      Fixed URL: not checked
    • Condition: $url =~ '^https:' and $uri !~ '^/(login|j_acegi_security_check)'
      TargetURL: http://{server-hostname}
      Fixed URL: not checked
  4. Manually edit $SJSWS_HOME/admin-server/config-store/{config-name}/config/{server-name}-obj.conf
    Find this line: NameTrans fn="ntrans-j2ee" name="j2ee"
    Move this line just below the last "</If>" line for the URL redirects.
  5. Deploy the configuration changes

The manual edit is unfortunate, but the default setup hands off requests to J2EE webapps before processing redirects... so requests handled by Hudson would ignore these redirect rules. Moving that line down allows the redirects to work.
NOTE: if you ever add/edit any of these redirects using SJSWS admin console then you need to again manually edit this file and move that line. It will always add new/editing entries just below that line.

How the redirects work: When you visit Hudson it will be HTTP. When you click the login link the first redirect jumps you over to HTTPS. The second redirect rule allows HTTPS requests for 3 things:

  1. /login -- viewing of the login page
  2. /loginError -- any errors when logging in
  3. /j_acegi_security_check -- this is where the actual login request with username/password is POSTed
After successful login Hudson will take you to some page that doesn't match the patterns allowed for HTTPS so it jumps you back to plain HTTP.

Posted at 01:41PM Mar 12, 2009 by mindless in Sun  |  Comments[1]

Tuesday Feb 12, 2008

Autopopulate name/email from LDAP when new Code Collaborator user logs in

I setup Code Collaborator for my group's code reviews.  Once I got LDAP authentication setup I found that when new users login it doesn't know their full name or email address.  I was able to populate these with a trigger in the Oracle database that execs ldapsearch to find the data.

First login as sysdba and grant permissions required for java code to exec an external program: 

exec dbms_java.grant_permission('CCOLLAB', 'SYS:java.io.FilePermission', '/usr/bin/ldapsearch', 'execute');
exec dbms_java.grant_permission('CCOLLAB', 'SYS:java.lang.RuntimePermission', 'writeFileDescriptor', '');
exec dbms_java.grant_permission('CCOLLAB', 'SYS:java.lang.RuntimePermission', 'readFileDescriptor', '');

Now login as the Code Collaborator database user (I used CCOLLAB, as seen above) and create the java class: 

create and compile java source named "LdapLookup" as
import java.sql.*;
import java.io.*;
import java.util.regex.*;
public class LdapLookup {
  public static void lookup(String username, String[] fullname, String[] email) throws SQLException {
    // ..code here to exec /usr/bin/ldapsearch, parse data from
    // ..its output and store in fullname[0] and email[0]
  }
}

..and the stored procedure:

create procedure do_LdapLookup(username varchar2, fullname OUT varchar2, email OUT varchar2)
as language java name 'LdapLookup.lookup(java.lang.String,java.lang.String[],java.lang.String[])';

Notice how Oracle uses String[] type for an OUT (or IN OUT) parameter.  Assigning to the first array element sends a value back.

Run the procedure to test it.. I found the first call got "ORA-29549: class CCOLLAB.LdapLookup has changed, Java session state cleared", but the following call worked.
Finally create the trigger:

create trigger NEW_USER_TRIGGER
before insert on COLLABUSER for each row
declare
  fullname varchar2(200); email varchar2(200);
begin
  do_LdapLookup(:new.user_login, fullname, email);
  if fullname is not null then
    :new.user_name := fullname;
    :new.user_email := email;
  end if;
end;

All done.  Now when a new user logs in for the first time (using their LDAP password), they get their full name and email address fields in Code Collaborator populated from the data already in LDAP.

Side note: Code Collaborator's installer didn't get LDAP auth setup in Tomcat correctly (at least it didn't work in my environment).  After install I edited the tomcat/conf/Catalina/localhost/ROOT.xml and changed:

<Realm allRolesMode="strictAuthOnly" className="org.apache.catalina.realm.JNDIRealm" connectionURL="ldap://server-name:389" userPattern="uid={0},ou=people,dc=sun,dc=com"/>

to:

<Realm allRolesMode="strictAuthOnly" className="org.apache.catalina.realm.JNDIRealm" connectionURL="ldap://server-name:389" userBase="ou=people,dc=sun,dc=com" userSearch="(&amp;(uid={0}))"/>

Posted at 04:49PM Feb 12, 2008 by mindless in Sun  |  Comments[0]

Friday Jan 25, 2008

Capture java thread dump from truss

I've been working with a daemon java process lately and sometimes I don't have a console where the process will write its thread dump when I send a QUIT signal.  Here's a little perl I used to get a readable thread dump from truss..

alias jdump 'kill -QUIT \!* ; truss -wall -p \!* | & perl -ne '\''next unless s|^/\d+:\t  ||;chomp;while($_){print (substr($_,0,1) eq "\\" ? substr($_,1,1) eq "n" ? "\n" : "\t" : substr($_,1,1));$_=substr($_,2)}'\'

jdump {pid} 

Posted at 11:49AM Jan 25, 2008 by mindless in Sun  |  Comments[1]

Thursday Jul 05, 2007

JNDI + JDBC and Debugging with Sun Java System Webserver 7

My web searches didn't find all these steps in one nice recipe, so I thought I'd write up what I did to replace a DriverManager.getConnection() call with a JNDI lookup to use a DB connection pool managed by Sun Java System Webserver 7.

First, the java code itself.. I found that either of these worked:

InitialContext ctx = new InitialContext();
DataSource ds = (DataSource)ctx.lookup("java:comp/env/jdbc/mydb");

or

InitialContext ictx = new InitialContext();
Context ctx = (Context)ictx.lookup("java:comp/env");
DataSource ds = (DataSource)ctx.lookup("jdbc/mydb");

Both followed by:

Connection c = ds.getConnection();

The InitialContext/Context classes are in javax.naming and DataSource is in javax.sql.  You may also use ds.getConnection(username, password).

Next, the web application configuration so the "jdbc/mydb" lookup will work. The WEB-INF/web.xml file needs a section like this inside the main <web-app> tag: 

<resource-ref>
  <description>My database connection</description>
  <res-ref-name>jdbc/mydb</res-ref-name>
  <res-type>javax.sql.DataSource</res-type>
  <res-auth>Container</res-auth>
  <res-sharing-scope>Shareable</res-sharing-scope>
</resource-ref>

I'm not too sure what the res-auth and res-sharing-scope items are about (probably more web searches can reveal this), but I used NetBeans 5.5 to create this entry and it added those.  In NetBeans, open the web.xml file and select the "References" section in the buttons across the top.  Under "Resource References" click Add and it shows a dialog to enter all these settings.

The next step is a bit of magic glue to connect this resource reference to its definition in SJSWS7 (pretty sure this is needed for SJSAS9 too).  I don't know why the server can't use the resource name directly.. also note that without this glue the error messages/java exceptions provide no clue about what is wrong.  The magic: create a WEB-INF/sun-web.xml file that simply maps between the resource name and the JNDI name.

<sun-web-app>
  <resource-ref>
    <res-ref-name>jdbc/mydb</res-ref-name>
    <jndi-name>jdbc/mydb</res-ref-name>
  </resource-ref>
</sun-web-app>

I suppose you could use different names, but why introduce confusion about which name to use?  To create this file in NetBeans I opened the Web Pages / WEB-INF subtree in the project viewer, then right clicked on WEB-INF and chose New / XML Document.

The last step is to define the database connection in the app container.  In SJSWS, login to the administration server and browse to the configuration for the instance in use.  Click the Java tab and Resources subtab.  Under JDBC Resources click New and follow the wizard to define the DB connection and pool settings.  The default value for the JNDI name was "jdbc-resource-1" but I used / instead of - in my name, "jdbc/mydb".  For the Datasource Class Name be sure to enter the DataSource implementation for the DB type, not the Driver class used with DriverManager.getConnection().

 

Yay, I have a database connection via a connection pool managed by SJSWS7.  But now I wanted to debug other things in my application.. in NetBeans 5.5 you can select Tools / Update from the menu and add support for deployment to SJSWS7.  However, I didn't find this at first so I added a few lines to my build.xml file so when I do "Build" in NetBeans it also deploys to my SJSWS server.

<target name="-post-dist">
  <echo>Deploy to my SJSWS instance</echo>
  <exec executable="/path/to/webserver7/bin/wadm">
    <arg value="--user=admin"/>
    <arg value="--password-file=/path-to/admin-pw"/>
    <arg value="--commands-file=/path-to/deploy-myapp"/>
    <arg value="--no-prompt"/>
  </exec>
</target>

The admin-pw file has a single line with wadm_password={password}.  The deploy-myapp file has:

puts [add-webapp --config=myconfig.mydomain.com --vs=myserver.mydomain.com --uri=/myapp /path/to/myapp/dist/MyApp.war]
puts [deploy-config myconfig.mydomain.com]


Next I setup the debugger, which is possible since both sides support JPDA (Java™ Platform Debugger Architecture).  Login to the SJSWS admin server.  In the settings for my configuration under Java tab / JVM Settings subtab there is a Debug section.  There is a checkbox to turn on debug mode and a line to see or set the port number used by the debugger.  After restarting the server in debug mode you can select "Attach Debugger" in the Run menu of NetBeans.  Enter the port number here and you can debug the app running in SJSWS.

Posted at 09:08AM Jul 05, 2007 by mindless in Sun  |  Comments[0]

Tuesday May 01, 2007

Subversion + Apache + LDAP on Solaris

A few weeks ago I setup a subversion server for some prototype development.  It took some research and quite a bit of trial and error to get all the dependencies and various pieces put together, so today I went through the process again and recorded the steps.  The goal is to get a subversion server setup using http(s) URLs and LDAP for authentication, plus ViewVC for web based viewing of the repository.

  1. Dependencies for Apache/Subversion
    • OpenSSL for https support.  Packages are included in Solaris 10.  Alternative: separate compilation of newer OpenSSL version.
    • LDAP for authentication support.  SUNWlldap package is included in Solaris 10.  Alternative: OpenLDAP

  2. Requirements for ViewVC:
    • Python to run ViewVC.  SUNWPython (python 2.4.4) package included in JDS.  Alternative: SMCpython (python 2.5.1) package from sunfreeware.com
    • SWIG for subversion-python bindings.  Compile and install from website (see some notes in subversion/bindings/swig/INSTALL included with subversion source code).  Alternative: SMCswig + SMClibgcc packages from sunfreeware.com
    • diffutils so ViewVC can display differences between file revisions.  Compile and install from website.  Alternative: SMCdiffu package from sunfreeware.com or SFWdiffu package (I think from Solaris Companion disk, installs into /opt/sfw/bin/gdiff)

  3. Apache 2.2.4 webserver
    4. set path=(/path/to/sunstudio,v11.0/SUNWspro/bin /usr/ccs/bin /usr/openwin/bin /usr/local/bin /usr/sbin /usr/bin .)
    ./configure --with-ssl=/usr/sfw --with-ldap --enable-mods-shared="ssl deflate rewrite ldap authnz-ldap dav dav-fs dav-lock"
    make install

    The apache modules list includes ssl for https support, ldap authentication modules (note the extra --with-ldap flag also needed), dav modules required for subversion, and deflate for compression when subversion interacts with the server.  Also threw in mod_rewrite.  After install, configure /usr/local/apache2/conf/httpd.conf (user/group, serveradmin, etc).

    Solaris 10 does include an Apache 2.0 package, but it does not have LDAP support.  I tried to compile just the LDAP modules and add them in, but was unsuccessful (probably because the core code wasn't built with the --with-ldap flag?), so that's why I did my own Apache 2.2 build.  By changing a couple paths in /lib/svc/method/http-apache2 and setting logfile/pidfile locations in httpd.conf you can use svcadm to manage this Apache build instead of the bundled one. 

  4. Subversion 1.4.3

    5. See additional detail about these steps in INSTALL and subversion/bindings/swig/INSTALL files.  I ran into a couple problems with the builds for subversion and python bindings.  The first was in finding the OpenSSL libraries near the end of the subversion build.. fixed this by setting LDFLAGS below.  The second was in building the python bindings with sunstudio compiler when using the SMCpython package.  The build process determines that python was built with gcc so it tries to use gcc for building the bindings even though gcc is not present.  After resolving this, the bindings failed to load due to missing symbols.. no idea why these libraries did not link to all the required libraries.. to avoid these problems, extract the subversion-1.4.3 and subversion-deps-1.4.3 files and then modify the configure script as follows:

    • Add ac_cv_python_compile="cc -DNDEBUG -O2 -Kpic" right after each line that defines that variable.
    • Add ac_cv_python_link="cc -G" right after each line that defines that variable.
    • Add ac_cv_python_libs="-lssl -lxml2" right after each line that defines that variable.
    setenv LDFLAGS '-R/usr/sfw/lib'
    ./configure --with-apxs=/usr/local/apache2/bin/apxs --with-ssl --without-berkeley-db --with-apr=/usr/local/apache2 --with-apr-util=/usr/local/apache2
    make install
    make swig-py
    make install-swig-py
    echo /usr/local/lib/svn-python > /usr/local/lib/python2.5/site-packages/subversion.pth
    unsetenv LDFLAGS
  5. Repository setup
    • svnadmin create /path/repository_name
    • Make repository writable for webserver user: chown -R webservd:webservd /path/repository_name
    • Add block in httpd.conf to make the repository accessible via the webserver and use LDAP for authentication:
    <Location /svn/test>
      DAV svn
      SVNPath /path/repository_name
      AuthType basic
      AuthName "svn repository"
      AuthBasicProvider ldap
      AuthLDAPUrl ldap://server.domain.com/ou=people,dc=domain,dc=com
      AuthzLDAPAuthoritative off
      <LimitExcept GET PROPFIND OPTIONS REPORT>
        require valid-user
        require user johndeveloper janedeveloper
      </LimitExcept>
      SetOutputFilter DEFLATE
    </Location>

    Adjust SVNPath, AuthName and AuthLDAPUrl as needed.  The <LimitExcept> part makes the repository have public read access and only require authentication for modifications/commits.  To always require authentication, remove the start/end tags for LimitExcept but keep the require line(s).  Always use require valid-user.. optionally use the require user to limit which LDAP users are allowed.

  6. ViewVC 1.0.4
    • ./viewvc-install
      Entered /usr/local/viewvc for install path.
    • ln -s ../../viewvc/templates/docroot /usr/local/apache2/htdocs/viewvc.d
    • Edit /usr/local/viewvc/viewvc.conf (set svn repository path, docroot, etc)
    • Make sure PATH for webserver environment lists /usr/local/bin before /usr/bin so that ViewVC will find GNU diff and not the Solaris /usr/bin/diff.  Alternative: get latest development version of ViewVC from its svn repository (or a newer released version than 1.0.4 if one has come out).  Then you can configure the location of diff in the viewvc.conf file.
    • Add block in httpd.conf so ViewVC can run.  Optionally add a <Location /viewvc> block similar to the one above (minus DAV and SVNPath) to use LDAP authentication for ViewVC too.
    ScriptAlias /viewvc /usr/local/viewvc/bin/cgi/viewvc.cgi
    <Directory /usr/local/viewvc/bin/cgi>
      Order allow,deny
      Allow from all
    </Directory>

Finally start or restart apache and try out all the pieces.  Do a checkout of the repository, commit a revision and view the log in a browser.

Posted at 10:16AM May 01, 2007 by mindless in Sun  |  Comments[6]

Friday Apr 27, 2007

Browser support for XML external entities

Today I was working with writing an XML log file and finding a nice way to view it in a browser.  I threw together some xsl to display my log entries and added the stylesheet reference at the top:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="log.xsl"?>

 

Modern browsers reading this XML file will actually get the xsl file, apply the transform and show you the result (HTML in this case).. nifty!  But of course at this point I hit the fundamental problem with XML logs.. while the log is in use it is generally not well formed XML because the final closing tag is missing (neither java.util.logging nor log4j appear to do any tricks to write the closing tag and overwrite it again on each new entry).  From the log4j documentation I got the idea to try using an XML external entity to make a well formed document in another file, while the log file itself can still be simply appended.

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="log.xsl"?>
<!DOCTYPE log [
 <!ENTITY logdata SYSTEM "log.xml">
]>
<log>
 &logdata;
</log>

 

Nice and simple.. I accessed this in my browser (Firefox) and found that no log entries were shown.  It seems that Firefox recognizes the entity but does not expand it.  I tried Opera and found that it didn't process the entity at all so &logdata; was visible.  For kicks I then tried IE7.. to my surprise, it worked exactly as it should.. the external entity was processed and the XSL applied to get my nice log display.

 

 

Posted at 06:35PM Apr 27, 2007 by mindless in Sun  |  Comments[0]