The life of a CA Copenhagen Campus Connection

Late 2008 I succeeded in persuading our database teacher to try out Glassfish for a new course on Web and Database programming for first year students. So before I knew it I was responsible for setting up a central Glassfish server for around 120 freshman programmers - UGH!

So how to best do this? This course has weekly lab-exercises and the teachers assistant need to grade them, so a distributed solution with a virtualbox image including netbeans and glassfish was not really the way forward - we needed a central server that the IT-dept. could maintain.

At DIKU we primarily use Gentoo Linux for our server park, so a standard HP DL360 G5 rack server was bought and a 64 bit version of Gentoo was installed from our customized Gentoo installation environment with Cfengine on top. Fortunately somebody had already made a Gentoo e-build package for Glassfish v2UR2, which easily could be updated. And within the hour I had a Glassfish server up and running - then the fun began  - it needed configuration.

Our education environment is based on Sunray terminals, but uses Linux based application hosts. Our app-hosts had Eclipse installed, so at first I thought that Eclipse could be used to interact with the remote Glassfish-server, but NONONO! Our Eclipse was the basic Eclipse installation, which didn't support Java EE applications, so I tried installing the Eclipse Enterprise Edition which has a nice Glassfish plugin - but alas - this plugin only support a local installation of Glassfish and couldn't connect to a remote server, so I turned to Netbeans 6.5.

The full edition of Netbeans 6.5 did the trick, but STILL required a local installation of Glassfish for the plugin to connect to the remote host. OK - so far so good. Now the students should be able to connect to the server from within Netbeans and deploy their web-apps.

Then for the configuration of the server: 120 freshman programmers, which need to be able to run their web-apps without crashing the server or overwriting each others programs. I turned to the Sun Java System Application server 9.1 admin guide (which is Glassfish).

Glassfish has support for virtual hosts, so I was wondering whether just setting up 120 virtual hosts would do the trick. Unfortunately the Netbeans Glassfish plugin, does NOT (currently) support deployment to a specific virtual host, so each student had to be given his/her own Glassfish domain which include two dedicated ports - one for the admin interface and one to view the web-apps.

I ended up configuring each domain with the following setup:

I started /opt/glassfish/bin/asadmin in interactive mode and typed:

create-domain --user di080156 --adminport 11156 --instanceport 12156 di080156

Our student accounts are named di080XXX for the student year 2008/2009 where XXX is an increasing number. The UNIX UID for these accounts are always 11XXX where XXX is the same number as in the account name. So I chose that the adminport number should be the same as their UNIX UID, which can easily be looked up in the system by using the command

purps@ask ~/$ id di080156
uid=11156(di080156) gid=10000(del1user) groups=10000(del1user)

The instanceport number is the UNIX UID + 1000

The name of the domain is the same as the student account number, so when I do a 'list-domains' from asadmin, it will be clearly visible which domains might need to get started.

When creating a new domain an admin-user and an admin-password need to be set. The admin username will be the student account number, while the password will be db + the students birth date (DDMMYY) which I could easily pull from our central postgreSQL server.

Now it will be relatively easy for each student to remember the most important data (the servername, the admin-port, the username and the password) to connect to the Glassfish server through Netbeans:

Log on to DIKUs system from a sunray terminal
open a xterm with 'besthost bach'
open /opt/netbeans-6.5/bin/netbeans &
(wait while Netbeans starts)

Choose File | New Project...
Choose Java EE and the type of project
Choose a name and place for the project

Click on button Target Server - choose Add...

Server: Glassfish V2 - click Next
Platform Location: /opt/glassfish-v2ur2

Choose Register Remote Domain og choose Next
Host: glassfish.diku.dk
Port: enter your UNIX UID for your student account
Choose Next
Type Admin username: your di080xxx account name
Type Admin password: dbDDMMYY - where DDMMYY is your birthdate
Choose Finish

Tuning the Glassfish-server

Unfortunately starting 120 domains requires A HELL OF A LOT OF RAM. Each domain is by default started with 512 MB. Our server "only" had 20 GB RAM and 140 GB of swap, so I needed to decrease the RAM usage of each domains JVM. I found the document "Tuning your Glassfish - Performance Tips" presentation and went to /opt/glassfish/domains and for each domain I changed domain.xml from

<jvm-options>-Xmx512m</jvm-options>

 to

<jvm-options>-Xms192m</jvm-options>
<jvm-options>-Xmx192m</jvm-options>

Afterwards each domain can be started with the command 'start-domain di080XXX' from asadmin, but after a while you don't want to start 120 domains manually, so a collegue of mine wrote a small script to start/stop all domains sequentially.

#!/bin/bash

ASADMIN=/opt/glassfish/bin/asadmin
DOMAINDIR=/opt/glassfish/domains

start_all(){
        for DOMAIN in ${DOMAINDIR}/*; do
                ${ASADMIN} start-domain ${DOMAIN##*/}
        done
}

stop_all(){
        for DOMAIN in ${DOMAINDIR}/*; do
                ${ASADMIN} stop-domain ${DOMAIN##*/}
        done
}

usage(){
        cat <<-HERE
        Usage: ${0} [command]

        where command is one of

          start     start all domains
          stop      start all domains
          restart   first stop and then start all domains

        HERE
}

case ${1} in
        start)
                start_all
                ;;
        stop)
                stop_all
                ;;
        restart)
                stop_all
                start_all
                ;;
        *)
                usage
                ;;
esac

Now I am anxiously awaiting the students use of the server. The RAM usage is horrifying and it is swapping like crazy, and I am unsure if I should stuff even more physical RAM in the machine or try turning down the JVM RAM usage on each domain even more. Here is a snapshot of top:

top - 12:23:00 up 22:22,  2 users,  load average: 1.55, 4.88, 4.06
Tasks: 212 total,   1 running, 211 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.2%us,  0.3%sy,  0.0%ni, 79.3%id, 19.1%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  20600040k total, 20497768k used,   102272k free,     6736k buffers
Swap: 143338536k total, 18882040k used, 124456496k free,    18708k cached
Which user (blank for all):
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                
23825 root      20   0  708m 166m 3820 S    1  0.8   1:49.27 java                    
 7208 root      20   0 10560 1288  876 R    1  0.0   0:00.37 top                     
31804 root      20   0  784m 167m 3820 S    1  0.8   1:50.51 java                    
  242 root      15  -5     0    0    0 D    0  0.0   4:08.55 kswapd0                 
  394 root      20   0  706m 167m 3848 S    0  0.8   1:47.89 java                    
 1537 root      20   0  787m 168m 3820 S    0  0.8   1:48.49 java