Monday May 19, 2008
Monday May 19, 2008
Here is the shortest example I could come up with to enter parameters on the command line (reader classes will be treated as just "it works this way" for now):
import java.io.*;
public class ReadString {
public static void main (String[] args) throws Exception {
System.out.print("Enter your name: ");
// open up standard input
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
String name = br.readLine();
System.out.println("Thanks: " + name);
}
}
Sun Inventory
I really doubt anyone actually reads this blog. But just in case, don't forget to register your systems at:
https://inventory.sun.com/inventory/
Layout management, how simple can it be?
Trying to teach Java to a 14 year old, and after working on some problems such
as: Solving a quadratric equation, checking if a number is prime, we have so far
changed the data inside the program, recompiled it and ran again....
We are reaching the point where we want to be able to enter the data in a fashionable way! Meaning time to address the "GUI" problem. While it is straightforward to create a frame and add components to it, it is tricky to layout the components inside the frame.
Here are some examples using the layout manager as little as possible (I am keeping this subject for way later...).
In the first example, we simply add some components to the panel used by the frame (FlowLayout is used by default)
In the second example, we use the GridLayout class to order the components.
In the third example, the components in a row are first added to a panel which is then added to the overall panel.
We are reaching the point where we want to be able to enter the data in a fashionable way! Meaning time to address the "GUI" problem. While it is straightforward to create a frame and add components to it, it is tricky to layout the components inside the frame.
Here are some examples using the layout manager as little as possible (I am keeping this subject for way later...).
In the first example, we simply add some components to the panel used by the frame (FlowLayout is used by default)
In the second example, we use the GridLayout class to order the components.
In the third example, the components in a row are first added to a panel which is then added to the overall panel.
import javax.swing.*;
public class Test {
public static void main(String[] args) {
JFrame frame = new JFrame();
JPanel panel = new JPanel();
JLabel l = new JLabel("label:");
JTextField f = new JTextField(10);
JButton b = new JButton("Calc");
panel.add(l);
panel.add(f);
panel.add(b);
frame.getContentPane().add(panel);
frame.setSize(300, 300);
frame.setVisible(true);
}
}
import java.awt.*;
import javax.swing.*;
public class Test2 {
public static void main(String[] args) {
JFrame frame = new JFrame();
JPanel panel = new JPanel();
panel.setLayout(new GridLayout(4,2));
JLabel l1 = new JLabel("label:");
JTextField f1 = new JTextField(10);
panel.add(l1);
panel.add(f1);
JLabel l2 = new JLabel("label:");
JTextField f2 = new JTextField(10);
panel.add(l2);
panel.add(f2);
JLabel l3 = new JLabel("label:");
JTextField f3 = new JTextField(10);
panel.add(l3);
panel.add(f3);
JButton b = new JButton("Calc");
panel.add(b);
frame.getContentPane().add(panel);
frame.setSize(300, 300);
frame.setVisible(true);
}
}
import java.awt.*;
import javax.swing.*;
public class Test3 {
public static void main(String[] args) {
JFrame frame = new JFrame();
JPanel panel = new JPanel();
panel.setLayout(new GridLayout(4,1));
JLabel l1 = new JLabel("label:");
JTextField f1 = new JTextField(10);
JPanel panel1 = new JPanel();
panel1.add(l1);
panel1.add(f1);
panel.add(panel1);
JLabel l2 = new JLabel("label:");
JTextField f2 = new JTextField(10);
JPanel panel2 = new JPanel();
panel2.add(l2);
panel2.add(f2);
panel.add(panel2);
JLabel l3 = new JLabel("label:");
JTextField f3 = new JTextField(10);
JPanel panel3 = new JPanel();
panel3.add(l3);
panel3.add(f3);
panel.add(panel3);
JButton b = new JButton("Calc");
JPanel panel4 = new JPanel();
panel4.add(b);
panel.add(panel4);
frame.getContentPane().add(panel);
frame.setSize(300, 300);
frame.setVisible(true);
}
}
Wednesday Jun 20, 2007
I moved to a different project couple of months ago.
We just delivered the first version of our product.
Here is an overview,
here is the link to it,
and here you will find what service tags
are about.
I keep in touch with some of the developers on the Sun grid team, here are some features recently
delivered by this team (in case someone actually reads this blog and did not see them somewhere else and since it is still called 'The Grid'!):
- International Access
- The application Catalog
- and something which was heavily requested at SC'06: The API to access Sun Grid without using the Web interface
CS 101
I have been looking recently for resources to start teaching my son who is in middle
school some introduction to computer programming.
I am not too sure where to start? Should I look into some intro to computer architecture first,
some Operating Systems concept...
My current thinking is to focus on programs, what is a computer program and how to write one
while introducing the above concepts in parallel. I will probably use Java as I think it is
much easier to learn than C for example (Yes, I know of all these programmers who said how
hard it is to switch to Java and OO concepts!).
Here are some web sites I found and plan to use:
Friday Dec 15, 2006
A quick analysis of the code is:
Taking a closer look, here is what is going on:
The moral of this story is that as everyone knows it is way too easy to make this sort of mistakes. To make concurrent programming little bit easier, Java 1.5 introduced the java.util.concurrent package and several classes to ease the development of multi-threads applications. Looking back at our problem, we want to ensure the run method is executed first, then the get method. A possible solution (I just started learning about these new classes!) is to use the new CountDownLatch class:
Also, since we are not using the wait and notifyAll calls directly any longer it is not necessary to synchronized the run and get methods. The implementation of the class becomes:
- Thread 1 and Thread 2 start. They both use a Runnable object which initialize the wait flag to true.
- The start method calls the run methods which sleeps for a while, then switch the wait flag to false and notify any possible blocking threads.
- In the mean time, the main thread calls the get method which wait to be notified, then check if the wait flag is true.
Taking a closer look, here is what is going on:
- The run method notifies other threads before the main thread is even waiting for the notification! the main thread then waits for a signal which already happened!
- An easy fix is of course to check if the wait flag has been turned on to false before waiting:
if (!(wait)) break; wait();
The moral of this story is that as everyone knows it is way too easy to make this sort of mistakes. To make concurrent programming little bit easier, Java 1.5 introduced the java.util.concurrent package and several classes to ease the development of multi-threads applications. Looking back at our problem, we want to ensure the run method is executed first, then the get method. A possible solution (I just started learning about these new classes!) is to use the new CountDownLatch class:
- A CountDownLatch is initialized with a given count. The await methods block until the current count reaches zero due to invocations of the countDown() method.
Also, since we are not using the wait and notifyAll calls directly any longer it is not necessary to synchronized the run and get methods. The implementation of the class becomes:
class CRun1 implements Runnable {
String s;
CountDownLatch c = new CountDownLatch(1);
CRun1(String s) {
this.s = s;
}
public void run() {
System.out.println("Lrun " + s);
try {
Thread.sleep(4000);
} catch (InterruptedException ex) {
} finally {
}
System.out.println("count down - notify for run " + s);
c.countDown();
}
public void get() {
while (true) {
try {
c.await();
break;
} catch (InterruptedException ex) {
}
}
System.out.println("recv notify...");
}
}
A Thread issue...
What's wrong with this program? It did not run as expected, JConsole did not detect a deadlock, and FindBugs did not detect a problem (it did report not to use sleep with a lock held)
class Run1 implements Runnable {
String s;
boolean wait = true;
Run1(String s) {
this.s = s;
}
public synchronized void run() {
System.out.println("run " + s);
try {
Thread.sleep(4000);
} catch (InterruptedException ex) {
}
wait = false;
System.out.println("notify for run " + s);
notifyAll();
}
public synchronized void get() {
while (true) {
try {
System.out.println("wait...");
wait();
if (!(wait)) break;
System.out.println("recv notify...");
} catch (InterruptedException ex) {
}
}
}
}
public class TestThread {
public synchronized void testIt() throws Exception {
Run1 run1 = new Run1("1");
Run1 run2 = new Run1("2");
Thread t1 = new Thread(run1);
t1.start();
Thread t2 = new Thread(run2);
t2.start();
run1.get();
run2.get();
}
public static void main(String[] args) throws Exception {
TestThread t = new TestThread();
t.testIt();
}
}
Wednesday Dec 06, 2006
Two interfaces are of utter importance to distributed programming in Java:
The Remote interface specifies that an object implementing this interface is an object in a different process (but of course it is technically possible to be in the same process), most likely on a different machine.
For example:
Remote objects must implements this interface and extends the UnicastRemoteObject class:
All methods in a class implementing the Remote interface must throw the RemoteException (This exception will occurs if the network is down).
Notice the signature of the above method:
Suppose now that you want to group the parameters to the function to integrate in an object:
and pass an object of this class such as:
to identicate and marshall the object over the wire (e.g, IP) it is necessary to have the object implement the Serializable interface:
So, while you will typically pass a Serializable object in a remote call, is this possible to pass a Remote object? Yes! This is typically used when the server needs to call back the client such as:
Suppose the computation takes a while. The above method returns right away, but calls back the client when the result is ready.
- the Remote interface
- the Serializable interface
The Remote interface specifies that an object implementing this interface is an object in a different process (but of course it is technically possible to be in the same process), most likely on a different machine.
For example:
public interface ComputeServer extends Remote {
public double integrate(String function, double min, double max) throws RemoteException;
}
Remote objects must implements this interface and extends the UnicastRemoteObject class:
public class ComputeServerImpl extends UnicastRemoteObject implements ComputeServer {
public double integrate(String function, double min, double max) throws RemoteException {
...
}
}
All methods in a class implementing the Remote interface must throw the RemoteException (This exception will occurs if the network is down).
Notice the signature of the above method:
public double integrate(String function, double min, double max) throws RemoteException {
Suppose now that you want to group the parameters to the function to integrate in an object:
class FunctionToIntegrate {
String function;
int min;
int max;
}
and pass an object of this class such as:
public double integrate(FunctionToIntegrate function);
to identicate and marshall the object over the wire (e.g, IP) it is necessary to have the object implement the Serializable interface:
class FunctionToIntegrate extends Serializable {
...
}
So, while you will typically pass a Serializable object in a remote call, is this possible to pass a Remote object? Yes! This is typically used when the server needs to call back the client such as:
public class ComputeServerImpl extends UnicastRemoteObject implements ComputeServer {
public void integrate(String function, Remote callback) throws RemoteException {
...
}
}
Suppose the computation takes a while. The above method returns right away, but calls back the client when the result is ready.
Friday Nov 10, 2006
The largest Supercomputer conference in the world, SC06, is next week and I will be there presenting Sun Grid. Sun will be presenting a bunch of other products in the HPC area.
I am one of the experts! (Well, my boss thinks I am!). Please, stop at our booth (#605), you will get some free hours on Sun Grid. And, if I don't know the answers to some of your questions, I will get back to you as soon as I can!
I am one of the experts! (Well, my boss thinks I am!). Please, stop at our booth (#605), you will get some free hours on Sun Grid. And, if I don't know the answers to some of your questions, I will get back to you as soon as I can!
DRMAA
I initially gave examples on how to distribute an application over the Grid using the Sun Grid Engine commands (e.g., qsub).
An other option is to use the Distributed Resource Management Application API (DRMAA) which provide options to submit jobs directly from within a program.
A sample code looks like in Java:
// Submit jobs to run
for (int i = 0; i < 50; i++) {
System.out.println("Creating Remote Command: " + i);
jt[i] = session.createJobTemplate();
jt[i].setRemoteCommand(exec);
ids[i] = session.runJob(jt[i]);
}
A full example is provided on the Developer Network site by Fay Salwen, one of our Grid experts.
Thursday Nov 09, 2006
These steps are used to connect to a GridFTP server and set up the security certificates appropriately; they are provided by Gabriele Carcassi, a senior engineer here at Sun.
We will be using the Commodity Grid (CoG) Kits on the client.
On your machine, as yourself:
On myserver, as root:
On the client machine, as yourself:
On the server, edit the /etc/grid-security/grid-mapfile to add an entry such as:
On the client, try the file-transfer script:
Et voila...
We will be using the Commodity Grid (CoG) Kits on the client.
On your machine, as yourself:
> ./cog-4_1_4/bin/grid-cert-request -cn "Pascal Ledru" >scp .globus/usercert_request.pem root@myserver.central.sun.com:usercert_request.ledru.pem
On myserver, as root:
> /usr/local/globus-4.0.3/bin/grid-ca-sign -in usercert_request.ledru.pem -out usercert.ledru.pem
On the client machine, as yourself:
> scp root@myserver.central.sun.com:usercert.ledru.pem .globus/usercert.pem > ./install/cog-4_1_4/bin/grid-proxy-init
On the server, edit the /etc/grid-security/grid-mapfile to add an entry such as:
"/O=Grid/OU=GlobusTest/OU=simpleCA-myserver/CN=Pascal Ledru" globus
On the client, try the file-transfer script:
./cog-file-transfer -s file:///home/pl108086/test.txt -d gridftp://myserver.central.sun.com:2811/tmp/TEST.txt
Et voila...
Globus on Solaris 10 x86
I just built the Globus Toolkit on Solaris 10 / x86 (AMD Opteron). Here are the steps I went thru
if these can be useful to anyone:
From the Download site, pick up the source: gt4.0.3-all-source-installer.tar bundle.
Make sure you have GNU tar and GNU make (I did not have them by default and installed these in /usr/local/bin)
As root, create the directory /usr/local/globus.4.0.3
Create a globus user, as this user here are the environment variables I have:
Run:
Then:
I ran into couple of problems. Initially did not have ar and pod2man in my path. So, I corrected my PATH by adding:
Then, I ran into this problem:
due to the fact I uses a JDK1.5 compiler and enum is a reserved word used in:
Then, I waited and waited and waited.... Eventually:
Then:
I then try to run setup-simple-ca as the globus user:
Got this error:
So changed
Got this error:
From the Download site, pick up the source: gt4.0.3-all-source-installer.tar bundle.
Make sure you have GNU tar and GNU make (I did not have them by default and installed these in /usr/local/bin)
As root, create the directory /usr/local/globus.4.0.3
Create a globus user, as this user here are the environment variables I have:
> printenv HZ= SHELL=/bin/bash TERM=xterm LC_ALL=C MAIL=/var/mail/globus PATH=/usr/local/bin:/usr/bin:/usr/sfw/bin:/usr/dist/share/devpro/5.x-i386/bin:/usr/ccs/bin:/usr/dist/exe PWD=/etc/grid-security LANG=C TZ=US/Mountain SHLVL=2 HOME=/users/globus LOGNAME=globus LC_CTYPE=C _=/usr/bin/tcsh HOSTTYPE=i86pc VENDOR=sun OSTYPE=solaris MACHTYPE=i386 USER=globus GROUP=other HOST=myhost GPT_LOCATION=/usr/local/globus-4.0.3 CC=/usr/dist/share/devpro/5.x-i386/bin/cc GLOBUS_LOCATION=/usr/local/globus-4.0.3 GCC=/usr/sfw/bin/gcc
Run:
./configure --prefix=$GLOBUS_LOCATION --with-flavor=gcc32dbg
Then:
make
I ran into couple of problems. Initially did not have ar and pod2man in my path. So, I corrected my PATH by adding:
/usr/ccs/bin:/usr/dist/exe
Then, I ran into this problem:
file:/tmp/testgt/gt4.0.3-all-source-installer/source-trees/wsrf/java/core/source/build.xml:211: Compile failed; see the compiler error output for details.
due to the fact I uses a JDK1.5 compiler and enum is a reserved word used in:
import org.apache.axus.enum.Scope;I edited the build file to add the -source 1.4 option such as:
source="1.4"
Then, I waited and waited and waited.... Eventually:
echo "Your build completed successfully. Please run make install." Your build completed successfully. Please run make install.
Then:
> make install /usr/local/globus-4.0.3/sbin/gpt-postinstall running /usr/local/globus-4.0.3/setup/globus/setup-globus-common..[ Changing to /usr/local/globus-4.0.3/setup/globus ] creating globus-sh-tools-vars.sh running /usr/local/globus-4.0.3/setup/globus/setup-globus-job-manager-fork..[ Changing to /usr/local/globus-4.0.3/setup/globus ] find-fork-tools: WARNING: "Cannot locate mpiexec" find-fork-tools: WARNING: "Cannot locate mpirun" checking for mpiexec... no checking for mpirun... no find-fork-tools: creating ./config.status config.status: creating fork.pm ..Done
I then try to run setup-simple-ca as the globus user:
> $GLOBUS_LOCATION/setup/globus/setup-simple-ca
Got this error:
creating CA config package... /usr/local/globus-4.0.3/setup/globus/setup-simple-ca: GLOBUS_LOCATION=/usr/local/globus-4.0.3: is not an identifier
So changed
#! /bin/shto
#! /bin/bash
Got this error:
creating CA config package... ERROR: Unable to configure the simple_ca 9a7cd758 setup packageCorrected GPT_LOCATION such as:
setenv GPT_LOCATION /usr/local/globus-4.0.3Then ran as root:
> $GLOBUS_LOCATION/setup/globus_simple_ca_9a7cd758_setup/setup-gsi -default > $GLOBUS_LOCATION/bin/grid-cert-request -host myhost.central.sun.com
Wednesday Sep 27, 2006
Sun recently came up with a new technology to write applications for the Sun Grid Compute Utility. Under the cover it is based on Jini.
It is available for download at on the Developer Network web site.
Installation notes are also available at this site.
I installed the Netbeans module, picked on of the examples: integralpi built the project which generates a zip file: gridjob.zip in the dist directory, logged in to Sun Grid, created a resource using gridjob.zip, created a job using gridjob.zip and specifying "start" as the executable and submitted the job. The job ran for few minutes, I downloaded the result and checked the value of pi in the cs_master file:
By default, 100 tasks are used. You can see the output of each worker:
It is available for download at on the Developer Network web site.
Installation notes are also available at this site.
I installed the Netbeans module, picked on of the examples: integralpi built the project which generates a zip file: gridjob.zip in the dist directory, logged in to Sun Grid, created a resource using gridjob.zip, created a job using gridjob.zip and specifying "start" as the executable and submitted the job. The job ran for few minutes, I downloaded the result and checked the value of pi in the cs_master file:
cat cs_master.o68935.2
Sleeping for 0 seconds
Your job-array 68936.4-102:1 ("cs_ephemeralworker -ephemeral") has been submitted
Sep 27, 2006 4:43:43 PM com.sun.computeserver.master.Master run
INFO: Master starting
Sep 27, 2006 4:44:05 PM com.sun.computeserver.integralpi.PiGenerator done
INFO: Master PI:3.1415906524138111994160239945727946065207750695568908515112969308267806110
10577374635946745511251244209674355608893092721700668334960937500
Sep 27, 2006 4:44:05 PM com.sun.computeserver.master.Master doWaitForFeedback
INFO: Master exiting via WAIT
Sep 27, 2006 4:44:05 PM com.sun.computeserver.mm.MasterAspect doOutput
INFO:
Job Succeeded
Sep 27, 2006 4:44:05 PM com.sun.computeserver.mm.MasterAspect doOutput
INFO:
Master Job Start Time : 2006-09-27 16:43:43.674
Master Job End Time : 2006-09-27 16:44:05.187
Total Job Execution Time : 21.513 Seconds
Total Processed Tasks : 100
Total Failed Tasks : 0
Total Submitted Tasks : 100
Generated : 100
Fissioned : 0
Task Throughput : 4.7 Tasks per Second
Average Task Execution Time : 0.215 Seconds per Task
By default, 100 tasks are used. You can see the output of each worker:
cat cs_ephemeralworker.o68936.86 Sleeping for 8 seconds Sep 27, 2006 4:44:03 PM com.sun.computeserver.worker.WorkerWatcher init INFO: Worker active on nyc1r226cpn06.retail.nyc1.sungrid.net Sep 27, 2006 4:44:09 PM com.sun.computeserver.worker.WorkerWatcher goAway INFO: Worker exiting nyc1r226cpn06.retail.nyc1.sungrid.net
Tuesday Sep 19, 2006
Want to get an idea of the CPU used by a job? Try the prstat command:
I can see that on average 40% of the cpu was used. We can also see the other processes running:
#!/bin/ksh prstat 20 20 > prstat.out & java ExampleLooking for the Java process in this example:
7956 user0103 203M 13M cpu0 0 0 0:00:20 30% java/12 7956 user0103 203M 13M cpu1 0 0 0:00:40 42% java/12 7956 user0103 203M 13M cpu1 0 0 0:01:00 47% java/12 7956 user0103 203M 13M cpu1 0 0 0:01:19 49% java/12 7956 user0103 203M 13M cpu1 0 0 0:01:40 49% java/12
I can see that on average 40% of the cpu was used. We can also see the other processes running:
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
27796 root 13M 10M sleep 49 0 1:24:39 0.2% dsmonitor/1
9 root 9080K 8140K sleep 59 0 0:00:46 0.1% svc.configd/14
304 root 5056K 2628K sleep 59 0 0:04:17 0.0% automountd/3
7955 user0103 4408K 3992K cpu1 34 0 0:00:00 0.0% prstat/1
754 root 31M 28M sleep 59 -10 0:27:05 0.0% sge_execd/1
7950 root 4156K 1468K sleep 54 -10 0:00:00 0.0% sge_shepherd/1
7954 user0103 1152K 912K sleep 14 0 0:00:00 0.0% 66414/1
7 root 9124K 8140K sleep 59 0 0:03:03 0.0% svc.startd/14
7956 user0103 7716K 1280K cpu0 24 0 0:00:00 0.0% java/1
323 nobody 2932K 1836K sleep 59 0 0:07:32 0.0% gmond/1
143 root 4340K 3456K sleep 59 0 0:04:54 0.0% nscd/26
27785 root 3832K 1972K sleep 59 0 0:00:10 0.0% syslogd/15
282 root 1952K 1160K sleep 100 - 0:02:49 0.0% xntpd/1
150 root 2256K 1476K sleep 59 0 0:00:00 0.0% devfsadm/6
122 root 2120K 1300K sleep 59 0 0:00:00 0.0% syseventd/14
Total: 40 processes, 156 lwps, load averages: 0.01, 0.00, 0.00
Monday Sep 18, 2006
I ran a quick experiment with each node executing 20 million tests:
Note that to avoid writting a different script I use the feature to pass a parameter to a job to specify the number of clients:
1 node: Average: 3.1411545250056307 with 1 nodes. 2 nodes: Average: 3.141313165330142 with 2 nodes. 20 nodes: Average: 3.1416765781116878 with 20 nodes. 200 nodes: Average: 3.141568127864779 with 200 nodes.Actually 3.1415 is a fairly good approximation: π to one MILLION decimal places
Note that to avoid writting a different script I use the feature to pass a parameter to a job to specify the number of clients:
#!/bin/ksh
#$ -N Buffon
#$ -cwd
# Set the environment variables
if [ -f $HOME/.profile ]; then
. $HOME/.profile
fi
numClients=$1
echo "Starting the server..."
echo "Number of clients: "$numClients
svrResp=$(qsub -N server startServers.ksh)
echo svrResp is $svrResp
svrJobId=$(echo "$svrResp" | awk '{print $3}')
echo svrJobId is $svrJobId
# Wait until the server is started
status="not running"
until [ "$status" == "r" ]
do
status=$( qstat | nawk '/'$svrJobId'/ {print $5}' )
echo Server job status is $status
sleep 10
done
#Wait until the serverhost file is created
filename="$HOME/serverhost"
until test -f $filename
do
sleep 10
done
# then pull the server node name from the file
servernode=$(cat $filename)
rm $filename
echo "Server is running on" $servernode "Submitting a set of clients to the grid for remote execution..."
qsub -N clients -t 1-$numClients startClient.ksh $servernode
echo "Submitting a cleanup job that will wait until the clients are complete"
qsub -hold_jid clients cleanup.ksh $svrJobId