Melvin Koh's Weblog
I'm just a contractor
I'm just a contractor
Tuesday July 04, 2006
Manage complex using hostgroups
A customer of mine are complaining that managing the host complexes in their cluster is extremely tedious, as they have ~200 exec hosts so they need to modify each of the hosts. Since their hosts are grouped in hostgroups, why not use them to ease the management.
Eg. setcomplex @hostgroupA mycomplex=5
will automatically set the complexes of each of the host listed in @hostgroupA. I heck up the simple script, written in perl:
#!/usr/bin/perl
@lines = `qconf -shgrp $ARGV[0] 2>&1`;
shift(@lines);
$string = join("", @lines);
$string =~ s/\\//;
@hosts = split(" " , $string);
shift(@hosts);
foreach(@hosts) {
`qconf -mattr exechost complex_values $ARGV[1] $_`;
}
Of course, we don't have to stop here. We can extend the script to perform many other host specific management. The flexibility of hostgroup allows us to define many hostgroups for many purposes.
Posted by melvin ( Jul 04 2006, 05:49:13 PM SGT ) Permalink Comments [0]
Thursday June 29, 2006
GT4 and Grid Engine Integration Package
If you're keen on using Grid Engine with Globus Toolkit 4, you need to know that the GT2 integration package will only work with the non webservice version of GRAM (pre-wsgram). If you need to use both, you need to install the latest GT4 package from the LeSC website, and you will encounter several issues.
1. You will encounter an exception when starting up the container:
"Script Execution error when executing shell /opt/globus4/libexec/globus-scheduler-provider-sge
java.io.IOException: java.io.IOException:
/opt/globus4/libexec/globus-scheduler-provider-sge: not found ......."
This is due to Globus MDS (if installed) unable to execute the GE info provider which is not provided by the integration package. It is safe to ignore this, or remove the entry from the MDS IndexService config.
2. While using globusrun-ws for WSGRAM will work, using globus-job-run will not. You need to add a the following lines to the beginning of the GE jobmanager (sge.pm file).
$ENV{"SGE_ROOT"} = $SGE_ROOT;
$ENV{"SGE_CELL"} = $SGE_CELL;
$ENV{"SGE_QMASTER_PORT"} = "536";
$ENV{"SGE_ARCH"} = "lx26-x86";
Also make sure that you've set "reporting=true" and "joblog=true" in the GE global configuration (qconf -mconf).
Posted by melvin ( Jun 29 2006, 02:31:17 PM SGT ) Permalink Comments [0]
Qmaster Monitoring
A very detailed monitoring of the qmaster, described by Stephen, will be useful for performance tuning.
Qmaster Monitoring
Posted by melvin ( Jun 29 2006, 10:36:52 AM SGT ) Permalink Comments [1]
Thursday June 22, 2006
APSTC new project websites
We have created a new project websites for our two research projects
under the Sun Grid Developer Network. For those who are still unaware,
the Sun Grid Developer Network is a project management website for
hosting all the cool projects that are related to the Sun Grid.
Our two project websites:
Grid Market Project
Grid Search Project
Visits the URL and take a look at what we're doing. :-)
Monday June 12, 2006
Avoid overscription for overlapped queues
It is common to have multiple queues for different purposes when designing the N1GE cluster. A large cluster that I designed for AIST, the F32 cluster, uses 4 specific queues and 1 general queue (all.q). The 4 queues have different ACLs for different groups of users, and all.q overlaps these queues. Since each hosts has 2 processors, the queues are configured to 2 slots per host, thus it is possible that there may be 4 jobs running in a single host (2 in all.q + 2 in specific queue). Here is the tips on how to prevent overscriptions in the overlapping queues.
The trick is to assign the slot complex and set its value to the number of processors it has. E.g.
qconf -me <exec_host>
..
complex_values slots=<number_of_cpus_or_slots>
...
Now the total number of jobs across the queues running on this host will not be more than the value assigned.
Posted by melvin ( Jun 12 2006, 11:32:33 AM SGT ) Permalink Comments [1]
Monday June 05, 2006
Failover using Shadow Host
A) Pre-Environmental setup for N1GE6
1. Copy all the necessary N1GE binaries files onto the system, unzip them and put them together (eg. /opt/n1ge6u8 = $SGE_ROOT)2. Ensure all the services and configuration are setup before the actual N1GE installation (services to be available on boot up)
i) Ensure that NFS servers, NFS clients are configured correctly
ii) Ensure that the required users are created, sgeadmin, normal users and are able to write to their own directory
iii) Ensure hostname of all machines are in the /etc/hosts with the appropriate IP if they are not in the DNS
iv) Ensure the port numbers for N1GE qmaster and execution daemons are added in the /etc/services (sge_qmaster 536/tcp, sge_execd 537/tcp)
v) Ensure RPC services (server and client) setup correctly (eg. rpcinfo -p, /sbin/service portmap status, /sbin/service nfs start).
B) Installation of Berkeley DB Spooling Server
1. Run the './inst_sge -db' command on the server that you have assigned as the RPC spooling server. Note that the DB spooling server must not be the qmaster server. Use default option for all and write down the value of these two fields after installation:
- Spooling server name
- DB spooling directory
2. Verify that:
- the sgebdb startup script /etc/rc.d/init.d/sgebdb is created
- the sgebdb daemon is running at the spooling DB server "ps -ef | grep sge"
C) Installation of Qmaster, Execution Host
1. Install the qmaster, invoke ./install_qmaster
i) Select the Berkeley DB option
ii) Choose “Y” when you are prompt to use the DB spooling server
iii) Specify the spooling server name and DB spooling directory when prompt about information
2. Verify that qmaster is installed successfully by typing the command “ps -ef | grep sge” and checking that the sge_qmaster and sge_schedd is running
D) Installation of Shadow Host
1. Type ./inst_sge -sm2. Verify that:
- the sge_shadowd daemon is running ( ps –ef | grep sge )
- there is an entry in the $SGE_ROOT/$SGE_CELL/common/shadow_masters file
E) Important Environment Variable
To change the time interval that the shadow host will take over after the master host is down, set the follow environment variables:
SGE_CHECK_INTERVAL – controls the interval in which the sge_shadowd checks the hearbeat file (60 seconds by default)
SGE_GET_ACTIVE_INTERVAL – controls the interval which a sge_shadowd instance tries to take over when the hearbeat file has not changed
SGE_DELAY_TIME – controls the interval in which sge_shadowd pauses if a takeover bid fails. used only when there are more than one shadow hosts
F) Verfication of Failover
To verify that the shadow host setup is correct, we need to simulate that a qmaster failure so that the shadow daemon will be activated. Note: A common mistake in simulating the failure is by stopping the qmaster daemon using "sgemaster stop" or even with "kill
i)Verify that the shadow daemon is running on the shadow host ( ps
ii) Kill the qmaster (kill
iii) Wait for the interval specified for the shadow host to takeover (default is about 10mins). iv) Verify that the qmaster and scheduler daemons are started (ps
v) The handover messages are logged in the follow files under $SGE_ROOT/$SGE_CELL/spool/qmaster directory
Sunday February 13, 2005
Sun Opening Global Grid
For those who still haven't heard, Sun has announced that it will be launching pay-as-you-use service - $1 per hour of CPU usage. Although Sun has termed it as "Grid" computing, but IMHO I think that it's more Utility computing than Grid, but I'm just being picky. Other than compute cycles, Sun will also be offering storage, development tools and the Java Enterprise System suites.
Read more about it here: http://www.thechannelinsider.com/article2/0,1759,1758805,00.asp
I'm extremely curious as to what software infrastructure is employed on these "Grid" systems, as the essence of Grid computing is about virtualization of resources. How about issues of usage accounting, security, privacy and repudiation?
Anyway, enough about that. There is another interesting article on the interview of Wolfgang Gentzsch, who was the lead in our N1 Grid Engine team, but has left Sun since a year or so ago. In the article, Wolfgang gave a very accurate description of the 4 types of Grid application in a compute Grid.
The article can be found here: http://news.zdnet.com/2100-9584_22-5572380.html
Posted by melvin ( Feb 13 2005, 01:46:10 PM SGT ) Permalink Comments [0]
Monday November 01, 2004
First Web Log
Hello all, this is my first log, so I guess I'll first do some introduction. My name is Melvin Koh, from the tiny island call Singapore. I'm currently part of the team call Asia Pacific Science and Technology Center, or APSTC for short. So what does APSTC do, you wonder? Well, checkout our website and find out!
APSTC Website
My role in the team is basically focus on research in the Grid Computing area. I believe many of you will claim that you know what is the Grid about, how how many REALLY understand what is Grid. Many people have misconception about what Grid can or cannot do. The comparison of Grid Computing to the power grid, where compute cycles can be drawn like electricity from a power socket does not help. Yes, maybe the power grid concept will be the ultimate vision, the Nirvana of Grid Research, but in reality, we are far from achieving that. In fact, lightyears away.
There is still a lot of problems to be solve, which makes the research so challenging. With many of the current open source/specification effort, most of which Sun is involved in, I believe that we are taking steps, albeit small ones, in the correct direction.
Posted by melvin ( Nov 01 2004, 02:53:02 PM SGT ) Permalink Comments [3]
This is a personal weblog, I do not speak for my employer.
