In this blog I continue my discussion on how to integrate "Sun Grid Engine and MD Nastran".
I'll explain how to utilize SGE to manage disk space, memory, and MD Nastran license tokens.
In my first blog in this series I discussed 2 frequent questions that Nastran users have before submitting their jobs:
1.) "What machine(s) have enough disk space to satisfy the Nastran output database file requirements (scratch and scr300)?"
2.) "What machine(s) have enough physical memory for optimum Nastran performance?"
I also mentioned in the prior blog that I wanted to show how to solve the issue of managing Nastran jobs in an environment that has a
limited pool of license "tokens"--(utilizing a combination of MSC's "ESTIMATE" program together with SGE's ability to
track "consumable resources").
Before I begin here's some background on SGE terminology that I'll be referring to:
1.) SGE has what's called "resource attributes" that are stored in an entity called the Grid Engine "complex".
Disk space, machine memory, and license tokens are examples of resource attributes.
2.) A resource attribute can be classified/defined as a "consumable resource"--when this is the case SGE will
do the bookkeeping necessary to track the availability of the resource (increasing or decreasing the
availability based on usage). Note that this availability can be monitored/modified through two different
methods: (1) by telling SGE how much of the resource you'll be using on the SGE job submittal command
(e.g., -l disk_space=10G), and/or (2) by using a "load sensor" script that dynamically determines the availability
of the "consumable resource"--a script that runs in the background and periodically determines the availability
of a particular resource.
The rest of this blog will be separated into 3 sections:
Section 1: "How to manage disk space"
Section 2: "How to manage Memory"
Section 3: "How to manage license tokens"
-->Section 1: "How to manage disk space"
Step #1: Create a "load sensor" script that does the following: (1) dynamically determines disk space on the filesystem of interest to you,
and (2) sets the corresponding "resource attribute" to the available disk space (in this case I created a "consumable resource" called "export_size").
[In Step 2 below I show how I added this "export_size" consumable resource to the SGE environment.]
In this example, I decided to use "/export" as the filesystem of interest--to change this to your filesystem just change the line [FS="/export"].
Here's the script (disk_space.sh) that I used to determine free disk space on /export:
tm19-231:$PWD#cat disk_space.sh
#!/bin/sh
FS="/export"
export FS
myhost=`uname -n`
ende=false
while [ $ende = false ]; do
# ----------------------------------------
# wait for an input
#
read input
result=$?
if [ $result != 0 ]; then
ende=true
break
fi
if [ "$input" = "quit" ]; then
ende=true
break
fi
#set export_host to free space on /export
dfutput="`df -kh $FS | tail -1`"
diskfree=`echo $dfutput | awk '{ print $4}'`
echo begin
echo "$myhost:export_size:${diskfree}"
echo end
done
# we never get here
exit 0
## end of disk_space.sh script
Step #2: Create a consumable resource called "export_size" using the "qconf -mc" edit command:
tm19-231:/dpl/sge.sc08#qconf -mc
"/var/tmp/2481-9OzK0o" 55 lines, 4571 characters
#name shortcut type relop requestable consumable default urgency
#----------------------------------------------------------------------------------------
...
export_size e_size MEMORY <= YES YES 0 0
...
# >#< starts a comment but comments are not saved across edits --------
Step #3: Point to the above disk_space.sh script (Step #1) that has been customized for your site's storage environment,
In my example I'm assuming all machines will use /export for their local Nastran scratch space--this
can be modified to include more than one filesystem by adding another complex attribute setting
in this script--you can also have one for each host.
Now use "aconf -mconf global" to indicate that this load_sensor should apply to all hosts.
Note that you can list (coma separated) more than one script for the load_sensor below:
# qconf -mconf global
"/var/tmp/9800-xONwHt" 50 lines, 1938 characters
execd_spool_dir /gridware/sge/default/spool
mailer /bin/mailx
xterm /usr/openwin/bin/xterm
load_sensor /gridware/sge/util/resources/loadsensors/tmpspace.sh, \
/dpl/sge.sc08/disk_space.sh
prolog none
epilog none
shell_start_mode posix_compliant
login_shells sh,ksh,csh,tcsh
min_uid 0
min_gid 0
user_lists none
xuser_lists none
projects none
xprojects none
enforce_project false
enforce_user auto
load_report_time 00:00:40
max_unheard 00:05:00
reschedule_unknown 00:00:00
loglevel log_warning
administrator_mail dale.layfield@sun.com
"/var/tmp/9800-xONwHt" 50 lines, 1938 characters
Step #3a: As an optional step you can set the consumable resource export_size value to an initial size for all hosts
You can use "qconf -me global" to modify/edit the exec host (global) setting to "export_size=500G.
To verify this setting use the following "show" command:
tm19-231:/dpl/sge.sc08#qconf -se global
hostname global
load_scaling NONE
complex_values nastran_tokens=500,export_size=500G
load_values NONE
processors 0
user_lists NONE
xuser_lists NONE
projects NONE
xprojects NONE
usage_scaling NONE
report_variables NONE
tm19-231:/dpl/sge.sc08#
Step #4: Now you can verify that the "export_size" is being dynamically set to the
actual /export free space. On my machine (hostname tm19-231) there is 50G available/free in /export:
m19-231:/dpl/sge.sc08#qhost -F
HOSTNAME ARCH NCPU LOAD MEMTOT MEMUSE SWAPTO SWAPUS
-------------------------------------------------------------------------------
global - - - - - - -
gc:nastran_tokens=500.000000
gc:export_size=500.000G
tm19-231 sol-amd64 8 0.09 8.0G 1.7G 8.0G 0.0
gc:nastran_tokens=500.000000
hl:export_size=50.000G
...............
Step #5: You can now submit your Nastran jobs with a request for the amount of disk space your job needs
and SGE will only dispatch the job to hosts that meet your requirements.
For example, because the following qsub command requests 10GB of disk space "-l export_size=10G" the job will only be
dispatched to host(s) that have sufficient free space in /export.
#cat nastran_qsub.sh
qsub -pe nastran 2 -q large.q -l mem_free=6G -l nastran_tokens=`token_estimate.sh` -l export_size=10G -S /bin/ksh nastran_sge.sh
-->Section II: "How to manage Memory"
Making sure that your job only runs on machine(s) with sufficient physical memory can be accomplished by using the built-in "consumable attribute" "mem_free". As shown in the above example you would specify how much memory you need using
"-l mem_free=nG"--your job would then be dispatched to only those machine(s) that meet this request.
-->Section III: "How to manage license tokens"
As I discussed in a prior blog I wanted to find a way avoid the scenario where a Nastran job can
stop/"time out" partially through the analysis because there are no more Nastran license tokens left in
the "token" pool. This scenario can occur if the customer site has a limited license
"token" pool. The problem then arises when two or more Nastran jobs are running.
First, the jobs request an initial set of license tokens to get started.
Then later during the analysis these same job(s) may need more tokens because of some additional
Nastran "feature" that gets invoked. At this point there may not be enough "tokens" left in the license pool,
resulting in the job(s) stopping/"timing out" for lack of licenses tokens.
A solution for this issue is by using MSC Software's "ESTIMATE" program. The ESTIMATE program does a quick scan
and analysis of the Nastran input file to determine its resource requirements (disk, memory, and license tokens). This
information can then be used along with a "consumable resource" to ensure that sufficient license tokens are
available. You can read more about the ESTIMATE program at:http://www.mscsoftware.com/support/prod_support/mdnastran/cog.pdf
Here's what I did to configure SGE to work with MSC's ESTIMATE program:
Step #1: Create a script (estimate.sh) that will call MSC's "ESTIMATE" program.
tm19-231:/dpl/sge.sc08/dmp_jobs#cat estimate.sh
#
MD2008=/msc/nastran/bin/md2008
#
$MD2008 estimate dmp_101c.dat report=keyword out=estimate.out
The output from the above estimate.sh script (with this particular Nastran input file) is:
tm19-231:/dpl/sge.sc08/dmp_jobs#estimate.sh
ESTIMATE - (Version 2008.0 Jun 6 2008 12:16:44)
======================
Licensing Information:
======================
Features Required:
NASTRAN
Token counts for each feature may be adjusted by modifying "/msc/nastran/md2008/solaris/feature.lis"
Feature Tokens
NASTRAN 250
Total: 250
WORDSIZE=32
K=1024
BUFFSIZE=8193
SOL=101
SE=No
SOLVER=Direct
NGRID=3811
NDOF=10845
N0D=202
N1D=0
N2D=0
N3D=575
MEMORY=128.0MB
DISK=65.4MB
DBALL=33.6MB
SCRATCH=16.8MB
SCR300=11.8MB
SDBALL=7813MB
SSCR=7813MB
ERRORS=1
JID=./dmp_101c.dat
tm19-231:/dpl/sge.sc08/dmp_jobs#
Step #2: Now parse the above "ESTIMATE" output and return the number of required license tokens.
tm19-231:/dpl/sge.sc08/dmp_jobs#cat token_estimate.sh
#
estimate.sh > estimate.out 2> /dev/null
nawk '/^ *Total/ {print $2}' < estimate.out
#
... token_estimate.sh returns the total number of Nastran license tokens required for this job-->250:
tm19-231:/dpl/sge.sc08/dmp_jobs#token_estimate.sh
250
Step #3: Create a "consumable resource" for Nastran tokens.
tm19-231:/dpl/sge.sc08#qconf -mc
...
nastran_tokens nt INT <= FORCED YES 0 0
,,,
Step #4: Set the consumable resource "nastran_tokens" to the total amount available in the
token pool (and make this available to all hosts and queues (i.e., "global").
For example, I added "nastran_tokens=500" to the exec host (global) setting using SGE edit
command "qconf -me global":
To verify this setting use the following "show" command:
tm19-231:/dpl/sge.sc08#qconf -se global
hostname global
load_scaling NONE
complex_values nastran_tokens=500,export_size=500G
load_values NONE
processors 0
user_lists NONE
xuser_lists NONE
projects NONE
xprojects NONE
usage_scaling NONE
report_variables NONE
Step #5: Now you can submit your Nastran job and specify the required nastran tokens needed for your job
using "-l nastran_tokens=`token_estimate.sh` on the SGE command line.
(in this example nastran_tokens=`token_estimate.sh`will resolve to nastran_tokens=250).
So, for example, with 500 total tokens available you would be able to submit 2 of these 250
token jobs concurrently-any additional jobs would not be dispatched until one of these
jobs completed and released its license tokens.
Here's an example taken from my earlier blog showing the use of the nastran_tokens attribute for a distributed memory parallel (DMP) Nastran job:
#cat nastran_qsub.sh
qsub -pe nastran 2 -q large.q -l mem_free=6G -l nastran_tokens=`token_estimate.sh` -l export_size=10G
-S /bin/ksh nastran_sge.sh
In my next blog in this series I'll show how I configured the various queues and parallel environment for MD Nastran.
