ISV engineering's HPC web log For HPC ISVs & OSS

In this blog I continue my discussion on how to integrate "Sun Grid Engine and MD Nastran".

I'll explain how to utilize SGE to manage disk space, memory, and MD Nastran license tokens.

In my first blog in this series I discussed 2 frequent questions that Nastran users have before submitting their jobs:

1.) "What machine(s) have enough disk space to satisfy the Nastran output database file requirements (scratch and scr300)?"

2.) "What machine(s) have enough physical memory for optimum Nastran performance?"

I also mentioned in the prior blog that I wanted to show how to solve the issue of managing Nastran jobs in an environment that has a
limited pool of license "tokens"--(utilizing a combination of MSC's "ESTIMATE" program together with SGE's ability to
track "consumable resources").

Before I begin here's some background on SGE terminology that I'll be referring to:

1.) SGE has what's called "resource attributes" that are stored in an entity called the Grid Engine "complex".
Disk space, machine memory, and license tokens are examples of resource attributes.

2.) A resource attribute can be classified/defined as a "consumable resource"--when this is the case SGE will
do the bookkeeping necessary to track the availability of the resource (increasing or decreasing the
availability based on usage). Note that this availability can be monitored/modified through two different
methods: (1) by telling SGE how much of the resource you'll be using on the SGE job submittal command
(e.g., -l disk_space=10G), and/or (2) by using a "load sensor" script that dynamically determines the availability
of the "consumable resource"--a script that runs in the background and periodically determines the availability
of a particular resource.

 

The rest of this blog will be separated into 3 sections:

Section 1: "How to manage disk space"

Section 2: "How to manage Memory"

Section 3: "How to manage license tokens"

 

---

-->Section 1: "How to manage disk space"

 

Step #1: Create a "load sensor" script that does the following: (1) dynamically determines disk space on the filesystem of interest to you,
and (2) sets the corresponding "resource attribute" to the available disk space (in this case I created a "consumable resource" called "export_size").
[In Step 2 below I show how I added this "export_size" consumable resource to the SGE environment.]

In this example, I decided to use "/export" as the filesystem of interest--to change this to your filesystem just change the line [FS="/export"].

Here's the script (disk_space.sh) that I used to determine free disk space on /export:

tm19-231:$PWD#cat disk_space.sh

#!/bin/sh

FS="/export"

export FS

myhost=`uname -n`

ende=false

while [ $ende = false ]; do

   # ----------------------------------------

   # wait for an input

   #

   read input

   result=$?

   if [ $result != 0 ]; then

      ende=true

      break

   fi

   if [ "$input" = "quit" ]; then

      ende=true

      break

   fi

   #set export_host to free space on /export

     dfutput="`df -kh $FS | tail -1`"

     diskfree=`echo $dfutput | awk '{ print $4}'`

     echo begin

     echo "$myhost:export_size:${diskfree}"

     echo end

done

# we never get here

exit 0

## end of disk_space.sh script

 

Step #2:  Create a consumable resource called "export_size" using the "qconf -mc" edit command:

tm19-231:/dpl/sge.sc08#qconf -mc

"/var/tmp/2481-9OzK0o" 55 lines, 4571 characters

#name               shortcut   type        relop requestable consumable default  urgency

#----------------------------------------------------------------------------------------

...

export_size         e_size     MEMORY      <=    YES         YES        0        0

...

# >#< starts a comment but comments are not saved across edits --------

 

Step #3:  Point to the above disk_space.sh script (Step #1) that has been customized for your site's storage environment,

In my example I'm assuming all machines will use /export for their local Nastran scratch space--this

can be modified to include more than one filesystem by adding another complex attribute setting

in this script--you can also have one for each host.

Now use "aconf -mconf global" to indicate that this load_sensor should apply to all hosts.

Note that you can list (coma separated) more than one script for the load_sensor below:

 

# qconf -mconf global

"/var/tmp/9800-xONwHt" 50 lines, 1938 characters

execd_spool_dir              /gridware/sge/default/spool

mailer                       /bin/mailx

xterm                        /usr/openwin/bin/xterm

load_sensor                  /gridware/sge/util/resources/loadsensors/tmpspace.sh, \

                             /dpl/sge.sc08/disk_space.sh

prolog                       none

epilog                       none

shell_start_mode             posix_compliant

login_shells                 sh,ksh,csh,tcsh

min_uid                      0

min_gid                      0

user_lists                   none

xuser_lists                  none

projects                     none

xprojects                    none

enforce_project              false

enforce_user                 auto

load_report_time             00:00:40

max_unheard                  00:05:00

reschedule_unknown           00:00:00

loglevel                     log_warning

administrator_mail           dale.layfield@sun.com

"/var/tmp/9800-xONwHt" 50 lines, 1938 characters

 

 

Step #3a: As an optional step you can set the consumable resource export_size value to an initial size for all hosts
You can use "qconf -me global" to modify/edit the exec host (global) setting to "export_size=500G.

To verify this setting use the following "show" command: 

tm19-231:/dpl/sge.sc08#qconf -se global

hostname              global

load_scaling          NONE

complex_values        nastran_tokens=500,export_size=500G

load_values           NONE

processors            0

user_lists            NONE

xuser_lists           NONE

projects              NONE

xprojects             NONE

usage_scaling         NONE

report_variables      NONE

tm19-231:/dpl/sge.sc08#

 

Step #4: Now you can verify that the "export_size" is being dynamically set to the

actual /export free space. On my machine (hostname tm19-231) there is 50G available/free in /export:

 

m19-231:/dpl/sge.sc08#qhost -F

HOSTNAME                ARCH         NCPU  LOAD  MEMTOT  MEMUSE  SWAPTO  SWAPUS

-------------------------------------------------------------------------------

global                  -               -     -       -       -       -       -

   gc:nastran_tokens=500.000000

   gc:export_size=500.000G

tm19-231                sol-amd64       8  0.09    8.0G    1.7G    8.0G     0.0

   gc:nastran_tokens=500.000000

   hl:export_size=50.000G

...............

Step #5: You can now submit your Nastran jobs with a request for the amount of disk space your job needs
and SGE will only dispatch the job to hosts that meet your requirements.

For example, because the following qsub command requests 10GB of disk space "-l export_size=10G" the job will only be
dispatched to host(s) that have sufficient free space in /export.

#cat nastran_qsub.sh
qsub -pe nastran 2 -q large.q  -l mem_free=6G -l nastran_tokens=`token_estimate.sh`  -l export_size=10G  -S /bin/ksh nastran_sge.sh

 

-->Section II: "How to manage Memory"

Making sure that your job only runs on machine(s) with sufficient physical memory can be accomplished by using the built-in "consumable attribute" "mem_free".  As shown in the above example you would specify how much memory you need using
"-l mem_free=nG"--your job would then be dispatched to only those machine(s) that meet this request.

 

---

-->Section III: "How to manage license tokens"

 

As I discussed in a prior blog I wanted to find a way avoid the scenario where a Nastran job can
stop/"time out" partially through the analysis because there are no more Nastran license tokens left in
the "token" pool. This scenario can occur if the customer site has a limited license
"token" pool. The problem then arises when two or more Nastran jobs are running.
First, the jobs request an initial set of license tokens to get started.
Then later during the analysis these same job(s) may need more tokens because of some additional
Nastran "feature" that gets invoked.  At this point there may not be enough "tokens" left in the license pool,
resulting in the job(s) stopping/"timing out" for lack of licenses tokens.

A solution for this issue is by using MSC Software's "ESTIMATE" program.  The ESTIMATE program does a quick scan
and analysis of the Nastran input file to determine its resource requirements (disk, memory, and license tokens).  This
information can then be used along with a "consumable resource" to ensure that sufficient license tokens are
available. You can read more about the ESTIMATE program at:http://www.mscsoftware.com/support/prod_support/mdnastran/cog.pdf

Here's what I did to configure SGE to work with MSC's ESTIMATE program:
 
Step #1:  Create a script (estimate.sh) that will call MSC's "ESTIMATE" program.

tm19-231:/dpl/sge.sc08/dmp_jobs#cat estimate.sh
#
MD2008=/msc/nastran/bin/md2008
#
$MD2008 estimate  dmp_101c.dat report=keyword out=estimate.out

The output from the above estimate.sh script (with this particular Nastran input file) is:

tm19-231:/dpl/sge.sc08/dmp_jobs#estimate.sh
ESTIMATE - (Version 2008.0 Jun  6 2008 12:16:44)

 ======================
 Licensing Information:
 ======================
 Features Required:
NASTRAN
Token counts for each feature may be adjusted by modifying "/msc/nastran/md2008/solaris/feature.lis"
 Feature         Tokens
 NASTRAN            250
 Total:             250

WORDSIZE=32
K=1024
BUFFSIZE=8193
SOL=101
SE=No
SOLVER=Direct
NGRID=3811
NDOF=10845
N0D=202
N1D=0
N2D=0
N3D=575
MEMORY=128.0MB
DISK=65.4MB
DBALL=33.6MB
SCRATCH=16.8MB
SCR300=11.8MB
SDBALL=7813MB
SSCR=7813MB
ERRORS=1
JID=./dmp_101c.dat
tm19-231:/dpl/sge.sc08/dmp_jobs#

Step #2: Now parse the above "ESTIMATE" output and return the number of required license tokens.

tm19-231:/dpl/sge.sc08/dmp_jobs#cat token_estimate.sh
#
estimate.sh > estimate.out 2> /dev/null
nawk '/^ *Total/ {print $2}' < estimate.out
#

... token_estimate.sh returns the total number of Nastran license tokens required for this job-->250:
tm19-231:/dpl/sge.sc08/dmp_jobs#token_estimate.sh
250

Step #3: Create a "consumable resource" for Nastran tokens.
 
tm19-231:/dpl/sge.sc08#qconf -mc
...
nastran_tokens      nt         INT         <=    FORCED      YES        0        0
,,,

Step #4: Set the consumable resource "nastran_tokens" to the total amount available in the
token pool (and make this available to all hosts and queues (i.e., "global").
 
For example, I added "nastran_tokens=500" to the exec host (global) setting using SGE edit
command "qconf -me global":

To verify this setting use the following "show" command:

tm19-231:/dpl/sge.sc08#qconf -se global

hostname              global

load_scaling          NONE

complex_values        nastran_tokens=500,export_size=500G

load_values           NONE

processors            0

user_lists            NONE

xuser_lists           NONE

projects              NONE

xprojects             NONE

usage_scaling         NONE

report_variables      NONE

Step #5: Now you can submit your Nastran job and specify the required nastran tokens needed for your job
using  "-l nastran_tokens=`token_estimate.sh` on the SGE command line.
(in this example nastran_tokens=`token_estimate.sh`will resolve to nastran_tokens=250).
So, for example, with 500 total tokens available you would be able to submit 2 of these 250
token jobs concurrently-any additional jobs would not be dispatched until one of these
jobs completed and released its license tokens.

Here's an example taken from my earlier blog showing the use of the nastran_tokens attribute for a distributed memory parallel (DMP) Nastran job:

#cat nastran_qsub.sh
qsub -pe nastran 2 -q large.q  -l mem_free=6G -l nastran_tokens=`token_estimate.sh`  -l export_size=10G
-S /bin/ksh nastran_sge.sh

 

---

 

In my next blog in this series I'll show how I configured the various queues and parallel environment for MD Nastran.