Continuing from my last blog in this series on integrating "Sun Grid Engine and MD Nastran" I will now describe how to submit MSC.Software's MD Nastran DMP (Distributed Memory Parallel) jobs to SGE. In subsequent blogs I'll explain how I configured the SGE queues and parallel execution environment "nastran" referenced below.
Here's a brief overview of how all this works for submitting DMP jobs:
First, some background. MD Nastran offers the ability to run certain solution sequences in parallel using the Message Passing Interface (MPI), an industry-wide standard library for C and Fortran message-passing programs. For the Solaris x86 version of MD Nastran Sun HPC ClusterTools (based on Open MPI) is used for the MPI.
There are 2 basic requirements for submitting MD Nastran (DMP) jobs:
(1) You must specify the number of parallel processes you want (MD Nastran command line keyword "dmp=n"),
(2) And, if you want to spread the parallel processes out to more than just the local machine you are submitting Nastran from then you must specify target "host" machine(s) using the MD Nastran command line keyword "hosts=" (e.g, hosts=node1,node2, .....). With these basic requirements in mind I show below how to use the SGE qsub command to control a Nastran DMP job submission, including the determination of the "host" machines to run these parallel jobs. I also show how to use MSC's job resources estimation tool ("estimate") to automate the determination of the required (memory, disk, and license tokens)
So, here's what you would do to run a DMP(Distributed Parallel) MD Nastran job using SGE.
Step #1. Create a script containing the qsub command.
....for example,
#cat nastran_qsub.sh
qsub -pe nastran 2 -q large.q -l mem_free=6G -l nastran_tokens=`token_estimate.sh` -l export_size=10G -S /bin/ksh nastran_sge.sh
The above SGE command line defines the following:
1. "qsub" : qsub is the SGE command that I used to submit the Nastran job submittal script "nastran_sge.sh" to the parallel envrionment (nastran) I configured within SGE. The standard Nastran job submittal command "mdnast2008..." that Nastran user's are familiar with is in the script "nastran_sge.sh" (see Step 2. below).
2." -pe nastran 2" : This tells qsub to submit "nastran_sge.sh" to the SGE parallel execution environment (nastran). The "nastran_sge.sh" script will then start two Nastran jobs (running in parallel) on one or more of the host machines as defined in the queue (large.q).
[The queues that are suitable for this job, like the large.q, are queues that are associated with the parallel environment interface "nastran" by the parallel environment configuration. Suitable queues also must satisfy the resource requirement specification specified by the qsub -l command (see item 4. below).] I'll explain how I configured the parallel environment "nastran" and the associated queues in a subsequent blog.
3. "-q large.q" : I configured three SGE queues for my Nastran environment (small.q, medium.q, and large.q)--each one having a different limit on the amount of elapsed time allowed for jobs within the queue. In this example I chose large.q to handle a long (elapsed time) running job.
4. "-l mem_free=6G" -l nastran_tokens=`token_estimate.sh` -l export_size=10G" defines the "complex resource attributes" that I configured within SGE. I'll explain how I created these in a later blog, but basically they allow the user to define the amount of available real memory (6GB in this case) required for this job, the number of Nastran license tokens that are required, and also the required amount of disk space (10GB in this case) for the Nastran database files---if either of these requirements (free memory, tokens, or disk space) is not met the jobs will not be dispatched.
[In this example, I only show MSC's "estimate" program being used to calculate the required license tokens. In a subsequent blog in this series I'll show
when/how you can use "estimate" to also automate the calculation of the memory and disk requirements.
Step #2. Create a wrapper script (referenced in Step 1. above) containing the standard MD Nastran DMP job submittal command:
...for example,
#cat nastran_sge.sh
#! /bin/ksh
#
# sge_nast: Sun Grid Engine wrapper script to use with MSC.Nastran V2001.0.9 and greater.
#
# Usage: qsub -pe nastran $Nproc .... nastran_sge.sh
#
#
#Set nastran information for Head node:
mdnast2008=/msc/nastran/bin/mdnast2008
#
#
#Set up list of host(s) for use by mdnast2008 job submittal command (see below):
#
HOSTS=""
while read FILE
do
NODE=`echo $FILE | awk '{ print $1}'`
HOST0="$HOSTS"
HOSTS="$HOST0:$NODE"
echo $HOSTS
done < $PE_HOSTFILE
# Remove leading ':'
HOSTS=`echo $HOSTS | sed 's/://'`
echo $HOSTS
echo $PE_HOSTFILE
echo $NSLOTS
NSLOTS=`echo $NSLOTS`
#
# Got hosts names, now run Nastran with DMP (Distributed Memory Parallel) spreading the parallel Nastran jobs across the list of
# host computers ($HOSTS):
#
$mdnast2008 /dpl/sge.sc08/dmp_jobs/dmp_101c.dat out=out.dmp.sge.2008.r2.101c dmp=$NSLOTS hosts=$HOSTS scr=yes bat=no auth=1700@matsci.sfbay
#
# End
#
The above script uses the following SGE parallel execution environment variables:
NSLOTS – The number of queue slots in use by a parallel job.
PE_HOSTFILE – The path of a file that contains the definition of the virtual parallel machine that is assigned to a parallel job by the grid engine system. This variable is used for parallel jobs only. It contains the list of "host" machines that satisfy the resource and queue requirements as specified on the above qsub command in Step 1.
The above mdnast2008 command will result in the parallel nastran jobs being started on different hosts (e.g., dmp=2 hosts=tmp19-231:tm19-232 ).
Below is an example of the output you'll see using the above scripts:
Note that the output from the SGE "qstat -f" command shown below shows an "r" to indicate the 2 nastran jobs are running:
200 0.55500 nastran_sg dpl r 02/08/2009 17:52:10 1
If one of the requested resources had not been available (memory, disk, or tokens) then instead of "r" you would see the following output from qstat -f, indicating that the job has been put into a "hold/wait" status, pending availability of the requested resource(s).
############################################################################
- PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS
############################################################################
201 0.55500 nastran_sg dpl qw 02/08/2009 18:05:58 2
tm19-231#nastran_qsub.sh
Your job 200 ("nastran_sge.sh") has been submitted
tm19-231#
tm19-231#qstat -f
queuename qtype resv/used/tot. load_avg arch states
---------------------------------------------------------------------------------
all.q@tm19-232 BIP 0/0/8 0.20 sol-amd64
---------------------------------------------------------------------------------
large.q@tm19-231 BIP 0/1/1 0.10 sol-amd64
200 0.55500 nastran_sg dpl r 02/08/2009 17:52:10 1
---------------------------------------------------------------------------------
large.q@tm19-232 BIP 0/1/1 0.20 sol-amd64
200 0.55500 nastran_sg dpl r 02/08/2009 17:52:10 1
---------------------------------------------------------------------------------
medium.q@tm19-232 BIP 0/0/4 0.20 sol-amd64
---------------------------------------------------------------------------------
small.q@tm19-231 BIP 0/0/4 0.10 sol-amd64
In subsequent blogs in this series I'll describe in detail how I configured SGE for the above MD Nastran parallel job submittal (including "how to create the "nastran" parallel environment", creating consumable resources (nastran_tokens), using MSC.Software's "estimate" program, and other MD Nastran specific configurations that I found useful in conjunction with SGE.
