Shared Pool

« Grid Engine Service... | Main | An Example to Manage... »
pageicon Saturday Jul 05, 2008

An Example To Add GE Adapter Service To an SDM system

First the SGE qmaster host needs  to be installed with the JMX feature:

node1# export SGE_QMASTER_PORT=6236
node1# export SGE_EXECD_PORT=6237

 node1# ./install_qmaster -jmx
....

Grid Engine JMX MBean server
----------------------------
Please give some basic parameters for JMX MBean server
...
Please enter JAVA_HOME or press enter [/usr/java] >>
Please enter additional JVM arguments (optional, default is [-Xmx256m]) >>
Please enter an unused port number for the JMX MBean server >> 6238
Enable JMX SSL server authentication (y/n) [y] >>

Enable JMX SSL client authentication (y/n) [y] >>

Enter JMX SSL server keystore path [/var/sgeCA/port8236/default/private/keystore] >>
Enter JMX SSL server keystore pw >>
Using the following JMX MBean server settings.
   libjvm_path              >/usr/java/jre/lib/sparcv9/server/libjvm.so<
   Additional JVM arguments >-Xmx256m<
   JMX port                 >6238<
   JMX ssl                  >true<
   JMX client ssl           >true<
   JMX server keystore      >/var/sgeCA/port8236/default/private/keystore<
   JMX server keystore pw   ><

Do you want to use these data (y/n) [y] >>
...
...

Initializing Certificate Authority (CA) for OpenSSL security framework
----------------------------------------------------------------------
Creating /var/opt/sge/6.2beta/default/common/sgeCA
Creating /var/sgeCA/port8236/default
Creating /var/opt/sge/6.2beta/default/common/sgeCA/certs
Creating /var/opt/sge/6.2beta/default/common/sgeCA/crl
Creating /var/opt/sge/6.2beta/default/common/sgeCA/newcerts
Creating /var/opt/sge/6.2beta/default/common/sgeCA/serial
Creating /var/opt/sge/6.2beta/default/common/sgeCA/index.txt
Creating /var/opt/sge/6.2beta/default/common/sgeCA/usercerts
Creating /var/sgeCA/port8236/default/userkeys
Creating /var/sgeCA/port8236/default/private
...
...

You selected the following basic data for the distinguished name of
your certificates:

Country code:         C=US
State:                ST=MA
Location:             L=BUR
Organization:         O=JAVA
Organizational unit:  OU=TSC
CA email address:     emailAddress=sdmadmin@netadm.com

Do you want to use these data (y/n) [y] >>

Creating CA certificate and private key
Generating a 1024 bit RSA private key
......................................++++++
............................................................++++++
writing new private key to '/var/sgeCA/port6236/default/private/cakey.pem'
-----
...
...
Creating 'daemon' certificate and key for SGE Daemon
----------------------------------------------------
Generating a 1024 bit RSA private key
......................++++++
.....................++++++
writing new private key to '/var/sgeCA/port6236/default/private/key.pem'
-----
Using configuration from /tmp/sge_ca114856.tmp
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'US'
stateOrProvinceName   :PRINTABLE:'MA'
localityName          :PRINTABLE:'BUR'
organizationName      :PRINTABLE:'JAVA'
organizationalUnitName:PRINTABLE:'TSC'
userId                :PRINTABLE:'root'
commonName            :PRINTABLE:'SGE Daemon'
emailAddress          :IA5STRING:'none'
Certificate is to be certified until Jun 18 14:08:41 2009 GMT (365 days)

Write out database with 1 new entries
Data Base Updated
created and signed certificate for SGE daemons

Creating 'user' certificate and key for SGE install user
--------------------------------------------------------
Generating a 1024 bit RSA private key
..........................................++++++
............++++++
writing new private key to '/var/sgeCA/port6236/default/userkeys/root/key.pem'
-----
Using configuration from /tmp/sge_ca114856.tmp
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'US'
stateOrProvinceName   :PRINTABLE:'MA'
localityName          :PRINTABLE:'BUR'
organizationName      :PRINTABLE:'JAVA'
organizationalUnitName:PRINTABLE:'TSC'
userId                :PRINTABLE:'root'
commonName            :PRINTABLE:'SGE install user'
emailAddress          :IA5STRING:'none'
Certificate is to be certified until Jun 18 14:08:44 2009 GMT (365 days)

Write out database with 1 new entries
Data Base Updated
created and signed certificate for user 'root' in '/var/sgeCA/port6236/default/userkeys/root'

Creating 'user' certificate and key for SGE admin user
------------------------------------------------------
Generating a 1024 bit RSA private key
...................++++++
............++++++
writing new private key to '/var/sgeCA/port6236/default/userkeys/sdmadmin/key.pem'
-----
Using configuration from /tmp/sge_ca114856.tmp
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'US'
stateOrProvinceName   :PRINTABLE:'MA'
localityName          :PRINTABLE:'BUR'
organizationName      :PRINTABLE:'JAVA'
organizationalUnitName:PRINTABLE:'TSC'
userId                :PRINTABLE:'sdmadmin'
commonName            :PRINTABLE:'SGE admin user'
emailAddress          :IA5STRING:'none'
Certificate is to be certified until Jun 18 14:08:47 2009 GMT (365 days)

Write out database with 1 new entries
Data Base Updated
created and signed certificate for user 'sgeadmin' in '/var/sgeCA/port6236/default/userkeys/sdmadmin'
Hit <RETURN> to continue >>


Grid Engine qmaster startup
---------------------------

Starting qmaster daemon. Please wait ...
registered:urn:st:b95cb272-8495-c690-8bb7-834b98369b98  [if you enabled the service tag...]
   starting sge_qmaster
Hit <RETURN> to continue >>

...
...
Using Grid Engine
-----------------

You should now enter the command:

   source /var/opt/sge/6.2beta/default/common/settings.csh

if you are a csh/tcsh user or

   # . /var/opt/sge/6.2beta/default/common/settings.sh

if you are a sh/ksh user.

This will set or expand the following environment variables:

   - $SGE_ROOT         (always necessary)
   - $SGE_CELL         (if you are using a cell other than >default<)
   - $SGE_CLUSTER_NAME (always necessary)
   - $SGE_QMASTER_PORT (if you haven't added the service >sge_qmaster<)
   - $SGE_EXECD_PORT   (if you haven't added the service >sge_execd<)
   - $PATH/$path       (to find the Grid Engine binaries)
   - $MANPATH          (to access the manual pages)
The SGE bootstrap file contents:

node1# cat $SGE_ROOT/$SGE_CELL/common/bootstrap
# Version: 6.2beta
#
admin_user             sdmadmin
default_domain          none
ignore_fqdn             true
spooling_method         classic
spooling_lib            libspoolc
spooling_params         /var/opt/sge/6.2beta/default/common;/var/opt/sge/6.2beta/default/spool/qmaster
binary_path             /var/opt/sge/6.2beta/bin
qmaster_spool_dir       /var/opt/sge/6.2beta/default/spool/qmaster
security_mode           none
listener_threads        2
worker_threads          2
scheduler_threads       1
jvm_threads             1

JMX Configuration Files Location and Their Contents:

node1# pwd
/var/opt/sge/6.2beta/default/common/jmx

node1# egrep -v '^#' management.properties  | more
com.sun.grid.jgdi.management.jmxremote.port=6238
com.sun.grid.jgdi.management.jmxremote.ssl=true
com.sun.grid.jgdi.management.jmxremote.ssl.need.client.auth=true
com.sun.grid.jgdi.management.jmxremote.authenticate=true
com.sun.grid.jgdi.management.jmxremote.login.config=GridwareConfig
com.sun.grid.jgdi.management.jmxremote.password.file=/var/opt/sge/6.2beta/default/common/jmx/jmxremote.password
com.sun.grid.jgdi.management.jmxremote.access.file=/var/opt/sge/6.2beta/default/common/jmx/jmxremote.access
com.sun.grid.jgdi.management.jmxremote.ssl.serverKeystore=/var/sgeCA/port8236/default/private/keystore
com.sun.grid.jgdi.management.jmxremote.ssl.serverKeystorePassword=

node1# egrep -v '^#' jmxremote.access
monitorRole   readonly
controlRole   readwrite

node1# egrep -v '^#' jmxremote.password
monitorRole  QED
controlRole  R&D

node1# egrep -v '^#' logging.properties
handlers = java.util.logging.FileHandler
.level = INFO
java.util.logging.ConsoleHandler.level = INFO
java.util.logging.ConsoleHandler.formatter = com.sun.grid.jgdi.util.SGEFormatter
java.util.logging.FileHandler.level = ALL
java.util.logging.FileHandler.pattern=jgdi%u.log
java.util.logging.FileHandler.formatter=com.sun.grid.jgdi.util.SGEFormatter
com.sun.grid.jgdi.util.SGEFormatter.columns = time thread source level message
com.sun.grid.jgdi.util.SGEFormatter.withStacktrace=true
com.sun.grid.jgdi.util.SGEFormatter.delimiter = |


 How to test whether or not the JMX feature is working:

You can use either the jconsole or the JMX event monitor.
 

% jconsole -J-Djava.security.manager=java.rmi.RMISecurityManager \
           -J-Djava.security.policy=$SGE_ROOT/util/rmiconsole.policy \
           -J-Djavax.net.ssl.trustStore=<server truststore> \
          [-J-Djavax.net.ssl.keyStore=/<safe>/mykeystore \
           -J-Djavax.net.ssl.keyStorePassword=<mykeystore_pw> \
           -J-Djavax.net.ssl.keyPassword=<mykeystore_pw> ] \
          [-J-Djavax.net.debug=ssl]

% jconsole -J-Djava.security.manager=java.rmi.RMISecurityManager \
           -J-Djava.security.policy=$SGE_ROOT/util/rmiconsole.policy \
           -J-Djavax.net.ssl.trustStore=/var/sgeCA/port6236/default/private/keystore 

Note: Need to do the X forwarding since jconsole will open a GUI window.
The following example can be used to connect to the qmaster host via JMX and monitor any Grid Engine events.
% java  [-Dcom.sun.grid.jgdi.keyStore=/var/sgeCA/port$SGE_QMASTER_PORT/$SGE_CELL/private/keystore \
        -Dcom.sun.grid.jgdi.caTop="$SGE_ROOT/$SGE_CELL/common/sgeCA" \
        -Djava.util.logging.config.file=util/shell_logging.properties ] \
        -cp $SGE_ROOT/lib/juti.jar:$SGE_ROOT/lib/jgdi.jar \
        com.sun.grid.jgdi.examples.jmxeventmonitor.Main

Example 1)

% java  -cp /var/opt/sge/6.2beta/lib/juti.jar:/var/opt/sge/6.2beta/lib/jgdi.jar \
        com.sun.grid.jgdi.examples.jmxeventmonitor.Main

        = Provide JMX port [6238] and the root password,
        = Enable SSL
        = Provide the keystore, its password [I put nothing] and caTop path.

Once connected, you need to select "all" and "auto commit" and click commit.

Example 2)

% java  -Dcom.sun.grid.jgdi.keyStore=/var/sgeCA/port6236/default/private/keystore \
        -Dcom.sun.grid.jgdi.caTop="/var/opt/sge/6.2beta/default/common/sgeCA" \
        -cp /var/opt/sge/6.2beta/lib/juti.jar:/var/opt/sge/6.2beta/lib/jgdi.jar \
        com.sun.grid.jgdi.examples.jmxeventmonitor.Main

        = Provide JMX port [6238] and the root password,
        = Enable SSL
        = Provide keystore password if defined [I defined nothing]


node1# env|grep SGE
SGE_CELL=default
SGE_EXECD_PORT=6237
SGE_QMASTER_PORT=6236
SGE_ROOT=/var/opt/sge/6.2beta
SGE_CLUSTER_NAME=p6236

After the SGE qmaster host is installed with the JMX feature, the SDM system is ready to add the GE adapter service as shown below.

  1. Login to the Grid Engine master host.  [node1]
  2. Startup an SDM executor process on the host if not already started.

    3. node1# sdmadm -s sdm62beta2 show_jvm -h node1
    name        host  state      used_mem  max_mem   message
    -----------------------------------------------------------------------------------------
    executor_vm node1 STARTED           4M       28M

  3. Define an environment variable that identifies the SDM master host ($SDM_SYSTEM).

    4. node1# echo $SDM_SYSTEM
    sdm62beta2

  4. To add the Grid Engine service, the following form of the SDM administration command.

You need to create the user certification for the SDM admin user and make the SDM admin user as a SGE admin user.

node1# cat /var/tmp/sdmadm_ca.txt
sdmadmin::sdmadmin@netadm.com

node1# $SGE_ROOT/util/sgeCA/sge_ca -usercert /var/tmp/sdmadm_ca.txt

Alternatively, the following can be done:

node1# $SGE_ROOT/util/sgeCA/sge_ca -user "sdmadmin::sdmadmin@netadm.com"

After this operation, the following files were generated on the machine:

node1# find /var/sgeCA | grep sdmadmin /var/sgeCA/port6236/default/userkeys/sdmadmin /var/sgeCA/port6236/default/userkeys/sdmadmin/rand.seed /var/sgeCA/port6236/default/userkeys/sdmadmin/key.pem /var/sgeCA/port6236/default/userkeys/sdmadmin/req.pem /var/sgeCA/port6236/default/userkeys/sdmadmin/cert.pem

node1# qconf -am sdmadmin
root@node1 added "sdmadmin" to manager list

As the user root on the SGE qmaster host, run the following command to create the keystore:

node1# export JAVA_HOME=/usr/java
node1# $SGE_ROOT/util/sgeCA/sge_ca -userks

node1# find /var/sgeCA |grep keystore
/var/sgeCA/port6236/default/userkeys/root/keystore
/var/sgeCA/port6236/default/userkeys/sgeadmin/keystore
 /var/sgeCA/port6236/default/userkeys/sdmadmin/keystore /var/sgeCA/port6236/default/private/keystore

As an SDM admin user, sdmadmin, run the following command on the qmaster host:

node1$ sdmadm add_ge_service -h node1 -j rp_vm -s gesvc [-start]

Note: You may give any name for the GE adapter service. In this example, the name, gesvc, is used.

When you add the GE service, it will open up an XML configuration editor window. The following xml file shows an example GE service configuration used in this example:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<common:componentConfig xsi:type="ge_adapter:GEServiceConfig"
                        mapping="default"
                        xmlns:executor="http://hedeby/sunsource.net/hedeby-executor"
                        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                        xmlns:reporter="http://hedeby/sunsource.net/hedeby-reporter"
                        xmlns:security="http://hedeby/sunsource.net/hedeby-security"
                        xmlns:resource_provider="http://hedeby/sunsource.net/hedeby-resource-provider"
                        xmlns:common="http://hedeby/sunsource.net/hedeby-common"
                        xmlns:ge_adapter="http://hedeby/sunsource.net/hedeby-gridengine-adapter">
    <common:slos>
        <common:slo xsi:type="common:FixedUsageSLOConfig"
                    urgency="50"
                    name="fixed_usage"/>
    </common:slos>
    <ge_adapter:connection keystore="/var/sgeCA/port6236/default/userkeys/sdmadmin/keystore"
                           password=""             [Use when no keystore or keystore w/ password]
                           username="sdmadmin"     [This user must be an SDM user with the SGE admin user privilege]
                           jmxPort="6238"
                           execdPort="6237"
                           masterPort="6236"
                           cell="default"
                           root="/var/opt/sge/6.2beta"
                           clusterName="p6236"/>
    <ge_adapter:sloUpdateInterval unit="minutes"
                                  value="5"/>
    <ge_adapter:execd adminUsername="root"
                      defaultDomain=""
                      ignoreFQDN="true"
                      rcScript="false"
                      adminHost="true"
                      submitHost="false"
                      cleanupDefault="true"/>
</common:componentConfig>

When you save the configuration, you may get the following error if you used the -start flag but the rp_vm was not running already.  Then you need to start the rp_vm manually.
Error: Configuration of GE service: geadapter has been added, but start of component failed.
[If you use -start but rp_vm didn't started already, you will get this error.]

node1# sdmadm -s sdm62beta2 startup_jvm -j rp_vm

NOTE: If you use an SGE admin user, who is not an SDM admin user, in the GE adapter connection configuration, you will get a "permission denied" error as shown below. In this error example, the SGE admin user, sgeadmin, is not an SDM admin user.

node1# sdmadm -s sdm62beta2 show_service
host  service    cstate   sstate
---------------------------------
node0 spare_pool STARTED  RUNNING
node1 geadapter  STARTING ERROR

node1# cat rp_vm-0.log
18/06/2008 13:55:24|10|I|startup jvm (pid=19903)
18/06/2008 13:55:31|11|I|Secure mbean server started (service:jmx:rmi:///jndi/rmi://node1:53391/sdm62beta2)
18/06/2008 13:55:34|12|E|Cannot create keystore from /var/sgeCA/port6236/default/userkeys/sgeadmin/keystore: /var/sgeCA/port6236/default/userkeys/sgeadmin/keystore (Permission denied)

node1# cat rp_vm.stderr
missing bundle key: Cannot create keystore from /var/sgeCA/port6236/default/userkeys/sgeadmin/keystore: /var/sgeCA/port6236/default/userkeys/sgeadmin/keystore (Permission denied)


Posted at 10:36AM Jul 05, 2008 by Chansup Byun in Grid  |  Comments[0]

Comments:

Post a Comment:
  • HTML Syntax: NOT allowed

« December 2009
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
  
       
Today

Feeds

Search this blog

Links

Weblog menu

Today's referrers

Today's Page Hits: 19