49er: Oh my darling clementine

Today Enrico Sirola has announced a project hosted at Google Code where he keeps a Python language binding for DRMAA OGF standard. Enrico says the binding is a complete rewrite and can be used without SWIG. It means deploying the binding should be much easier than previous versions! As used the binding should be usable with all Distributed Resource Manager systems for which a shared DRMAA C library is available.

As to satisfy user requests like the one explaiend in [GE users] worth a wiki entry for SGE with OpenMPI and Infiniband a last-minute  enhancement was implemented for SGE 6.2 with HP-2008-07-11-0.  It adds a set of new parameters to the execd_params section of sge_conf(5):

    S_DESCRIPTORS, H_DESCRIPTORS, S_MAXPROC, H_MAXPROC,
     S_MEMORYLOCKED, H_MEMORYLOCKED, S_LOCKS, H_LOCKS
          Specifies soft and hard resource limits as  implemented
          by  the  setrlimit(2) system call. See this manual page
          on your system for more information.  These  parameters
          complete  the list of limits set by the RESOURCE LIMITS
          parameter of the queue configuration  as  described  in
          queue_conf(5).  Unlike the resource limits in the queue
          configuration, these resource limits are set for  every
          job  on  this  execution host. If a value is not speci-
          fied, the resource limit is inherited from  the  execu-
          tion   daemon  process.  Because  this  would  lead  to
          unpredicted results, if only one limit of a resource is
          set  (soft  or  hard), the corresponding other limit is
          set to the same value.

          S_DESCRIPTORS and H_DESCRIPTORS  specify  a  value  one
          greater  than  the  maximum file descriptor number that
          can be opened by any process of a job.

          S_MAXPROC and H_MAXPROC specify the maximum  number  of
          processes  that  can be created by the job user on this
          execution host

          S_MEMORYLOCKED and H_MEMORYLOCKED specify  the  maximum
          number  of  bytes  of virtual memory that may be locked
          into RAM.

          S_LOCKS and H_LOCKS specify the maximum number of  file
          locks any process of a job may establish.
          All of these values can be specified using  the  multi-
          plier letters k, K, m, M, g and G, see sge_types(1) for
          details.

Actually it does what is needed, but I see it still as a hack. As a regular enhancement, it had been done as change with queue_conf(5) that is used to control all the other resource limits. Anyways it fixed the CRs 6722507 and 6533487. For the latter there is an IZ counterpart (#2370).

A drmaa4ruby based workflow processor for DRMAA jobs can be found under flow.rb.

For a summary of it's characteristics refer to the README file.

This processor should work not only with SGE, since drmaa4ruby itself bases on DRMAA OGF standard.

Today will be the LSARC inception meeting for the impending release of Sun Grid Engine 6.2.

LSARC stands for "Layererd Software Architecture Review Committee" and is a Sun-internal process to ensure certain quality criterions are met.

Actually the specification for SGE 6.2 enhancement Advance Reservation is also availalbe.

The specification document for the SGE 6.2 support for Service Management Facility under Solaris 10 is now publicly availalbe.

In new SGE 6.2 there will be two changes that I consider important even though they won't make it into the headlines. Both have to do with the -masterq option that can be used to control the queue of a master task of a parallel job:

• issue #2378 - ensures "-masterq <wc_queue_list>" and "-soft -q <wc_queue_list>" can be used independent
  

• issue #2603 - ensures "-masterq <wc_queue_list>" and "-hard -q <wc_queue_list>" can be used independent

the first issue was raised by an open-source user who had problems with dispatching his parallel tasks. Over the second I stumbled during the SGE beta program. The beta program participant had issues with N1GE 6 - Scheduler Hacks: Exclusive master host for the master task in Stephan Grells blog. To be honest I wasn't aware such nasty workarounds are common at sites where SGE is deployed. Anywas, if all goes well, it shouldn't be needed anymore.

Today we had a training for support folks in Regensburg. I thought I make the presentation I held there available to the community. Subject of it is an overview about the major scalability improvements in 6.0ux, 6.1ux and 6.2. You find it under Scalability 6.x

I finally chose to blog and be it just as to fulfill the wish of my friend Daniel Templeton.

The theme of my blog "Oh my darling Clementine" is meant to be as reverence to the prospectors craftwork.