Dienstag September 05, 2006

A warm welcome to Praha The Sun re-org worked for us very well. I use the chance to send a warm welcome to our new colleagues in Praha.With this new branch in our team will we become really distributed. We do now have the folks from the open source helping us with the project and 3 locations within Sun, where we are doing our development from. It will be interesting to see how fast we can do the integration and which new projects we can now take on. So, again, a warm welcome to our new team members. I hope we will continue as we started last week with the big get to gether. ( Sep 05 2006, 09:55:12 AM CEST ) Permalink

N1GE 6 - Profiling

The Grid Engine software provides a profiling facility to determain where the qmaster and the scheduler spend their time. This has been introduced long before the N1GE 6 software. With the development of N1GE 6 it was greatly improved and its improvement continued over the the different updates we had for the N1GE 6 software. It was used very extensivly to analyse bottlenecks and find missconfigurations in existing installations. Until now, the source code was the only documentation for the output format, which might change with every new udpate and release. Lately a document was added to the source repository to give a brief overview of the output format and the different switches. The document is not complete, though it is a good start.

Profiling document
( Sep 05 2006, 09:42:47 AM CEST ) Permalink Kommentare [1]

Dienstag August 08, 2006

N1GE 6 - Scheduler Hacks: Exclusive master host for the master task
There was some discussion on the Open Source mailing lists and a lot of interests how one can single out the master task to a special host and have all the slave tasks on the compute nodes. There can be multiple reasons to do this, the one I heard most was that the master tasks needs a lot of memory and a special host exists just for that purpose.
During the discussion, we came across 3 work-arounds for the problem. I will start with the easiest setup and end with the most complicated. Since they are all workarounds, none of them is perfect. Never the less, they do archive the goal more or less.

1) Using the host sorting mechanism:

Description:

Dienstag April 25, 2006

N1GE 6 - Monitoring the qmaster
With the update 7 of the N1GE 6 software we added a new switch to monitor the qmaster. The qmaster monitoring allows to get statistics on each thread displaying what they have been busy with and how much time they spend on it. There are two switches to controll the statistic output:

qconf -mconf
qmaster_params               Monitor_Time=0:0:20 LOG_Monitor_Message=1

MONITOR_TIME
Specifies the time interval when the monitoring information should be printed. The monitoring is disabled per default and can be  enabled by specifying an interval. The monitoring is per thread and is written to the messages file or displayed by the "qping -f" command line tool. Example: MONITOR_TIME=0:0:10 generates the monitoring information most likely every 10 seconds and prints it. The specified time is a guideline and not a fixed interval. The used interval is printed and can be everything between 9 seconds and 20 in this example.

LOG_MONITOR_MESSAGE
The monitoring information is logged into the messages files per default. In addition it is provided for qping and can be requested by it. The messages files can become quite big, if the monitoring is enabled all the time, therefore this switch allows to disable the logging into the messages files and the monitoring data will only be available via "qping -f".

A description of the output format can be found here.

Example output in the qmaster messages file ($SGE_ROOT/<CELL>/spooling/qmaster/messages):

04/25/2006 19:06:17|qmaster|scrabe|P|EDT: runs: 1.20r/s (clients: 1.00 mod: 0.05/s ack: 0.05/s blocked: 0.00 busy: 0.00 | events: 0.05/s added: 0.05/s skipt: 0.00/s) out: 0.00m/s APT: 0.0001s/m idle: 99.99% wait: 0.00% time: 19.98s
04/25/2006 19:06:17|qmaster|scrabe|P|MT(2): runs: 0.25r/s (execd (l:0.00,j:0.00,c:0.00,p:0.00,a:0.00)/s GDI (a:0.05,g:0.00,m:0.00,d:0.00,c:0.00,t:0.00,p:0.00)/s event-acks: 0.05/s) out: 0.05m/s APT: 0.0002s/m idle: 100.00% wait: 0.00% time: 20.10s
04/25/2006 19:06:18|qmaster|scrabe|P|MT(1): runs: 0.19r/s (execd (l:0.00,j:0.00,c:0.00,p:0.00,a:0.00)/s GDI (a:0.05,g:0.00,m:0.05,d:0.00,c:0.00,t:0.00,p:0.00)/s event-acks: 0.00/s) out: 0.05m/s APT: 0.0001s/m idle: 100.00% wait: 0.00% time: 21.15s
04/25/2006 19:06:27|qmaster|scrabe|P|TET: runs: 0.67r/s (pending: 9.00 executed: 0.67/s) out: 0.00m/s APT: 0.0205s/m idle: 98.63% wait: 0.00% time: 21.00s
04/25/2006 19:06:37|qmaster|scrabe|P|EDT: runs: 1.60r/s (clients: 1.00 mod: 0.05/s ack: 0.05/s blocked: 0.00 busy: 0.00 | events: 1.10/s added: 1.10/s skipt: 0.00/s) out: 0.05m/s APT: 0.0002s/m idle: 99.97% wait: 0.00% time: 20.00s
04/25/2006 19:06:39|qmaster|scrabe|P|MT(1): runs: 0.37r/s (execd (l:0.00,j:0.00,c:0.00,p:0.00,a:0.00)/s GDI (a:0.14,g:0.00,m:0.05,d:0.00,c:0.00,t:0.05,p:0.00)/s event-acks: 0.05/s) out: 0.32m/s APT: 0.0024s/m idle: 99.91% wait: 0.00% time: 21.55s

N1GE 6 - Scheduler Hacks: Seperated Master host for pe jobs
In the distributions of pe jobs over a range of hosts, the pe provides a set of allocation rules. These rules allow the admin to specify that a host should be filed up first before another is used, that each host is used before any host runs a second task, or that the job uses a specified amount of slots on each host it is using. This solves most of the use cases around pe jobs.
In this commend I would like to scatch out a scenario which cannot be addressed with the existing allocation rules, the exclusive use by the master task of the master host while all other hosts will use the fill-up allocation rule. This can become handy if the master task of a job requires a lot of memory while the slave tasks do the computation and only one machine with a lot of memory is available. The big machine can and should run multiple master tasks of this job kind.

There are two solutions to the problem. One could separated the memory intense computation out into an extra job and work with job dependencies or one configures N1GE to handle the above use case as specified without any job modifications.

I have the following setup:

 qstat -f
queuename                qtype used/tot. load_avg arch          states
----------------------------------------------------------------------------
all.q@big                    BIP   0/4       0.02     sol-sparc64
----------------------------------------------------------------------------
small.q@small1        BIP   0/1       0.00     lx24-amd64
----------------------------------------------------------------------------
small.q@small2        BIP   0/1       0.02     sol-sparc64

And a configured pe in all queue instances:

qconf  -sp make
pe_name               make
slots                          999
user_lists                 NONE
xuser_lists              NONE
start_proc_args     NONE
stop_proc_args        NONE
allocation_rule        $fill_up
control_slaves         TRUE
job_is_first_task    FALSE
urgency_slots           min

We now go ahead and change the load_threshold in the all.q@big queue instance to be a load value that is not used in the other queue instances, such as:

qconf -sq all.q
qname                 all.q
hostlist              big
seq_no                0
load_thresholds       NONE,[big=load_avg=4]

The used load threshold has to be a real load value and cannot be a fixed or consumable value.

Next step to make our enviroment work is to change the scheduler configuration to the following:

qconf -ssconf
algorithm                         default
schedule_interval                 0:2:0
maxujobs                          0
queue_sort_method                 load
job_load_adjustments              load_avg=4.000000
load_adjustment_decay_time        0:0:1

By changing the configuration of the scheduler to use the job_load_adjustments like this, it will add an artificial load to each host, that will run a task. With this configuration we can start one task on the big machine in each scheduling run. Since the load_adjustment_decay_time is only 1 second, the scheduler has forgotten about the artificial load in the next scheduling run and can start a new task on the big host. This way, we archive what we have been looking for.

One important note:
The big machine is only allowed to have one queue instance, or all queue instances of the big machine have to share the same load threshold. If that is not the
case, it will not work.


( Apr 25 2006, 10:37:37 AM CEST ) Permalink

Freitag November 25, 2005

N1GE 6 - A couple lines on halftime and usage
The usage stored in the sharetree is controlled by its halftime. The halftime can be configured by using the halftime parameter in the scheduler configuration. This parameter specifies the halftime of the usage in hours. 1 hour is the smallest halftime one can set. A setting of 0 will turn the decrease of the usage of, which means that it will always grow.

A short coming on this setting is, that all three usages (io, mem, cpu) are treated equally. The halflife_decay_list the way to overcome this short coming. It allows to set different decay rates for io, mem and cpu usage. The time in this list is specified in minutes and in addition allows to turn of past usage at all, when the time is set to a negative value. A zero means no decrease of usage. If a decay time of -1 is set, all past usage is removed right away and only the usage of the current running jobs is used during the sharetree computation.

The settings of the halflife_decay_list will override the halftime settings.

Some examples ('qconf -msconf'):

1) Using half_life_decay_list to get the same result as with halftime

    halftime                           24
    halflife_decay_list        cpu=1440:mem=1440:io=1440

2) Disabeling past usage

    halftime                           168
    halflife_decay_list        cpu=-1:mem=-1:io=-1

3) Disable past usage for mem the others have a halftime of 1 day

    halftime                           24
    halflife_decay_list        mem=-1

Important:

    More than parameter in the halflife_decay_list would crash the system until version 6.0u7. We fixed issue 1826 in 6.0u7

( Nov 25 2005, 07:10:11 PM CET ) Permalink Kommentare [0]

Dienstag September 20, 2005

N1GE 6 - We want you!
We want you! We need you as testers!

We just finished the first part of the next N1Ge 6 update and would like you to take a look at what we have archived so far. In addition to the usual bug fixes the next update addresses some performance issues as well. Knowing that a software will always be to slow, we mad a cut where we currently are took a look at it and thought that it is time to ask for your opinion before we continue. The open source announcement gives you an overview of the improved parts in grid engine and the link to where you can download it.

In addition to the mentioned changes we included to additional work packages:

1) reworked qstat XML:

Donnerstag August 11, 2005

N1GE 6 - Scheduler Hacks: consumables as load threshold

This feature is new to N1GE 6 and can be quite powerful, if used correctly. One could model similar behavior earlier with custom load sensors and load adjustments but this is now much more reaktive, build in and easier to use. One could also find similarities to the subordinate queues. However, this solution is much more generic. It takes any consumable while the subordinate queues are fixed on slots.

To use this feature one has to create a consumable, assign it to a host or the global host and pick a queue which should be in alarm state, when a certain amount is consumed. Assuming that one assigns 5 licenses to a host, and the queue instance should go into alarm state, when 2 licenses are consumed, one would set:

>qconf -mc
licence lic int <= YES YES 0 0

>qconf -me test_host
complex_values lic = 4

>qconf -mq test.q
load_thresholds lic = 3

This is all that is needed. If now two jobs requesting lic are running on the all.q on the test_host, the test.q will go into alarm state and no further jobs will be started in that queue.

As one can see, this allows very complicated scenarios and a new way of handling job priorities and subordinating queues independly of the number of used slots.

Important:
An important difference is the possible definition of the consumable with is used as a load_threshold on queue level. This is not possible with load values.

I am sure it will take a while to play with this feature before useful scenarios are obvious.

( Aug 11 2005, 01:34:34 PM CEST ) Permalink Kommentare [1]

Donnerstag Juli 21, 2005

N1GE 6 - Scheduler Hacks: Sorting queues

I just received a question asking on how to use the queue sequence numbers and what to do with them. I will give a short overview in this blog and hope to give enough pointers for ones own experiments. Based on the documentation, the scheduler  can sort the queue instances in two ways:

• load based (from the hosts)
• sequence number based (from the queues)

Donnerstag Juli 14, 2005

N1GE 6 - health monitoring

Freitag Juni 17, 2005

N1GE 6 - Migration From LSF to Sun N1 GE Software Kirk Patton (Transmeta) describes in his paper how he replaced LSF by N1GE 6 and why
they moved to N1GE 6. This is a nice success story and gives a good comparison between
the two systems.

http://www.sun.com/bigadmin/features/articles/n1ge_migration.html


( Jun 17 2005, 09:29:21 AM CEST ) Permalink Kommentare [0]

Donnerstag Mai 12, 2005

N1GE 6 - Scheduler: Department-Based policy setup
In his latest howto Chris Dagdigian wrote about configuring the different policies of the N1GE 6 system to enable Department-Based scheduling. He gives a good summary of the available policies and how to setup the share tree policy. His Howto can be found under: http://bioteam.net/dag/sge6-funct-share-dept.html.
( Mai 12 2005, 10:51:28 AM CEST ) Permalink Kommentare [0]

Montag April 25, 2005

N1GE 6 - Scheduler Hacks: The ticket policy hierarchy
The N1GE 6 support fair share scheduling through its ticket policy. The ticket policy consists of three different parts:

- the Share Tree policy
- the functional policy
- the override policy

The share tree and the functional policy have a fixed amount of tickets, which gets distributed over all jobs in the system. The override policy is open. If "share override tickets" or "share functional tickets" are enabled, the ticket amount of a job depends in its submission time. The ticket amount in the share tree always depends on the submission time as well as the past usage. Of course to all three of them depend on the assignments from the configuration.

The statement, that the jobs' ticket amount depends on the submission time is a bit simple. An additional parameter comes into play through the Ticket Policy Hierarchy. It specifies the order in which the different ticket policies are called. The default is: OFS, which means:

1. O = override ticket policy
2. F = functional ticket policy
3. S = share tree policy

The real dependency for the final tickets are:

compute override tickets:
 - assigned tickets by the configuration
 - job submission time

compute functional tickets:
 - assigned tickets by the configuration
 - assigned override tickets
 - job submission time

compute share tree tickets:
 - assigned tickets by the configuration
 - usage
 - assigned override tickets
 - assigned functional tickets.
 - job submission time

As you see, the previous computed tickets have an effect on the following ticket policy. The next example will demonstrate it.

Setup:
qconf -msconf
   ticket_policy_hierarchy   OFS
   weight_tickets_functional         100000
   weight_tickets_share              0
   weight_ticket                     1.000000
   weight_waiting_time               0.000000
   weight_deadline                   3600000.000000
   weight_urgency                    0.000000
   weight_priority                   0.000000

qconf -aprj PRJ1
   name PRJ1
   oticket 10
   fshare 0
   acl NONE
   xacl NONE

qconf -muser <NAME>
   fshare 100

Jobs:
7 jobs without a project:
   qsub $SGE_ROOT/examples/jobs/sleeper.sh

2 PRJ1 jobs:
   qsub -P PRJ1 $SGE_ROOT/examples/jobs/sleeper.sh

qstat output:
JobId     P      S   Project  Tot-Tkt   ovrts   otckt  ftckt   stckt   shr
--------------------------------------------------------
223690 1.00000   qw     PRJ1   25010       0      10   25000       0    0.35
223691 0.50000   qw     PRJ1   12505       0       5   12500       0    0.18
223683 0.33320   qw       NA    8333       0       0    8333       0    0.12
223684 0.24990   qw       NA    6250       0       0    6250       0    0.09
223685 0.19992   qw       NA    5000       0       0    5000       0    0.07
223686 0.16660   qw       NA    4166       0       0    4166       0    0.06
223687 0.14280   qw       NA    3571       0       0    3571       0    0.05
223688 0.12495   qw       NA    3125       0       0    3125       0    0.04
223689 0.11107   qw       NA    2777       0       0    2777       0    0.04

Setup change:
I know modify the policy hierarchy:

qconf -msconf
   ticket_policy_hierarchy   FSO

and the qstat output changes to:
JobId     P      S   Project  Tot-Tkt   ovrts   otckt  ftckt   stckt   shr
--------------------------------------------------------
223683 1.00000   qw       NA   25000       0       0   25000       0    0.35
223684 0.50000   qw       NA   12500       0       0   12500       0    0.18
223685 0.33333   qw       NA    8333       0       0    8333       0    0.12
223686 0.25000   qw       NA    6250       0       0    6250       0    0.09
223687 0.20000   qw       NA    5000       0       0    5000       0    0.07
223688 0.16667   qw       NA    4166       0       0    4166       0    0.06
223689 0.14286   qw       NA    3571       0       0    3571       0    0.05
223690 0.12540   qw     PRJ1    3135       0      10    3125       0    0.04
223691 0.11131   qw     PRJ1    2782       0       5    2777       0    0.04

( Apr 25 2005, 12:27:04 PM CEST ) Permalink Kommentare [0]

Montag April 18, 2005

N1GE 6 - Scheduler Hacks: job execution priority

The nice level of a job can be set in different ways. The simple way is to turn the reprioritization feature off (it is the default setting) and set the nice level via the queue configuration.

qconf -mq all.q
   priority  0

All jobs running in the queue instance will run with defined nice level. One can now easily configure different cluster queues (such as low, medium, and high priority) with different nice levels.

This is the easy way and allows the user to decide how important the job is by submitting it to a specific cluster queue.

This approach is not always fine grained enough. Sometimes it is important to rank the jobs based on the scheduling priority. A high priority job should not only be scheduled as fast as possible but also run on a lower nice level than low priority jobs. The importance ranking for the scheduling decision is done via the ticket policy and others. But only the ticket policy has a direct impact on the job nice level when "reprioritize" is enabled. There are two places to enable and controll job reprioritization:

qconf -mconf
   reprioritize  0

qconf -msconf
   reprioritize_interval  0:0:0

One could assume, that one can also influence the reprioritization via:

qocnf -mconf <host_name>

but, even though the setting is accepted, if does not have an effect. The "reprioritize" flag enables/disables the feature. If it sets to true, the execd will monitor the usage of each job  that it is running. It knows the amount of tickets for each job and will ensure, that the ticket ratio between the jobs is the same ratio as the usage between the jobs. Every job gets initial start tickets. The scheduler will most certainly change them while the job is running. Therefore we have the reprioritize_interval, which will update the jobs on the execd side and ensure that the ratio between the usage reflects the ratio between the tickets via the nice level. Since it takes some time to adjust the jobs' usage via the nice level, the tickets should not be send too often. The recommendation is 2 minutes for the reprioritize_interval.

If the reprioritize_interval is set to 0:0:0, the reprioritize feature is disabled (e.q. reprioritize  is set 0). It also works the other way around, setting reprioritize_interval enables to feature by setting reprioritize to 1.

A sample setup with two projects:

Mittwoch April 13, 2005

N1GE 6 - Scheduler Hacks: Comment on the qmaster <-> scheduler protocol

If one is using any of the ticket policies one will most likely see something similar to:

04/12/2005 09:16:28|qmaster|xxx|E| orders user/project version (16468) is not uptodate (16469) for user/project "PRJ147"