Freitag September 08, 2006

It was fun, It was interesting. I still move on.
It is time to say goodbye. This will be the last entry in this blog. I will be leaving Sun in a week to start a new adventure. I did enjoy working with the Sun Grid team. I got to know lots of passionate and knowledge people. I hope the contacts will not entirely go away even though I switch cities, signed up with the competion, and will most likely be as busy as I was now. 
From what I have seen so fare my new home town, Aachen, will be nearly as nice as Regensburg. Its a bit bigger and right at the border to Netherlands and Belgium.
So, for everyone, who wants to stay in contact, my email address is: sgrell @ gmx.de.

Good bye,

Stephan

( Sep 08 2006, 08:49:49 PM CEST ) Permalink Kommentare [2]

Dienstag September 05, 2006

A warm welcome to Praha The Sun re-org worked for us very well. I use the chance to send a warm welcome to our new colleagues in Praha.With this new branch in our team will we become really distributed. We do now have the folks from the open source helping us with the project and 3 locations within Sun, where we are doing our development from. It will be interesting to see how fast we can do the integration and which new projects we can now take on. So, again, a warm welcome to our new team members. I hope we will continue as we started last week with the big get to gether. ( Sep 05 2006, 09:55:12 AM CEST ) Permalink

N1GE 6 - Profiling

The Grid Engine software provides a profiling facility to determain where the qmaster and the scheduler spend their time. This has been introduced long before the N1GE 6 software. With the development of N1GE 6 it was greatly improved and its improvement continued over the the different updates we had for the N1GE 6 software. It was used very extensivly to analyse bottlenecks and find missconfigurations in existing installations. Until now, the source code was the only documentation for the output format, which might change with every new udpate and release. Lately a document was added to the source repository to give a brief overview of the output format and the different switches. The document is not complete, though it is a good start.

Profiling document
( Sep 05 2006, 09:42:47 AM CEST ) Permalink Kommentare [1]

Dienstag August 08, 2006

N1GE 6 - Scheduler Hacks: Exclusive master host for the master task
There was some discussion on the Open Source mailing lists and a lot of interests how one can single out the master task to a special host and have all the slave tasks on the compute nodes. There can be multiple reasons to do this, the one I heard most was that the master tasks needs a lot of memory and a special host exists just for that purpose.
During the discussion, we came across 3 work-arounds for the problem. I will start with the easiest setup and end with the most complicated. Since they are all workarounds, none of them is perfect. Never the less, they do archive the goal more or less.

1) Using the host sorting mechanism:

Description:

Dienstag April 25, 2006

N1GE 6 - Monitoring the qmaster
With the update 7 of the N1GE 6 software we added a new switch to monitor the qmaster. The qmaster monitoring allows to get statistics on each thread displaying what they have been busy with and how much time they spend on it. There are two switches to controll the statistic output:

qconf -mconf
qmaster_params               Monitor_Time=0:0:20 LOG_Monitor_Message=1

MONITOR_TIME
Specifies the time interval when the monitoring information should be printed. The monitoring is disabled per default and can be  enabled by specifying an interval. The monitoring is per thread and is written to the messages file or displayed by the "qping -f" command line tool. Example: MONITOR_TIME=0:0:10 generates the monitoring information most likely every 10 seconds and prints it. The specified time is a guideline and not a fixed interval. The used interval is printed and can be everything between 9 seconds and 20 in this example.

LOG_MONITOR_MESSAGE
The monitoring information is logged into the messages files per default. In addition it is provided for qping and can be requested by it. The messages files can become quite big, if the monitoring is enabled all the time, therefore this switch allows to disable the logging into the messages files and the monitoring data will only be available via "qping -f".

A description of the output format can be found here.

Example output in the qmaster messages file ($SGE_ROOT/<CELL>/spooling/qmaster/messages):

04/25/2006 19:06:17|qmaster|scrabe|P|EDT: runs: 1.20r/s (clients: 1.00 mod: 0.05/s ack: 0.05/s blocked: 0.00 busy: 0.00 | events: 0.05/s added: 0.05/s skipt: 0.00/s) out: 0.00m/s APT: 0.0001s/m idle: 99.99% wait: 0.00% time: 19.98s
04/25/2006 19:06:17|qmaster|scrabe|P|MT(2): runs: 0.25r/s (execd (l:0.00,j:0.00,c:0.00,p:0.00,a:0.00)/s GDI (a:0.05,g:0.00,m:0.00,d:0.00,c:0.00,t:0.00,p:0.00)/s event-acks: 0.05/s) out: 0.05m/s APT: 0.0002s/m idle: 100.00% wait: 0.00% time: 20.10s
04/25/2006 19:06:18|qmaster|scrabe|P|MT(1): runs: 0.19r/s (execd (l:0.00,j:0.00,c:0.00,p:0.00,a:0.00)/s GDI (a:0.05,g:0.00,m:0.05,d:0.00,c:0.00,t:0.00,p:0.00)/s event-acks: 0.00/s) out: 0.05m/s APT: 0.0001s/m idle: 100.00% wait: 0.00% time: 21.15s
04/25/2006 19:06:27|qmaster|scrabe|P|TET: runs: 0.67r/s (pending: 9.00 executed: 0.67/s) out: 0.00m/s APT: 0.0205s/m idle: 98.63% wait: 0.00% time: 21.00s
04/25/2006 19:06:37|qmaster|scrabe|P|EDT: runs: 1.60r/s (clients: 1.00 mod: 0.05/s ack: 0.05/s blocked: 0.00 busy: 0.00 | events: 1.10/s added: 1.10/s skipt: 0.00/s) out: 0.05m/s APT: 0.0002s/m idle: 99.97% wait: 0.00% time: 20.00s
04/25/2006 19:06:39|qmaster|scrabe|P|MT(1): runs: 0.37r/s (execd (l:0.00,j:0.00,c:0.00,p:0.00,a:0.00)/s GDI (a:0.14,g:0.00,m:0.05,d:0.00,c:0.00,t:0.05,p:0.00)/s event-acks: 0.05/s) out: 0.32m/s APT: 0.0024s/m idle: 99.91% wait: 0.00% time: 21.55s

N1GE 6 - Scheduler Hacks: Seperated Master host for pe jobs
In the distributions of pe jobs over a range of hosts, the pe provides a set of allocation rules. These rules allow the admin to specify that a host should be filed up first before another is used, that each host is used before any host runs a second task, or that the job uses a specified amount of slots on each host it is using. This solves most of the use cases around pe jobs.
In this commend I would like to scatch out a scenario which cannot be addressed with the existing allocation rules, the exclusive use by the master task of the master host while all other hosts will use the fill-up allocation rule. This can become handy if the master task of a job requires a lot of memory while the slave tasks do the computation and only one machine with a lot of memory is available. The big machine can and should run multiple master tasks of this job kind.

There are two solutions to the problem. One could separated the memory intense computation out into an extra job and work with job dependencies or one configures N1GE to handle the above use case as specified without any job modifications.

I have the following setup:

 qstat -f
queuename                qtype used/tot. load_avg arch          states
----------------------------------------------------------------------------
all.q@big                    BIP   0/4       0.02     sol-sparc64
----------------------------------------------------------------------------
small.q@small1        BIP   0/1       0.00     lx24-amd64
----------------------------------------------------------------------------
small.q@small2        BIP   0/1       0.02     sol-sparc64

And a configured pe in all queue instances:

qconf  -sp make
pe_name               make
slots                          999
user_lists                 NONE
xuser_lists              NONE
start_proc_args     NONE
stop_proc_args        NONE
allocation_rule        $fill_up
control_slaves         TRUE
job_is_first_task    FALSE
urgency_slots           min

We now go ahead and change the load_threshold in the all.q@big queue instance to be a load value that is not used in the other queue instances, such as:

qconf -sq all.q
qname                 all.q
hostlist              big
seq_no                0
load_thresholds       NONE,[big=load_avg=4]

The used load threshold has to be a real load value and cannot be a fixed or consumable value.

Next step to make our enviroment work is to change the scheduler configuration to the following:

qconf -ssconf
algorithm                         default
schedule_interval                 0:2:0
maxujobs                          0
queue_sort_method                 load
job_load_adjustments              load_avg=4.000000
load_adjustment_decay_time        0:0:1

By changing the configuration of the scheduler to use the job_load_adjustments like this, it will add an artificial load to each host, that will run a task. With this configuration we can start one task on the big machine in each scheduling run. Since the load_adjustment_decay_time is only 1 second, the scheduler has forgotten about the artificial load in the next scheduling run and can start a new task on the big host. This way, we archive what we have been looking for.

One important note:
The big machine is only allowed to have one queue instance, or all queue instances of the big machine have to share the same load threshold. If that is not the
case, it will not work.


( Apr 25 2006, 10:37:37 AM CEST ) Permalink

Freitag November 25, 2005

N1GE 6 - A couple lines on halftime and usage
The usage stored in the sharetree is controlled by its halftime. The halftime can be configured by using the halftime parameter in the scheduler configuration. This parameter specifies the halftime of the usage in hours. 1 hour is the smallest halftime one can set. A setting of 0 will turn the decrease of the usage of, which means that it will always grow.

A short coming on this setting is, that all three usages (io, mem, cpu) are treated equally. The halflife_decay_list the way to overcome this short coming. It allows to set different decay rates for io, mem and cpu usage. The time in this list is specified in minutes and in addition allows to turn of past usage at all, when the time is set to a negative value. A zero means no decrease of usage. If a decay time of -1 is set, all past usage is removed right away and only the usage of the current running jobs is used during the sharetree computation.

The settings of the halflife_decay_list will override the halftime settings.

Some examples ('qconf -msconf'):

1) Using half_life_decay_list to get the same result as with halftime

    halftime                           24
    halflife_decay_list        cpu=1440:mem=1440:io=1440

2) Disabeling past usage

    halftime                           168
    halflife_decay_list        cpu=-1:mem=-1:io=-1

3) Disable past usage for mem the others have a halftime of 1 day

    halftime                           24
    halflife_decay_list        mem=-1

Important:

    More than parameter in the halflife_decay_list would crash the system until version 6.0u7. We fixed issue 1826 in 6.0u7

( Nov 25 2005, 07:10:11 PM CET ) Permalink Kommentare [0]

Dienstag September 20, 2005

N1GE 6 - We want you!
We want you! We need you as testers!

We just finished the first part of the next N1Ge 6 update and would like you to take a look at what we have archived so far. In addition to the usual bug fixes the next update addresses some performance issues as well. Knowing that a software will always be to slow, we mad a cut where we currently are took a look at it and thought that it is time to ask for your opinion before we continue. The open source announcement gives you an overview of the improved parts in grid engine and the link to where you can download it.

In addition to the mentioned changes we included to additional work packages:

1) reworked qstat XML:

Freitag August 26, 2005

Sun Grid Engine Team - flooding in Regensburg

Having the Sun lab in a safe distance form the river Donau, the team was never in danger. However walking through the Altstadt yesterday evening showed a slightly different picture. It is not as bad as it was a couple years ago, but it is still bad. The highest water mark was reached last night. The forcast says, that the water mark will start to drop this evening. We were lucky during the walk through the city. It rain stoped and the clouds opened up to give way for a crystal clear sky with only a couple clounds left. This gave us a breth taking sunset, which one cannot really see in the pictures. The news magazine covers the current flooding in south Germany, Austria, ans Switzerland.

The flooding in the news

BBC:

The Guardian

UNESCO

Spiegel

And now four pictures I took on my way through the Altstadt:










( Aug 26 2005, 02:12:11 PM CEST ) Permalink Kommentare [0]

Donnerstag August 11, 2005

N1GE 6 - Scheduler Hacks: consumables as load threshold

This feature is new to N1GE 6 and can be quite powerful, if used correctly. One could model similar behavior earlier with custom load sensors and load adjustments but this is now much more reaktive, build in and easier to use. One could also find similarities to the subordinate queues. However, this solution is much more generic. It takes any consumable while the subordinate queues are fixed on slots.

To use this feature one has to create a consumable, assign it to a host or the global host and pick a queue which should be in alarm state, when a certain amount is consumed. Assuming that one assigns 5 licenses to a host, and the queue instance should go into alarm state, when 2 licenses are consumed, one would set:

>qconf -mc
licence lic int <= YES YES 0 0

>qconf -me test_host
complex_values lic = 4

>qconf -mq test.q
load_thresholds lic = 3

This is all that is needed. If now two jobs requesting lic are running on the all.q on the test_host, the test.q will go into alarm state and no further jobs will be started in that queue.

As one can see, this allows very complicated scenarios and a new way of handling job priorities and subordinating queues independly of the number of used slots.

Important:
An important difference is the possible definition of the consumable with is used as a load_threshold on queue level. This is not possible with load values.

I am sure it will take a while to play with this feature before useful scenarios are obvious.

( Aug 11 2005, 01:34:34 PM CEST ) Permalink Kommentare [1]

N1GE 6 u6 is available Grid Engine 6.0 Update 6 is now ready and courtesy binaries are
available for download. The SGE 6.0u6 release fixes bugs of the
software, installation procedure and man pages.

   http://gridengine.sunsource.net/project/gridengine/60patches.txt

The courtesy binaries are available at:

   http://gridengine.sunsource.net/project/gridengine/download60.html

The list of fixed problems is available at

   http://gridengine.sunsource.net/project/gridengine/60patches.txt

The patch installation notes are available at

   http://gridengine.sunsource.net/project/gridengine/install60patch.txt

Please send support questions and feedback to the "users" mailing of the
Grid Engine open source project.

The corresponding source code tag in the CVS repository has the name

   V60u6_TAG

A snapshot of the sources is available at the Documents & files page

   http://gridengine.sunsource.net/servlets/ProjectDownloadList

as file

   sge-V60u6_TAG-src.tar.gz ( Aug 11 2005, 01:24:59 PM CEST ) Permalink Kommentare [0]

Donnerstag Juli 21, 2005

N1GE 6 - Scheduler Hacks: Sorting queues

I just received a question asking on how to use the queue sequence numbers and what to do with them. I will give a short overview in this blog and hope to give enough pointers for ones own experiments. Based on the documentation, the scheduler  can sort the queue instances in two ways:

• load based (from the hosts)
• sequence number based (from the queues)

Donnerstag Juli 14, 2005

N1GE 6 - health monitoring

Freitag Juni 17, 2005

N1GE 6 - Migration From LSF to Sun N1 GE Software Kirk Patton (Transmeta) describes in his paper how he replaced LSF by N1GE 6 and why
they moved to N1GE 6. This is a nice success story and gives a good comparison between
the two systems.

http://www.sun.com/bigadmin/features/articles/n1ge_migration.html


( Jun 17 2005, 09:29:21 AM CEST ) Permalink Kommentare [0]

Donnerstag Mai 12, 2005

Index
I am using this page to reference my past entries. I believe that the past entries an be as interesting as the new ones and that an index as some advantage over the existing search engines. Hm.. maybe it just my pet peeve that I need  stuff to be orgenized and structured. I find an index very helpfull, therefore I will provied it for you as the reader and for me to find my own posts faster. I am using my blog as a place to store information about the N1 GE 6, which have been discussed in mail threads or were unclear to users.

N1GE 6:

General:

Comments: