fintanr's weblog

Archives
« November 2009
MonTueWedThuFriSatSun
      
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
      
Today

the links



Twitter Updates

    follow me on Twitter













    blogs.sun.com
    Weblog
    Login



    All | Books | Gadgets | General | JES | OpenSolaris | Perl | Services | Slamd | Solaris | Travels | Tunes

    « LowTek (or computer... | Main | Predictive self... »
    20040903 Friday September 03, 2004

    A Sample Slamd Deployment - Benchmarking Directory Server
    A few weeks ago I did a brief posting about Slamd, so I figured its time to flesh it out with a few more details. I won't go into the details on setting up a slamd server, all of this is dealt with in the Slamd Documentation

    The Sample Rig

    The rig I'm going to discuss here is a Sun Java System Directory Server box which I set up a few weeks ago. Hardware wise, the directory server machine is a v20z, the client machines that I use for this particular rig are six old Netra boxes and an Ultra 30 as the slamd server.

    Diagrammatically the layout looks like

    The rig is on a private subnet. The gigabit switch is dedicated to just this rig.

    The LDAP Data, using MakeLDIF

    One of the tools that comes with Slamd is MakeLDIF. In general production LDAP data is not available for benchmarking purposes, so MakeLDIF allows you to create a sample ldif file which closely represents real world LDAP data.

    For the rig above we have created sample datasets of one hundred thousand, two hundred and fifty thousand, one million and five million users. For the purposes of this article I will use a simplified version of the two hundred and fifty thousand user example.
    Firstly you create a sample template to use with MakeLDIF, i.e. 

    define suffix=dc=example,dc=com
define maildomain=example.com
define numusers=250000

branch: [suffix]

branch: ou=People,[suffix]
subordinateTemplate: person:[numusers]

template: person
rdnAttr: employeeNumber
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
givenName: <first>
sn: <last>
cn: {givenName} {sn}
initials: {givenName:1}{sn:1}
uid: {givenName}.{sn}
mail: {uid}@[maildomain]
userPassword: password
telephoneNumber: <random:telephone>
homePhone: <random:telephone>
pager: <random:telephone>
mobile: <random:telephone>
employeeNumber: <sequential:1>
street: <random:numeric:5> <file:streets> Street
l: <file:cities>
st: <file:states>
postalCode: <random:numeric:5>
postalAddress: {cn}${street}${l}, {st}  {postalCode}
description: This is the description for {cn}.

    (you can download this template here).

    So this is a pretty generic ldap config, nothing complex in it.
    The major things to note here are the various tags. The <first> and <last> tags that lookup first and last names respectively in two files associated with slamd, first.names and last.names MakeLDIF has an algorithim built in to ensure that no combination of first and last names is repeated in the data.
    The tag <sequential> creates sequentially incrementing values, while <random>, as you might guess, generates random numbers. <random> also takes some varying attributes that allow you to create (in this example) phone numbers etc.

    So after creating your template your all set to generate your data. In our case we want to grab the login information as well for our benchmarking, so we run the following 

    java -jar MakeLDIF.jar -t twofifty.template -L logins.txt -o twofifty.ldif

    Well actually, lets time it to show its a nice quick process (this is on an ultra 30). 
    # timex java -jar MakeLDIF.jar -t twofifty.template -L logins.txt -o twofifty.ldif
Processed 1000 entries
Processed 2000 entries
<snip>..........

Processed 249000 entries
Processed 250000 entries
Processing complete.
250002 total entries written.

real       34.60
user       31.41
sys         1.81

    and thats it for MakeLDIF, you end up with an ldif file with your benchmarking data. 

    # ls -l twofifty.ldif  
-rw-r--r--   1 root     3456     150331659 Sep  3 15:33 twofifty.ldif

    MakeLDIF is explained in much more detail in the Slamd Documentation.

    Loading the data

    The data load into the directory server is extremely straight forward. The import cache was increased to 1/2 a gig from the default of 20Mbs, and we just use ldif2db. In this case we are using the userRoot backend db, so the exact command is 
    ./ldif2db -n userRoot -i /export/data/twofifty.ldif

    Slamd Job Data

    Slamd has a large set of available jobs for benchmarking purposes. For this article I will look at just two, the prime job and the searchrate job.

    For the purposes of repeatable benchmarks I tend to lean towards using commandline tools as much as possible, so with Slamd I use the CommandLineJobScheduler.
    First off you have to generate a configuration file, lets say we want to generate the config file for the SearchRate job. 

    #cd /opt/slamd/webapps/slamd/WEB-INF
#java -cp lib/slamd_server.jar:classes:lib/ldapjdk.jar \
        CommandLineJobScheduler \
        -g com.sun.slamd.example.SearchRateJobClass \
        -f /tmp/searchrate.conf
    Which gives you a file such as this.

    We then go and customise this file for our particular config. So in our case we want to specify the specific clients we are going to use, the duration, search base etc. The config file is pretty much self explanatory, but for the sake of completeness I'll stick up the major parameters that I changed here. 

    duration=1200
num_clients=6
requested_clients=jescl1,jescl2,jescl3,jescl4,jescl5,jescl6
num_threads=300
param_ldaphost=ds
param_binddn=cn=Directory Manager
param_bindpw=something_secret
param_searchbase=ou=People,dc=example,dc=com
param_searchscope=One Level Below Base
param_warm_up=300
param_cool_down=300
    A similar type config file is generated for the prime job as well.

    Slamd Client Configs

    For each of the slamd clients we use this config file. The only things really worth mentioning for this example are the following. 
    # Specify the amount of memory to use in megabytes.
INITIAL_MEMORY=256
MAX_MEMORY=512

# Specify the address and port information for the SLAMD server.
SLAMD_ADDRESS=slamdserv
SLAMD_LISTEN_PORT=3000
SLAMD_MANAGER_PORT=3001
SLAMD_STAT_PORT=3003

LOG_FILE=/var/tmp/client.log

    Kicking off the job

    So we have our slamd job config files, our rig is loaded with the ldap data, its time to get some numbers. We start up each of the slamd clients (log into the machine and run /opt/slamd_client/start_client.sh), and then kick off the job using the CommandLineJobScheduler (you can of course do all of this via the web interface as well). 

    java -cp lib/slamd_server.jar:classes:lib/ldapjdk.jar \
        CommandLineJobScheduler \
	-f /tmp/prime.conf
    And then sit back ;).
    Your client logs will show up an entry such as 
    The SLAMD client has started.
Ready to accept new job requests.
Received a request to process job 20040831124550-0897501
Received a request to start processing.
    Once the job has finished you will see something like 
    Done processing job 20040831124550-0897501

    The directory server is now primed, so we repeat the process with the searchrate job, and grab our results.

    A bit further into the automation

    Now obviously enough kicking everything off manually is not exactly desirable, so we have a couple of little wrappers that we throw around the entire process. I won't go into a huge amount of detail, but to do a full benchmarking run in a consistent and coherent manner (logging a bug off one run that can't be reproduced is going to do nothing but irritate people) we have a few steps that we go through.

    First off the system under test is rebooted, once it comes back up it then reboots the slamd server and all of the client machines. Then once these are all back up again it logs into the slamd server, starts up the directory server backend and the slamd server again, then in turn goes to each one of the clients and starts up the slamd client.

    The job is then submitted via the CommandLineJobScheduler and off we go. Once the length of time for the benchmark has passed we monitor the client log files to check that everything has finished, grab the results and start the entire process again.

    OS Tunings

    The group that I work in sets up benchmarks slightly differently than our colleagues in the Market Development Engineering group. We aim to set things up as close to out of the box performance as possible, avoiding things such as sq_max_size=0[1] etc. Rather we will set up the OS as close to what you as a customer would set up (within reason, obviously we don't want things such as i/o wait time on a system, so we may create say a raid 0 stripe for log files or datasets etc).
    The ndd tunings that were used in this config are 
    ndd -set /dev/tcp tcp_conn_req_max_q 1024
ndd -set /dev/tcp tcp_conn_req_max_q 4096
ndd -set /dev/tcp tcp_keepalive_interval 600000
ndd -set /dev/tcp tcp_ip_abort_cinterval 10000
ndd -set /dev/tcp tcp_ip_abort_interval 60000
ndd -set /dev/tcp tcp_smallest_anon_port 8192
    while in /etc/system we set 
    set rlim_fd_max=100000
set rlim_fd_cur=100000
    I guess you could say that we use tunings to make things realistic and also to remove any variance that may arise in repeated runs of the benchmark.

    Directory Server Tunings

    Sun Java System Directory Server has multitudes of things to tune. For the specific example mentioned above I did the following (the eventual tunings went a lot further, but thats for another post).

    First off we did runs with logging disabled and enabled, obviously enough disabling logging makes life a lot faster, but its not that realistic. The result size limit was dropped down to 100 from 2000, operation time limit down to 300, idle timeout set to half an hour, database cache upped to half a gig (this is the most important one to be honest) as was the memory cache.

    And into the future.....

    Well seeing as Adam has just putback plockstat and its associated DTrace provider I just have to play a bit with DTrace on the directory server rig. Once I get some D scripts together I will post them up here.

    Next up for slamd articles, an example of a multitier setup, using several components from the Java Enterprise System.

    [1] sq_max_size=0 should never be used on a production system, see the sq_max_size section of the Solaris Tunables Guide for more details.
    (2004-09-03 10:42:19.0) Permalink Comments [2]

    Comments:

    Excellent article! More stuff like this!

    Posted by Kenneth G on September 06, 2004 at 05:43 PM IST #


    Thanks ;). What other kind of articles would you be interested in seeing. I have a few more on slamd currently in the pipeline (work commitments mean it may be a few weeks before I complete them).
    - Fintan

    Posted by fintanr on September 06, 2004 at 05:57 PM IST #

    Post a Comment:

    Comments are closed for this entry.