Tips for developers Unix Coach

Last night we  had lots of fun increasing our cholesterol and alcohol blood levels at the latest IO-SUG meeting at the Sun offices in Herzliya. A great turnout, we are definitely a growing group.

We had just received the latest official 2009.06 CDs which were quickly snapped up by those present, just in time.

Cyril Plisko and Haim Tzadok seeded the heated discussions with a presentation and demo on Zones. The group was divided between those that were familiar and already enthusiastic users of Zone, and others that had never encountered this and were more used to the Vmware style of virtualization.

Looking forward to the next meeting !

 Some photos from the event, (sorry for the poor quality from my cellphone):

I complete the short series of scripting lessons with a web-loader (that I have used in my day to day work), which utilizes expect for controlling multiple computers, each running a wget script for loading a specific site. You should first read my articles on expect and wget before this one, for the background details.

To simplify the application, I have created two scripts. One script, named "run_agent" is used to login to a slave computer and run the wget loading script (the exact script from my previous column).  A second script, named "run_all_agents" simply runs the first in a loop, one for each slave computer.

In a top-down fashion, assuming we have 3 slave computers, named loader1, loader2 and loader3, the script would look like this:

run_all_agents

 # ‬This script gets one parameter‭, ‬the number of processes to fork on each slave
if‭ [[ $# < ‬1‭ ]]; ‬then‭  # ‬check the number of command line arguments
‭        ‬echo‭ "‬Usage‭: ‬run_all_agents‭ #‬processes(eg‭. ‬1000‭)"‬
‭        ‬exit
fi

start‭=‬`date‭ +%‬H‭:%‬M‭:%‬S`
printf‭ "%‬s‭ " $‬1‭ $‬start
# Slave names are hard-coded for simplicity
‭./‬run_agent loader1‭ $‬1‭ > /‬dev/null‭ ‬&
‭./‬run_agent loader2‭ $‬1‭ > /‬dev/null‭ ‬&
‭./‬run_agent loader3‭ $‬1‭ > /‬dev/null‭ ‬&
wait
end‭=‬`date‭ +%‬H‭:%‬M‭:%‬S`
printf‭ "%‬s‭ " $‬end
echo

The script "run_agent" assumes that the wget loading script is in a fixed location on all slaves ( here /root/Wget/Scripts) Also, that they all have the same ssh login and password, hard-coded here as "root" and "12345".

run_agent

 #!/usr/bin/expect 
if {$argc<2 } { 
   send_user "usage: $argv0 hostname_of_loader no_of_processes\n" 
   exit 
} 
log_user 1 
set loader [lindex $argv 0] 
set nprocs [lindex $argv 1] 
set timeout ­1 
spawn ssh root@$loader 
expect "word:" 
send 12345\r 
expect "#" 
send "cd /root/Wget/Scripts\r" 
expect "#" 
send "bash ./wload $nprocs\r" 
expect "#" { 
    set Output $expect_out(buffer) 
    send_user $Output 
} 
exit

So, here we have create a scalable loader in less that 100 lines of code by taking advantage of the wonderful Unix philosophy of easy gluing together of multiple dedicated and efficient facilities, as opposed to writing a single monolithic program.

So, you have built a great web-site with PHP or Java, and you have tested it by accessing the page in your browser. It seems to pop-up really quickly, do you think it is ready for deployment ? What if 100 people access it at the same time, or 10,000 ? What would its response time be ? If it is too low, forget about it, no-one has time these days to wait more than a few seconds before jumping to the next item in the google search list.

Wget is a great tool for scripting programs that download content from the Web. I often use it as a "quick and dirty" tool for loading a web-site in order to test and monitor its performance.

There are many commercial and open-source tools available, but none are quite as simple as wget, which has a zero learning curve and probably the least amount of bugs.

Here is a simple bash script (save as "wload") that forks multiple processes that each access your site ,  and prints out the total time taken for all requests to complete. Note that it uses a file named pages.txt which contains a list of pages that you want each process to download (one address per line ):

if [[ $# < 1 ]]; then 
        echo "Usage: wload #processes(eg. 1000)"
        exit
fi
start=`date +%H:%M:%S`
for ((i=1; i<=$1; i++))
do
  wget -o /dev/null -q -O /dev/null -i pages.txt &
done
wait
end=`date +%H:%M:%S`
printf "%s " `hostname` $1 $start $end
echo

Note that it prints out the hostname of the loading machine, since I expect to run this script on multiple machines to further increase the loading capacity. Also, I write the files to /dev/null to remove any possible I/O overhead.

An example of a pages.txt file:

http://192.168.1.100/index.html
http://192.168.1.100/link1.html
http://192.168.1.100/img1.jpeg
http://192.168.1.100/link2.html
http://192.168.1.100/img2.jpeg

Next week, I will show how to use Expect to create a controlling program that spawns these scripts on multiple machines, and collates the results for ease of use.

Have you ever needed to control multiple computers ?
If so, don't you get tired with the endless cycle logging in and running your commands manually ?
Wouldn't it be great to automate this by a script ?

One of my favorite tools for scripting tests that require multiple networked computers is Expect, a set of extensions to the TCL scripting language. It enables automation of repetitive tasks usually required to be performed manually.

Basically, it lets you script an interactive session, by sending keystrokes, as well as giving you a rich library of parsing procedures to help your script understand what is being written back, and then letting your script answer.

Example #1

Assume you needed to list the running processes on a remote machine. You would probably need to do the following manually:

ssh username@machine
Wait for password prompt
Type in your password
Wait for the shell prompt
Type in the command ps
Wait for output to complete
View results
Exit the ssh shell

This is not very complicated, but imagine if you needed to do this on say 10 machines, boring ...
Instead just type in the following script, and send it as a parameter the hostname, username and password to login. Say we save the file as "remote_ps", then you would execute it as follows:

$ remote_ps host user password

I'll bet you agree that this is a lot less typing than previously.

So, here is the script ...

#!/usr/bin/expect
set host [lindex $argv 0]
set user [lindex $argv 1]
set password [lindex $argv 2]
spawn ssh ${user}@${host}
expect "word: "
send ${password}\r
expect "$"
send "ps\r"
expect "*$"
send "exit\r"
expect eof

By the way, expect is part of the default installation on Mac OS, on OpenSolaris you will need to get the package. Just use the built in package manager and search for expect, it is in the repository.

The Israeli OpenSolaris User Group 

JavaEdge Conference

The purpose of these articles is to quickly get an overview of the tools that we use for basic performance monitoring of Solaris systems. No prior experience with Unix is assumed, just as long as you know how to open up a terminal and type in commands, you will be able to run these tools.  Of course access to a Solaris machine is mandatory, if you don't have this, I would suggest installing VirtualBox on your computer, and install OpenSolaris inside it.

I belong to a team of engineers at Sun (ISV engineering) that spend a lot of time helping our partners discover the bottle-necks of their applications, and helping them to improve performance by taking full advantage of the machines capabilities. 

Before diving into the tools themselves, let's first look at the elements of a generic computer, and then we'll associate the related tool with the appropriate element.

I like to divide a computer into the following primary sub-systems:

1. Processors: From simple single processor, single core to multi-processors each with multiple cores. We see a trend of computer manufacturers moving to multi-core processors, and less and less computers, mainly low-end laptops are now only single-core machines. This is very important, since it is harder to design software to take advantage of multiple cores, than writing simple programs which are single-threaded. The main tool I used for monitoring processors is mpstat.
   
2. Memory: Computer systems can have a large variety of different memory architectures, composed of fast caches( level 1, level 2, level 3), buffers, main-memory and solid-state memory. We also have NUMA machines (non-uniform memory access), where different banks of memory are further away from the CPU, and take longer to access. For example in multi-processor AMD machines,  each CPU has its own bank of memory, directly connected. Each CPU however has access to all memory banks of other CPUs, but this access takes longer. The first tool I use for monitoring memory is vmstat.
   
3. Disks: The storage sub-system of the computer can be composed of multiple disks, possibly in various RAID configurations. All disks have memory caches on-board to speed up repeated accesses, in addition the operating system maintains its own buffers with various look ahead strategies to minimize actual I/O operations. The main tool I use for monitoring the disk sub-system is iostat.
   
4. Networks: Most applications depend on an efficient networing infrastructure, and most servers come with multiple network interfaces that can be aggregated to increase throughput.  We want to measure the load on the interfaces, and often also need to measure the packet latencies. Netstat is the main tool I use for monitoring the network packet rates, and snoop for looking at the packets themselves.