..bits & bytes teleported

The DTrace community doesn't really need another blog post to introduce the technology to folks who are into this kind of a stuff. Why this blog post then ? Please read on. (Note that, this is strictly for newbies of various things)

Scenario #1: I had to explain DTrace to my friend who was going to talk about some JVM + DTrace stuff in a presentation in her college. I suggested her the topic. Now the hard part was to even explain the basic idea behind DTrace and convince her enough so that she in turn could convince her evaluators- both parties being almost completely ignorant about DTrace. 

Scenario #2: I have been exploring a bit about MySQL + DTrace and was going to talk on it at the Bangalore MySQL UG meet yesterday. As expected, the audience was completely ignorant about DTrace and perhaps, of more importance- all of they ran MySQL on Linux for their business. Ah now things gets interesting. I wanted to 'wow' them with DTrace

One common thing between these, is that the audience in both cases are DTrace newbie. Before delving right into what DTrace can give us with all the technical details of probes, providers, consumers, production systems, et cetra , its very useful to set the stage with some real life analogies is set for the audience to move on to the excellent technical manuals available out there.

The remaining part of this post has the text from which my words were derived in both cases:

To introduce DTrace, you will have to speak about the importance of run time observability of production systems. Perhaps, that sentence was not very informative, however, please stay with me. I shall explain it.

Production systems:  We call a production system as such, when it is used to provide any kind of consumable service to consumers. Consider a scenario: the billing system in a departmental store serves all the billing terminals- the ones at which you get your stuffs billed. The billing system is perhaps run off a single powerful computer or a network of many powerful computers housed somewhere in the store or more likely in a central location, not in the store. The billing system is an example of a production system. Its in use and you cannot
take it down. If you do or it decides to do so on its own, you will face the wrath of customers and bear losses in your business.

Observability: In simple words, observing a system means to learn to see the various ways your program- a 10 line program or a billing system- is behaving at run time. Now, what do we mean by 'behaving' ?
Let's say, our program is now using 10% of the total CPU and other point of time it is using 90% of the CPU. That's one behavior of the program. More simply, let's say, for input X, the program is in function, F and for input Y the program goes to function, G.

Observing a system during its run time is hence also called dynamic observation of the system, 'cause it happens in real time. Since we basically 'trace' the behavior of the system at various instants of time, we can also refer to it as 'Dynamic Tracing'.

Off Radioactive elements and DTrace

Tracing of a system is a lot similar to the way Radioactive elements are used as tracers in human body to identify various abnormal body processes. Our topic of interest here, DTrace works a lot like this.

Using the DTrace framework, we can implant probes (or tracing points, like 'radioactive tracers') in our programs and can subsequently monitor those points for activities of interest. What we are interested in here, is Live Analysis and not post mortem analysis

I would also like to suggest you to go through the slides that I used for the talk. The later slides are MySQL specific, but the first few give some general information. It also incorporates some information that I was suggested by Richard on the dtrace-discuss list. Thanks to Michael and James for their suggestions. James also suggested some basic DTrace scripts that I could use to wow the audience. However, I will save them for the getting started session. I have just aquainted them.

Now its time to read http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_Intro

I hope you find this post useful, and appreciate your comments.

Release Candidate of OpenSolaris 2008.11 is out. We shall take it for a spin using Sun xVM VirtualBox 2.0 on Ubuntu 8.10.

                  +                         +         


Download the OpenSolaris 2008.11 RC1 ISO from here:

Create a new Virtual Machine using VirtualBox with the main points of the configuration, as follows:

• Make sure you set up  your VM to boot from the ISO you just downloaded
  
• I have allocated ~ 1.5 GB of primary memory to OpenSolaris. You may give more, or less (1 GB is recommended)
• I have allocated a Virtual Hard disk of 21 GB. You may use less, depending on your usage. OpenSolaris 2008.11 when installed takes about ~ 2.5 GB of disk space
• I want to use my USB devices on OpenSolaris, so I have enabled them. You may *not* enable them
  

Start the VirtualBox and you will be greeted with the GRUB boot menu. Select the first option:




The Live session is now ready for use:




To start the installation, Double-Click on the 'Install OpenSolaris' Desktop icon. The installer starts, showing the current disks available. Select the one you want to use and also choose the mode of partitioning. I am using:  Use the whole disk.



Then, in a next step you will have to provide access credentials:




The installation process should now start and enjoy the eye-candy tour of the various features:




Time-Slider:









The installation finishes up after sometime, and you are asked to reboot:



On reboot, you will be pleasantly surprised to see the white characters on black screen no more there. What you see is a cool blue background with a progress indicator in the foreground:




After you are done logging in at the GDM, your OpenSolaris session is ready to use:



VirtualBox Guest Additions:

To get the Full Screen mode working properly and better mouse integration, install the VirtualBox Guest Additions:

• Go to Devices > Mount CD/DVD ROM Image and Add the VirtualBox Guest Additions ISO
• It will be mounted under /media/VBOX-*
• Navigate to that directory, and do:   pfexec pkgadd -d VBoxSolarisAdditions.pkg
  

Join in the discussion at http://www.opensolaris.org/jive/thread.jspa?threadID=78357&tstart=0

Monthly newsletter by, for and about the OpenSolaris Community: http://www.sun.com/emrkt/opensolaris/ignite/index.html

First Issue: October 2008

Subscribe Now

http://kenai.com/projects/zback

Hidden in Plain Sight is a ACM publication on DTrace and is a good reference document to understand the functional architecture of DTrace and how it enables dynamic code instrumentation.

A very exciting recent development in the OpenSolaris world is a Hadoop Live CD (What is Hadoop?) based on OpenSolaris.

What does it have?

The CD image provided gives users an environment emulating a fully distributed, three-node virtual Hadoop cluster. One of the reasons we used OpenSolaris is its ability to emulate a multinode cluster environment in a very small memory foot print. A three-node Map/Reduce cluster can be brought up on a machine with as little as 800 MB of memory. Each additional virtual cluster node only requires about 40 MB of additional memory, in addition to the memory used by Hadoop. This means that people can take Hadoop for a spin, even on their laptop.

A quick look at the Quict Start Guide shows that, it uses 3 Solaris zones- a Global zone for job tracking and 2 worker zones for carrying out the jobs.

Links:

• Project Page: http://opensolaris.org/os/project/livehadoop/
• Downloads: http://opensolaris.org/os/project/livehadoop/Downloads/
• VirtualBox allows you try the ISO without burning it to a CD

When multiple processes are running on a single CPU system, context switches often happen between them.

Here is a simple D script which can be used with DTrace to collect some information related to which process is on the CPU during a time interval:

#pragma D option quiet

sysinfo:::pswitch
{
/* who got the CPU */

printf("\n%s got the CPU", execname); /* execname is a DTrace 'In-Built' variable whose value is the currently executing process */

}

Watching it in action:

Save the above file to- whogotthecpu.d, and run it using:

# dtrace -s whogotthecpu.d

You should see the output like this:

dtrace got the CPU
kdeinit got the CPU
sched got the CPU
sched got the CPU
dtrace got the CPU
kdeinit got the CPU
sched got the CPU
sched got the CPU
kdeinit got the CPU
sched got the CPU
sched got the CPU
kdeinit got the CPU
sched got the CPU
kdeinit got the CPU
Xorg got the CPU
sched got the CPU
sched got the CPU
sched got the CPU
sched got the CPU
kdeinit got the CPU
sched got the CPU
Xorg got the CPU
sched got the CPU
fsflush got the CPU
.
.

How does it work?

• We make use the sysinfo probe provider, which has a probe by the name 'pswitch' which is triggered whenever a process switch happens on the CPU. This is indicated by the line:
  

  sysinfo:::pswitch
   

• When the probe is fired, we simply print the name of the process.

How many times?

Let us now add a line to the script so that it also tells us the number of times each process got the CPU.

We do this by making use of a DTrace aggregate- count(). We simply add this line:

@[execname]=count() 

to the script. The script should now look like this:

#pragma D option quiet

sysinfo:::pswitch
{
/* who got the CPU */

printf("\n%s got the CPU", execname);


/*.. and how many times? */

@[execname]=count()
}

Run it as earlier, and you should see output similar to this:

.
.
.
.
dtrace got the CPU
kdeinit got the CPU
sched got the CPU
sched got the CPU
dtrace got the CPU
kdeinit got the CPU
sched got the CPU
sched got the CPU
kdeinit got the CPU
sched got the CPU
sched got the CPU
kdeinit got the CPU
sched got the CPU
kdeinit got the CPU
Xorg got the CPU
sched got the CPU
sched got the CPU
sched got the CPU
sched got the CPU
kdeinit got the CPU
sched got the CPU
Xorg got the CPU
sched got the CPU
fsflush got the CPU
devfsadm got the CPU
sched got the CPU
Xorg got the CPU
kdeinit got the CPU
sched got the CPU
kdeinit got the CPU
  korgac                                                            1
  kpdf                                                              1
  devfsadm                                                          2
  kwrapper                                                          2
  nscd                                                              2
  fsflush                                                           3
  dtrace                                                            6
  Xorg                                                             24
  kdeinit                                                          40
  sched                                                           150

For learning about DTrace, visit these links boomarked at http://delicious.com/amitkumarsaha/dtrace

• Part 1: Introduction
• Part 2: Scripts and the D Language
• Part 3: Advanced Scripting and Aggregations
• Part 4: Solving Everyday Problems
• Part 5: Completing the Picture

For a detailed introduction to DTrace, refer: http://www.sun.com/software/solaris/howtoguides/dtracehowto.jsp

Here is a paragraph from the guide above which serves as a good very-short way to introduce some one to DTrace-

DTrace dynamically modifies the operating system kernel and user processes to record data at locations of interest, called probes. A probe is a location or activity to which DTrace can bind a request to perform a set of actions, like recording a stack trace, a timestamp, or the argument to a function. Probes are like programmable sensors scattered all over your Solaris system in interesting places. DTrace probes come from a set of kernel modules called providers, each of which performs a particular kind of instrumentation to create probes.

DTrace Script: A DTrace script is a set of instructions written in the D scripting language which is fed to the DTrace framework for carrying out relevant DTrace tasks

Components of a D script:

1. Probes
2. Predicates
3. Actions
   

DTrace is somewhat similar to event-handling framework in GUI programming. You declare some probing points. When the probing points or situation is encountered, the predicate is matched and if the predicate macthes, the actions are executed.

For example consider the following simple D-script

syscall::write:entry 
/execname == "bash"
{
printf("bash with pid %d called write system call\n",pid)
}

For the above script:

DTrace Script Component	Relevant part in the script above
Probe	syscall::write:entry
Predicate	execname=="bash"
Action	printf("bash with pid %d called write system call\n",pid);

Save the script to a file, say 'demo-1.d', and execute the D script using: (the '-s' switch indicates that the next value is the script file)

amit@opensolaris:~/scripts/D# dtrace -s demo-1.d

dtrace: script 'demo-1.d' matched 1 probe
CPU     ID                    FUNCTION:NAME
  0  60983                      write:entry bash with pid 5263 called write system call

  0  60983                      write:entry bash with pid 5263 called write system call

  0  60983                      write:entry bash with pid 5263 called write system call

  0  60983                      write:entry bash with pid 6692 called write system call

  0  60983                      write:entry bash with pid 5263 called write system call

What is this '60983'?

It is called the probe id.

dtrace -l lists all the probes available on the system. So a simple grep shows up the result:

amit@opensolaris:~/scripts/D# dtrace -l | grep 60983
60983    syscall                                               write entry

Predicates and Actions  in a Dtrace script can be ommited.

For eg. when the action is ommitted, it just prints the probe name:

amit@opensolaris:~/scripts/D# dtrace -s demo-1.d
dtrace: script 'demo-1.d' matched 1 probe
CPU     ID                    FUNCTION:NAME
  0  60983                      write:entry
  0  60983                      write:entry
  0  60983                      write:entry
  0  60983                      write:entry
  0  60983                      write:entry
  0  60983                      write:entry
  0  60983                      write:entry
  0  60983                      write:entry
  0  60983                      write:entry
  0  60983                      write:entry
  0  60983                      write:entry
  0  60983                      write:entry
  0  60983                      write:entry
  0  60983                      write:entry
  0  60983                      write:entry
  0  60983                      write:entry
^C

As said in the beginning, a very useful introduction to DTrace is at http://www.sun.com/software/solaris/howtoguides/dtracehowto.jsp which points to other several resources.

Yet another OpenSolaris 2008.05 review- this time by a Linux magazine- Linux Format.

They call it the- 'The Linux Format Test' and is up here.

• The review is much appreciative of ZFS and opines OpenSolaris 2008.05 as a definite check-out
• It suffers from poor mistakes- IFS for Image Packaging System- WTF! and Ian Murdoch !

We will use this space to help the growing community of OpenSolaris users with tips on installation, configuration, applications, etc. The Observatory is not just a blog aggregator. We will be providing original content and also scanning the web for other great tips. Ideally, The Observatory will become a primary source of information on how best to use OpenSolaris.

----- The space that is being talked about here is The Observatory

This blog has been started by three open source enthusiasts: Brian Leonard, Gregg Sporar and Roman Strobl. Our passion is evangelizing OpenSolaris. Some of you may know us from our previous roles where we focused on NetBeans (see Brian's blog, Gregg's blog and Roman's blog)

 This is great news! 

The blog already has an entry http://blogs.sun.com/observatory/entry/top_5_unsuccessful_searches_on to address the most common unsuccessful searches for OpenSolaris 2008.05:

• 1- sudo
• 2- top
• 3- mplayer
• 4- vlc
• 5- mc

Make sure you subscribe to the feed. I love this Roman!

Firefox 3 is now available for use to BeleniX 0.7 users from http://www.belenix.net/binfiles/SUNWfirefox.tar.gz

Here is the announcement by Moinak

Instructions:

1. Uncompres: tar -zxvf SUNWfirefox.tar.gz
2. Remove existing Firefox packages: pkgrm SUNWfirefox SUNWfirefox-root
3. Add new package: pkgadd -d . SUNWfirefox

Solaris 10, SXDE, OpenSolaris, Project Indiana are terms which often boggle the newbie into Solaris. This 10-min nettalk 'Sorting out Solaris Releases' clears up the encumbrances about the terms and gives you a clear view of where Solaris is now and where is it heading. 

Thanks Angad for pointing it out. 

Related:

If you are a relative newbie into Solaris, you migh also be interested to know: What is Nevada?

Sometime back, I had written an article titled- Introducing strace - a System call tracing and Signal reporting tool for system call tracing on Linux systems.

I had a brief look at it, but seems like 'truss' performs a similar thing (more? or less?) on Solaris systems. Here's a quick look:

Consider the simple C code snippet below:

  
int main()
{
printf("hello world");
return 0;
}

After you have the object code, execute the binary (here 'a.out') using

$ truss a.out

You should see a output similar to this:

  
  
  
execve("a.out", 0xFFBFEA6C, 0xFFBFEA74)  argc = 1
resolvepath("/usr/lib/ld.so.1", "/lib/ld.so.1", 1023) = 12
getcwd("/home/as227057/dev-tools/mysql-install/libexec", 1017) = 0
resolvepath("/home/as227057/dev-tools/mysql-install/libexec/a.out", "/home/as227057/dev-tools/mysql-install/libexec/a.out", 1023) = 52
stat("/home/as227057/dev-tools/mysql-install/libexec/a.out", 0xFFBFE848) = 0
open("/var/ld/ld.config", O_RDONLY)             Err#2 ENOENT
stat("/usr/sfw/lib/libc.so.1", 0xFFBFE300)      Err#2 ENOENT
stat("/lib/libc.so.1", 0xFFBFE300)              = 0
resolvepath("/lib/libc.so.1", "/lib/libc.so.1", 1023) = 14
open("/lib/libc.so.1", O_RDONLY)                = 3
mmap(0x00010000, 32768, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_ALIGN, 3, 0) = 0xFF3A0000
mmap(0x00010000, 1015808, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFF280000
mmap(0xFF280000, 905589, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_TEXT, 3, 0) = 0xFF280000
mmap(0xFF36E000, 32401, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_INITDATA, 3, 909312) = 0xFF36E000
mmap(0xFF376000, 6368, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANON, -1, 0) = 0xFF376000
munmap(0xFF35E000, 65536)                       = 0
memcntl(0xFF280000, 144068, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3)                                        = 0
mmap(0x00010000, 24576, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFF390000
munmap(0xFF3A0000, 32768)                       = 0
getcontext(0xFFBFE538)
getrlimit(RLIMIT_STACK, 0xFFBFE518)             = 0
getpid()                                        = 5251 [5249]
setustack(0xFF392A88)
ioctl(1, TCGETA, 0xFFBFDB9C)                    = 0
fstat64(1, 0xFFBFDAB8)                          = 0
stat("/platform/SUNW,Sun-Fire-T200/lib/libc_psr.so.1", 0xFFBFD660) = 0
resolvepath("/platform/SUNW,Sun-Fire-T200/lib/libc_psr.so.1", "/platform/sun4v/lib/libc_psr.so.1", 1023) = 33
open("/platform/SUNW,Sun-Fire-T200/lib/libc_psr.so.1", O_RDONLY) = 3
mmap(0x00010000, 32768, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_ALIGN, 3, 0) = 0xFF270000
munmap(0xFF272000, 24576)                       = 0
mmap(0x00000000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFF3A0000
close(3)                                        = 0
hello worldwrite(1, " h e l l o   w o r l d", 11)               = 11
_exit(0)