Alan Hargreaves' Weblog

Nailing Down a Troll

#opensolaris was invaded by a troll today. Not the first time, certainly won't be the last. What is interesting is that we managed to nail this one down on a timeframe for one of his predictions (well he appears to believe that it's a fact).

Philv (I've saved the email address and full name so I can remind this individual of his comments in 8 months time) firmly believes that in 8 months or less from the posting of this blog entry that Microsoft will buy Sun.

The conversation excerpts from #opensolaris at freenode.org follows.

philv: Much better than Sun's, otherwise SUN wouldn't be in talks with
       Microsoft over a buyout
philv: Microsoft is buying SUN.
Tpenta: references?
philv: The Wallstreet Journal
Tpenta: url please
philv: I don't read online news, it could be hacked.
philv: No, you just wait and see.
philv: And when I turn out to be right, you'll all be crying.
philv: And I'll be laughing
Tpenta: ok phil. put up or shut up time. Give me a time frame. If the
        time frame elapses, I'd like to see an admission of "oops i was
        wrong"
philv: Tpenta: 8 months.
philv: Maybe less.
Tpenta: ok, so in 8 months I *expect* to see an oops I was wrong from
        you if it does not occur, correct?
philv: No, because FreeNode will be gone by then.
Tpenta: Phil. I'm posting a blog entry on this today, I will be bringing
        this up again (with your name) in 8 months time, you will only
        need to comment on the blog

It will be interesting to see what Philv has to say in 8 months time.

Philv, you will note that I have kept your name and email address out of this entry. I'll grant you that much in the way of privacy, but you should expect an email in eight months from today. By the way, if you turn out to be correct, I will be more than happy to say "Oops I was wrong".

Technorati Tags: Troll, Irc, Sun

Posted at 03:35PM Jun 30, 2005 by Alan Hargreaves in General  |  Comments[5]

Out of Context Sound Bites

Christopher Blizzard did a quick blog entry about one of Scott's comments in this interview.

The following is the text of an email that I've just sent off, then thought that the comments really need to be a bit more public.

I know that I have an obvious bias here, but it might have been nice if you placed the McNealy quote in context. You didn't even mention that he went on to explain and justify it.

The following paragraph was

Let me justify that because I don't think (Gore) justified his (comments about inventing the Internet). I think I can justify ours. (Former Sun Chief Technologist) Bill Joy, as far as I can tell, kind of pioneered the whole concept of open-source kernels at (Berkeley Software Distribution) and created the licensing mechanism. We brought him into Sun, and we were kind of the Red Hat of Berkeley software before (Linux kernel inventor) Linus (Torvalds) was out of diapers. So we've been doing this forever.

Out of context sound bites are nice for the press, but as technical people I had hoped that we were above that kind of thing.

I actually think that this is one of the most candid interviews that Scott's done in an awful long time.

I know that there are a lot of Linux folk out there who like to beat up on Sun, Scott and Jonathan. Fair enough, they are public figures. That's part of the job description. However, ridiculing someone by quoting them out of context is gutter press stuff.

Keeping discussion in context, and better yet, sticking to the technical bits which we are all much better at, raises the level of the discussion and lends credibility to all participants.

technorati Tag: Sun

Posted at 07:26PM Jun 29, 2005 by Alan Hargreaves in General  | 

Non-debug Opensolaris

A little more progress today, as shown by the following flash screen grab.

You may notice the absence of the string "DEBUG Enabled" after the initial Solaris Banner.

Be patient, there is a nice long wait in the middle of this one and the astute of you will notice that for some reason my '-' key doesnt work while I'm recording this.

So the pieces are starting to fall together. I just have to get the automation working properly and straighten out some inconsistancies. That will have to wait as I am taking tomorrow off as my little boy is receiving an award at a school assembly and that's far more important :)

Technorati Tag: opensolaris

Posted at 10:35PM Jun 27, 2005 by Alan Hargreaves in OpenSolaris  |  Comments[3]

Non-debug builds of Open Solaris not far away

I've been having a look at what needs to happen to make this work. My main aim in doing this is to make good application of the KISS Principle. I wanted to make minimal change for the best effect.

I've had the answer for bfu and Install for a while now (and the modifications to those are only a matter of a few lines with some smart ksh variable substitution), but it was waiting on us coming up with a simple way of generating the closed binaries tarchive for both the debug and non-debug builds.

A few ideas had occured to me, but they all had largish changes to the way things worked involved, and I really didn't want to go that way.

Without going in to too much detail (as I haven't had a chance to speak to Mike Kupfer about it yet), it should involve a pretty simple addition to the build() function in nightly.sh script to generate the tarchives as a part of a normal nightly run inside Sun if the appropriate flag is set in the environment script for nightly.sh.

Mike already has a script for generating the debug tarchive. If we run this during a nightly on a non-debug build, we should end up with a non-debug set of closed binaries. This will definitely help with folks already attempting to benchmark!

With luck we might have this going shortly.

Technorati Tag: OpenSolaris

Posted at 07:15PM Jun 23, 2005 by Alan Hargreaves in OpenSolaris  |  Comments[1]

July Australian Personal Computer Magazine Features Solaris 10

This one has been in the works for a couple of months.

Some time back we were approached by the magazine to do a feature issue on Solaris 10, including a copy of Solaris 10 on the DVD that ships with each issue.

It's happened with the July Issue.

The DVD includes the four iso images that comprise Solaris 10 FCS for x86 and amd64.

Reviews from APC staff include

"Solaris is a flexible and powerful OS that can run everything from a home PC to massive multi-CPU servers. Now it is free,Solaris seems a viable option for small to medium businesses wanting a secure and stable OS. To cap it off, a great office suite - StarOffice 7 - is bundled with the program." Tony Sarno, Editor, APC Magazine

"Sun's Solaris is an operating system that typically powers serious computing infrastructures. It is the most successful and widespread commercial Unix OS in the world.... For all its power, Solaris installs as easily on a PC as Windows XP or SUSE Linux." Peter Sbarski, APC reviewer

Features include

• a 2 page article on Sun's strategies with Solaris titled, "Open for Business",
• a first installment of an on-going workshop feature, this month titled, "Solaris, so good", 4 pages instructions on installation.

Also included is installation support for one month from Sun Australia. More information about support can be found by purchasing the magazine and finding the link information enclosed therein. Also at that link you will find pointer to how to install Solaris in a dual boot configuration with Windows XP.

Technorati Tags: Solaris, x86, amd64, magazine, unix, Sun Microsystems.

Posted at 11:07AM Jun 15, 2005 by Alan Hargreaves in Solaris  |  Comments[2]

I can feel it coming in the air tonight

With apologies to Phil Collins ;)

The following have appeared in the last hour or so.

• Sun begins open-source Solaris era
• Discussion on the above news item.
• Sun to Release OpenSolaris Code, New Developer Web Portal
• Sun releasing OpenSolaris technologies via open source
• Sun Micro to start publishing Solaris source code
• Voices of OpenSolaris (well it appeared a few days ago)

Stay Tuned.

Technorati Tags: OpenSolaris, Solaris

Posted at 02:59PM Jun 14, 2005 by Alan Hargreaves in OpenSolaris  |  Comments[2]

How do Solaris Filesystems Update Statistics Without Intimate Knowledge of the cpu Structure?

  cpu_stat.cpu_sysinfo.bread	/* physical block reads			*/
  cpu_stat.cpu_sysinfo.bwrite	/* physical block writes (sync + async)	*/
  cpu_stat.cpu_sysinfo.lread	/* logical block reads			*/
  cpu_stat.cpu_sysinfo.lwrite	/* logical block writes			*/
  cpu_stat.cpu_sysinfo.bawrite	/* physical block writes (async)	*/
  cpu_stat.cpu_vminfo.pgin	/* pageins				*/
  cpu_stat.cpu_vminfo.pgpgin	/* pages paged in			*/
  cpu_stat.cpu_vminfo.anonpgin	/* anon pages paged in			*/
  cpu_stat.cpu_vminfo.execpgin	/* executable pages paged in		*/
  cpu_stat.cpu_vminfo.fspgin	/* fs pages paged in			*/
  cpu_stat.cpu_vminfo.maj_fault	/* major page faults			*/

/*
 * Allocate and initialize a buf struct for use with pageio.
 */
struct buf *
pageio_setup(struct page *pp, size_t len, struct vnode *vp, int flags)

/*
 * Common code for reading a buffer with various options
 *
 * Read in (if necessary) the block and return a buffer pointer.
 */
struct buf *
bread_common(void *arg, dev_t dev, daddr_t blkno, long bsize)

/*
 * Read in the block, like bread, but also start I/O on the
 * read-ahead block (which is not allocated to the caller).
 */
struct buf *
breada(dev_t dev, daddr_t blkno, daddr_t rablkno, long bsize)

/*
 * Common code for writing a buffer with various options.
 *
 * force_wait  - wait for write completion regardless of B_ASYNC flag
 * do_relse    - release the buffer when we are done
 * clear_flags - flags to clear from the buffer
 */
void
bwrite_common(void *arg, struct buf *bp, int force_wait,
				int do_relse, int clear_flags)

/*
 * Release the buffer, marking it so that if it is grabbed
 * for another purpose it will be written out before being
 * given up (e.g. when writing a partial block where it is
 * assumed that another write for the same block will soon follow).
 * Also save the time that the block is first marked as delayed
 * so that it will be written in a reasonable time.
 */
void
bdwrite(struct buf *bp)

Posted at 01:00AM Jun 10, 2005 by Alan Hargreaves in OpenSolaris  | 

DTrace - Using (placing) SDT probes

This is a slight re-write of the talk that I gave at the SOSUG meeting in Sydney on May 31.

I've also included in this entry some demonstrations that I've done up using flash and vnc2swf. These are all set up to neither automatically play nor loop. In order to start them you will need to bring up the flash menu (right button insider the demo) and select play. Hopefully I can find a better way to do this in the future.

Updates

June 6 - Thanks to Peter, I now have Start buttons on the demos.

June 7 - A question in the comments propted me to talk a little more about what the assembly looks like and how much space a probe adds to a binary. See the Comments section on this entry.

Overview

• What is a SDT probe?
• Example - kernel io provider
• Slight Digression - kernel mib provider
• Example - User Space

What is an SDT Probe?

It stands for Statically Defined Tracing Probe.

In reality, with the exception of fbt and pid probes, all probes are static. They just don't appear under the sdt provider as they have been written to be under other providers. A provider is simply a grouping of probes.

The fact that they are static probes does not detract from the dynamic nature of DTrace, which refers to the fact that they can be dynamically enabled.

These probe points can be placed anywhere in the code in order to make that code more observable.

There are a number of probes that do exist under the sdt provider. On the notebook that I write this on there are around 200.

   ID   PROVIDER    MODULE                          FUNCTION NAME
  460        sdt      unix         page_get_replacement_page page-get
  461        sdt      unix                page_get_cachelist page-get
  462        sdt      unix                 page_get_freelist page-get
  475        sdt      unix                       lgrp_choose lgrp_choose_start
  490        sdt      unix              av_dispatch_autovect interrupt-complete
  491        sdt      unix                       intr_thread interrupt-complete
  492        sdt      unix                            cmnint interrupt-complete
  494        sdt      unix              av_dispatch_autovect interrupt-start
  495        sdt      unix                       intr_thread interrupt-start
  496        sdt      unix                            cmnint interrupt-start
  513        sdt   genunix                   callout_execute callout-end
  514        sdt   genunix                   callout_execute callout-start
  550        sdt   genunix                 kcpc_pcbe_tryload kcpc-pcbe-spec
  588        sdt   genunix                  priv_policy_only priv-ok
  589        sdt   genunix                priv_policy_choice priv-ok
  590        sdt   genunix                       priv_policy priv-ok
  591        sdt   genunix                  priv_policy_only priv-err
  592        sdt   genunix                priv_policy_choice priv-err
  ...

A little background on the output of dtrace -l as seen above.

The first column is the probe number. This number will vary from system to system depending on their order of creation. You generally won't need to know this.

The second column is the provider. As mentioned earlier, the provider is simply a grouping of probes. For example, mib, io, sdt and sched.

Next we have the module. In kernel space, this will be the kernel module in which the probe resides. In user space it will be the object. For example a.out, libc, ...

Then we have the probe function. This is simply the name of the function into which the probe was written.

Finally there is the probe name. The name given to the probe.

So a probe is made up of a 4-tuple of these.

probeprov:probemod:probefunc:probename

Incidentally, the names used in the above layout are the variable names that can be used from within the D language to access the particular information about the probe.

To add a probe to the code we use the DTRACE_PROBE macros by first including <sys/sdt.h>

As a part of loading the module (in kernel space) or building the object (in user space) the function call generated by this macro is replaced by a number of nop instructions.

Kernel SDT probes looks like

    DTRACE_PROBE(name);
    DTRACE_PROBE1(name, type1, arg1);
    DTRACE_PROBE2(name, type1, arg1, type2, arg2);
    DTRACE_PROBE3(name, type1, arg1, type2, arg2, type3, arg3);
    DTRACE_PROBE4(name, type1, arg1, type2, arg2, type3, arg3, type4, arg4);

Each probe has a name and zero or more type/argument pairs. Each of these pairs describe a variable that is passed to the probe.

User SDT probes look a little different.

    DTRACE_PROBE(name);
    DTRACE_PROBE1(name, arg1);
    DTRACE_PROBE2(name, arg1, arg2);
    DTRACE_PROBE3(name, arg1, arg2, arg3);
    DTRACE_PROBE4(name, arg1, arg2, arg3, arg4);

You'll notice that we don't provide the type information in the code. I'll get to how we define them later on.

OK, so let's look at some examples, starting in kernel space.

The io provider

This shows all kinds of nifty stuff when dealing with I/O.

 $ dtrace -l -P io
    ID   PROVIDER            MODULE                 FUNCTION NAME
  521         io           genunix                  biodone done
  522         io           genunix                  biowait wait-done
  523         io           genunix                  biowait wait-start
  532         io           genunix           default_physio start
  533         io           genunix            bdev_strategy start
  534         io           genunix                  aphysio start
 1263         io               nfs                 nfs4_bio done
 1264         io               nfs                 nfs3_bio done
 1265         io               nfs                  nfs_bio done
 1266         io               nfs                 nfs4_bio start
 1267         io               nfs                 nfs3_bio start
 1268         io               nfs                  nfs_bio start

The io provider is implemented in sys/sdt.h as wrappers around the DTRACE_PROBE macros.

#define DTRACE_IO(name)                                         \
        DTRACE_PROBE(__io_##name);

#define DTRACE_IO1(name, type1, arg1)                           \
        DTRACE_PROBE1(__io_##name, type1, arg1);

#define DTRACE_IO2(name, type1, arg1, type2, arg2)              \
        DTRACE_PROBE2(__io_##name, type1, arg1, type2, arg2);

#define DTRACE_IO3(name, type1, arg1, type2, arg2, type3, arg3) \
        DTRACE_PROBE3(__io_##name, type1, arg1, type2, arg2,    \
            type3, arg3);

#define DTRACE_IO4(name, type1, arg1, type2, arg2,              \
    type3, arg3, type4, arg4)                                   \
        DTRACE_PROBE4(__io_##name, type1, arg1, type2, arg2,    \
            type3, arg3, type4, arg4);

The only thing that these wrappers add is the __io_ prefix to the name. The kernel DTrace module picks this up as the provider name.

These probe points have been placed where io stats get updated.

This is probably better demonstrated with an example.

On starting this demonstration you will see the command issued. It will then appear to pause for 10 seconds (because of the tick-10s probe) and then dump a histogram showing the times taken to do each IO involved in saving a Star Office presentation.

The tick-10s probe fires after 10 seconds and we use it to exit the command.

We don't list the probe module or probe func in the wait-start/wait-done probe as we don't care what these values are, we simply want to match these probe names.

The string between the forward slashes is called a predicate. This defines the conditions under which the probe will fire. In the wait-start probe, the condition is that the process name (execname) is "soffice.bin". The wait-done probe will only fire if self->start is non-null.

self->start probably also needs a little explanation. This is a thread local variable. It has scope of the lifetime of the thread and DTrace program. That is, if we had two seperate threads, the value of self->start could be different for each. start is simply the variable name that I chose to use to record the timestamp at which this probe fired.

Assigning 0 to self->start frees the memory assoiated with it.

The quantize function is what produces the histogram.

The mib provider

The mib provider, while not defined in terms of the DTRACE_PROBE macros, is also a static probe.

What makes this one interesting is that this minor change to two macros has an enormous effect on the observability of the network stack.

In order to show this properly, the full definition of BUMP_MIB and UPDATE_MIB from inet/mib2.h are shown.

#define BUMP_MIB(s, x)          {                               \
        extern void __dtrace_probe___mib_##x(int, void *);      \
        void *stataddr = &((s)->x);                             \
        __dtrace_probe___mib_##x(1, stataddr);                  \
        (s)->x++;                                               \
}

#define UPDATE_MIB(s, x, y)     {                               \
        extern void __dtrace_probe___mib_##x(int, void *);      \
        void *stataddr = &((s)->x);                             \
        __dtrace_probe___mib_##x(y, stataddr);                  \
        (s)->x += (y);                                          \
}

The lines could have been written

#define BUMP_MIB(s, x)          {                               \
        DTRACE_PROBE2(__mib_##x, int, 1, void *, &((s)->x))     \
        (s)->x++;                                               \
}

#define UPDATE_MIB(s, x, y)     {                               \
        DTRACE_PROBE2(__mib_##x, int, y, void *, &((s)->x))     \
        (s)->x += (y);                                          \
}

These two macros would previously have been defined simply without the reference to DTRACE_PROBE2. They are used exclusively to increment the network kstats. As a result, we end up with a probe point for every kstat update within the network code. On the machine that I am writing this on, that means 436 probes!

The variables passed in to these macros are

• s - The address of the kstat structure
• x - The name of the kstat variable to be increased
• y - The amount by which the kstat will be increased

This produces a probe with

• arg0 - The amount by which the kstat will be increased
• arg1 - The value of the kstat before it is increased

Let's look at an example of tracing a mib probe.

OK, this is a pretty contrived example. What it does is to dump the kernel stack when we first increase the value of the rawipOutDatagrams kstat and inform us of it's current value and what it will be incremented by. I fired this probe by pinging localhost.

User Space

We use a slightly different macro in user space

#define DTRACE_PROBE1(provider, name, arg1) {                          \
        extern void __dtrace_##provider##___##name(unsigned long);     \
        __dtrace_##provider##___##name((unsigned long)arg1);           \
}

The arguments are

• provider - name of the provider (duh!)
• name - name of the probe
• arg1, ... - the actual value the probe will report

Note that we don't define the type. This is done differently in user space and we'll get to that shortly.

Let's take a simple little program (helloworld.c)

#include <stdio.h>
#include <unistd.h>

int
main(int ac, char **av) {
        int i;
        for (i = 0 ; i < 5; i++) {
                printf("Hello World\n");
                sleep(2);
        }
}

and compile and run it.

Which is pretty much as you would expect.

Let's try adding a probe to this. Say we wanted to be able to monitor the loop variable.

#include <stdio.h>
#include <unistd.h>
#include <sys/sdt.h>

int
main(int ac, char **av) {
        int i;
        for (i = 0 ; i < 10; i++) {
                DTRACE_PROBE1(world, loop, i);
                printf("Hello World\n");
                sleep(2);
        }
}

The additions to the code are those in green. Basically We need to include <sys/sdt.h> and add the probe.

But wait, in user space there's more.

We still need to define the types of the arguments and the stability levels. This gets linked into the code later (as you will see in the demonstration).

Let's create msderv.d.

provider world {
        probe loop(int);
};

#pragma D attributes Evolving/Evolving/Common provider world provider
#pragma D attributes Private/Private/Common provider world module
#pragma D attributes Private/Private/Common provider world function
#pragma D attributes Evolving/Evolving/Common provider world name
#pragma D attributes Evolving/Evolving/Common provider world args

The stuff in provider is relatively self explanatory and I'm not going to go into the stability stuff here. Chapter 39 of the reference manual does a good job of that.

In order to build the probes incorporating the provider description we need another step in the build.

Let's build and run it without enabling any probes first.

Exactly as before, which we should expect.

The options to the dtrace command deserve some explanation.

• -32 - I don't have an opteron notebook, I need a 32 bit object.

• -G - Generate ELF files containing the embedded DTrace program. The probes specified in the files listed in the -s options are saved into these objects to be linked in to the binary. This also goes looking through for the probe points in the other objects and replaces them with nops so that the default is that the probes are disabled.
• -s - As previously mentioned, the file associated with this argument is treated as a D program containing the declarations of the probe points.

Let's look at both the counter and the first argument to printf.

i is now observable, but with no overhead unless we are tracing it.

Conclusion

SDT probes are an easy way to make stuff visible.

The helloworld example was trivial, but, imagine being able to place probes like this into large applications or drivers, sendmail might be a good example in user space, Veritas Volume Manager might be a good candidate for kernel.

We get observability without the need for

• Seperate instrumented binaries
• Restart
• Reboot (in the case of drivers/kernel)

With next to no overhead if they are not being observed.

Having placed probes into kernel code for debugging purposes, I can definitely recommend putting SDTs into your code to improve observability and diagnosability.

Technorati Tags: OpenSolaris, Solaris

Posted at 10:22AM Jun 06, 2005 by Alan Hargreaves in Solaris  |  Comments[3]

Nvidia Drivers available for Solaris on x86 and x64

Further to my not that I gave a brief demo of looking glass at SOSUG on Tuesday night, I just checked the nVIDIA drivers download page.

There is a bit more information at http://www.nvidia.com/object/solaris_display_1.0-7664.html, including a link to a discussion board on the driver.

The driver is supported for Quadro cards, but also simply works for many of the GeForce ones (like what's in my notebook. It should also be noted that it will only work with Solaris 10 and anything newer (e.g. Solaris Express and Open Solaris).

Well, simply works is a little bit of an over simplification.

You'll need to add the device name into /etc/driver_aliases if you are not using a supported card.

I'll try to get some instructions up in the next day or so (unless someone beats me to it).

Technorati Tag: Solaris

Posted at 01:41PM Jun 02, 2005 by Alan Hargreaves in Solaris  |  Comments[8]

SOSUG Meeting last night

We held the first Sydney Open Solaris User Group meeting last night in the iForce Centre in North Sydney.

About 20-25 people showed up (and the Sun folks appeared to be outnumbered).

Unfortunately I missed the first ten minutes or so as I got assigned door duty (the doors on the building lock at 6pm so someone had to stay out and let folks in. So I missed the Introduction and welcome by Che Kristo and James Eagleton.

Peter Lees

Peter gave an introduction to what Open Solaris will and will not be.

There was some good discussion about the license and it appears we were able to clear up a few things. For example how the license is file based and that it was designed to be re-usable.

Brendan Gregg

Brendan gave a demonstration of tracking down a problem without and then with DTrace. Brendan's style as an instructor really came to the fore as he involved everyone with the "how do we do this with the tools we already know".

He also spoke briefly about the work going on in creating a DTrace Toolkit and encouraged folks to contribute.

Alan Hargreaves

I spoke about Using SDT Probes, followed by a quick demo of looking glass under Solaris x86 with the shortly to be released Nvidia driver. I had a bit of trepidation about his as I'd only had an hour or so to get it working properly and I'd only just done so before leaving for the meeting.

Beer and Pizza

By the time that I finished, it was about 8pm and food had arived, so unfortunately we did not get to the video of Liane's presentation on SMF, but there is certainly interest in seeing it. Perhaps next month.

There was a lot of good informal discussion during this time.

Oh yes, we did have a door prize. Kavit (elric) Munshi won a copy of Sun Cluster 3 Programming by Joseph Bianco, Kevin Rabito and Peter Lees.

The next meeting should be in about a month and the idea of holding it at the James Squire Brew House was floated. Brendan also suggested that we might want to reserve some time at the next meeting for 5 minute informal sessions of "this is what I'm playing with", or "wouldn't this be a good idea" type material.

All in all, I think we all had a pretty good evening.

Che had a very nice looking video camera there so hopefully we should have video some time in the next week. Kavit assures me that when he watched it last night that it looked pretty good.

Technorati Tags: OpenSolaris, Solaris

Update June 2

Inserted South Park Brendan and modified his hyperlinks.

Posted at 03:39PM Jun 01, 2005 by Alan Hargreaves in Open Source Solaris  |  Comments[2]