I finished Beautiful Code this week, and have been reflecting on the book and its development. In particular, I have thought back to some of the authors' discussion, in which some advocated a different title. Many of us were very strongly in favor of the working title of "Beautiful Code", and I weighed in with my specific views on the matter:

   Date: Tue, 19 Sep 2006 16:18:22 -0700
   From: Bryan Cantrill 
   To: [Beautiful Code Co-Authors]
   Subject: Re: [Beautifulcode] reminder: outlines due by end of September

   Probably pointless to pile on here, but I'm for "Beautiful Code", if
   only because it appropriately expresses the nature of our craft.  We
   suffer -- tremendously -- from a bias from traditional engineering that
   writing code is like digging a ditch:  that it is a mundane activity best
   left to day labor -- and certainly beneath the Gentleman Engineer.  This
   belief is profoundly wrong because software is not like a dam or a
   superhighway or a power plant:  in software, the blueprints _are_ the
   thing; the abstraction _is_ the machine.

   It is long past time that our discipline stood on its feet, stepped out
   from the shadow of sooty 19th century disciplines, and embraced our
   unique confluence between mathematics and engineering.  In short,
   "Beautiful Code" is long, long overdue.

Now, I don't disagree with my reasoning from last September (though I think that the Norma Rae-esque tone was probably taking it a bit too far), but having now read the book, I stand by the title for a very different reason: this book is so widely varied -- there are so many divergent ideas here -- that only the most subjective possible title could encompass them. That is, any term less subjective than "beautiful" would be willfully ignorant of the disagreements (if implicit) among the authors about what constitutes ideal software.

To give an idea of what I'm talking about, here is the breakdown of languages and their representations in chapters:

Language	Chapters
C	11
Java	5
Scheme/Lisp	3
C++	2
Fortran	2
Perl	2
Python	2
Ruby	2
C#	1
JavaScript	1
Haskell	1
VisualBASIC	1

(I'm only counting each chapter once, so for the very few chapters that included two languages, I took whatever appeared more frequently. Also, note that some chapters were about the implementation of one language feature in a different language -- so for example, while there are two additonal chapters on Python, both pertain more to the C-based implementation of those features than to their actual design or use in Python.)

Now, one could argue (and it would be an interesting argument) about how much choice of language matters in software or, for that matter, in thought. And indeed, in some (if not many) of these chapters, the language of implementation is completely orthogonal to the idea being discussed. But I believe that language does ultimately affect thought, and it's hard to imagine how one could have a sense of beauty that is so uncritical as to equally accommodate all of these languages.

More specifically: read say, R. Kent Dybvig's chapter on the implementation of syntax-case in Scheme and William Otte and Douglas Schmidt's chapter on implementing a distributed logging service using an object-oriented C++ framework. It seems unlikely to me that one person will come away saying that both are beautiful to them. (And I'm not talking new-agey "beautiful to someone" kind of beautiful -- I'm talking the "I want to write code like that" kind of beautiful.) This is not meant to be a value judgement on either of these chapters -- just the observation that their definitions of beauty are (in my opinion, anyway) so wildly divergent as to be nearly mutually exclusive. And that's why the title is perfect: both of these chapters are beautiful to their authors, and we can come away saying "Hey, if it's beautiful to you, then great."

So I continue to strongly recommend Beautiful Code, but perhaps not in the way that O'Reilly might intend: you should read this book not because it's cover-to-cover perfection, but rather to hone your own sense of beauty. To that end, this is a book best read concurrently with one's peers: discussing (and arguing about) what is beautiful, what isn't beautiful, and why will help you discover and refine your own voice in your code. And doing this will enable you to write the most important code of all: code that is, if nothing else, beautiful to you.

So my copy of Beautiful Code showed up last week. Although I am one of the (many) authors and I have thus had access to the entire book online for some time, I do all of my pleasure reading in venues that need the printed page (e.g. the J Church) and have therefore waited for the printed copy to start reading.

Although I have only read the first twelve chapters or so, it's already clear (and perhaps not at all surprising) that there are starkly different definitions of beauty here: the book's greatest strength -- and, frankly, its greatest weakness -- is that the chapters are so incredibly varied. For one chapter, beauty is a small and titilating act of recursion; for the next, it's that a massive and complicated integrated system could be delivered quickly and cheaply. (I might add that the definition of beauty in my own chapter draws something from both of these poles: that in software, the smallest and most devilish details can affect the system at the largest and most basic levels.)

If one can deal with the fact that the chapters are widely divergent, and that there is not even a token attempt to weave them together into a larger tapestry, this book (at least so far, anyway) is (if nothing else) exceptionally thought provoking; if Oprah were a code cranking propeller head, this would be the ideal choice for her book club.

Now in terms of some of my specific thoughts that have been provoked: as I mentioned, quite a few of my coauthors are enamored with the elegance of recursion. While I confess that I like writing a neatly recursive routine, I also find that I frequently end up having to unroll the recursion when I discover that I must deal with data structures that are bigger than I anticipated -- and that my beautiful code is resulting (or can result) in a stack overflow. (Indeed, I spent several unpleasant days last week doing exactly this when I discovered that pathologically bad input could cause blown stacks in some software that I'm working on.)

To take a concrete example, Brian Kernighan has a great chapter in Beautiful Code about some tight, crisp code written by Rob Pike to perform basic globbing. And the code is indeed beautiful. But it's also (at least in a way) busted: it overflows the stack on some categories of bad input. Admittedly, one is talking about very bad input here -- strings that consist of hundreds of thousands of stars in this case -- but this highlights exactly the problem I have with recursion: it leaves you with edge conditions that on the one hand really are edge conditions (deeply pathological input), but with a failure mode (a stack overflow) that's just too nasty to ignore.

Now, there are ways to deal with this. If one can stomach it, the simplest way to deal with this is to setup a sigaltstack and then siglongjmp out of a SIGSEGV/SIGBUS signal handler. You have to be very careful about doing this: the signal handler should look at the si_addr field in the siginfo and comparing it to the stack bounds to confirm that it's a stack overflow, lest it end up siglongjmp'ing out of a non-recursion induced SIGSEGV (which, needless to say, would make a bad problem much worse). While an alternative signal stack solution may sound hideous to some, at least the recursion doesn't have to go under the knife in this approach. If having a SIGSEGV handler to catch this condition feels uncomfortably brittle (as well it might), or if one's state cannot be neatly unwound after an arbitrary siglongjmp (as well it might not), the code will have to change: either a depth counter will have to be passed down and failure propagated when depth exceeds a reasonable maximum, or the recursion will have to be unrolled into iteration. For most aesthetic senses, none of these options is going to make the code more beautiful -- but they will make it indisputably more correct.

I was actually curious about where exactly the Pike/Kernighan code would blow up, so I threw together a little program that uses sigaltstack along with sigsetjmp/siglongjmp to binary search to find the shortest input that induces the failure. My program, which (naturally) includes the Pike/Kernighan code, is here.

Here are the results of running my program on a variety of Solaris platforms, with each number denoting the maximum string length that can be processed by the Pike/Kernighan code without the possibility of stack overflow.

	x86		SPARC
	32-bit	64-bit	32-bit	64-bit
Sun cc, unoptimized	403265	187225	77649	38821
gcc, unoptimized	327651	218429	69883	40315
Sun cc, optimized	327651	327645	174723	95303
gcc, optimized	582489	524227	149769	87367

As can be seen, there is a tremendous range here, even across just two different ISAs, two different data models and two different compilers: from 38,821 on 64-bit SPARC using Sun cc without optimization to 582,489 on 32-bit x86 using gcc with optimization -- an order of magnitude difference. So while recursion is a beautiful technique, it is one that ends up with the ugliest of implicit dependencies: on the CPU architecture, on the data model and on the compiler. And while recursion is still beautiful to me personally, it will always be a beauty that is more superficial than profound...

For those who didn't see it, Team DTrace was on the Scoble Show. As I mention at the end of the interview, this was the day after my younger son was born (if you look closely at my right wrist, you will note that I am still wearing my hospital bracelet in the interview). You can see the cigars that I offered at the end of the interview (and they were damn fine cigars, by the way) in a photo of Team DTrace that Scoble took afterwards. After the photo, we returned to our office and smoked the cigars -- and then had an unplanned conversation with the building management about never again smoking cigars in the office. (I responded that I was done having kids, so they had nothing to worry about.)

Finally, as for DTrace on the iPhone (to which we made brief reference in the interview): it is now our understanding that alas, DTrace is not on the iPhone -- Apple has apparently not yet ported DTrace to the ARM -- but that a DTrace port "may be" in the works. So the dream is alive!

Our family is very happy to welcome its newest addition: Alexander Morgan Gaffikin Cantrill, born on June 17th at 3:22pm, and weighing in at a whopping 9 pounds, 7 ounces. (And that was five days early!) It's amazing how much one forgets over nearly three years; I've found myself cramming on forgotten (if simple) ideas like burping and "tummy time". (But I can still swaddle like an all-pro!)

Joyent -- the originators of the Ruby 1.8.5 DTrace provider and Ruby 1.8.6 DTrace provider -- have set up a dedicated DTrace site, which Jason and company discuss in their latest podcast. If you are a Rails shop that cares about performance (and, yes, they very much exist), the resources at the Joyent page should become invaluable to you. And as long as we're on the topic, if you're in San Francisco this Tuesday (May 22nd), and you have fifteen clams burning a hole in your pocket, you might be interested in attending a panel that Jason and I will both be on: Ruby on Rails: To Scale or Not to Scale? (Once we have a few drinks in us, Jason and I also anticipate hosting a follow-up panel: "Jason Hoffman and Bryan Cantrill: Will the Real Doogie Howser Please Stand Up?")

As is known but perhaps not widely reported, all three of us on Team DTrace are products of Brown University Computer Science. More specifically, we were all students in (and later TAs for) Brown's operating systems course, CS169. This course has been taught by the same professor, Tom Doeppner, over its thirty year lifetime, and has become something of a legend in Silicon Valley, having produced some of the top engineers at major companies like NetApp, SGI, Adobe, and VMware -- not to mention tons of smaller companies. And at Sun, CS169 has cast a particularly long shadow, with seven CS169 alums (Adam, Dan, Dave, Eric, Matt, Mike and me) having together played major roles in developing many of the revolutionary technologies in Solaris 10 (specifically, DTrace, ZFS, SMF, FMA and Zones).

I mention the Brown connection because this past Thursday, Brown hosted a symposium to honor both the DTrace team in particular and the contributions of former CS169 undergraduate TAs more generally. We were each invited to give a presentation on a topic of our choosing, and seizing the opportunity for intellectual indulgence, I chose to reflect on a broad topic: the inculcation of systems thinking. My thoughts on this topic deserve their own lengthy blog entry, but this presentation will have to suffice for now -- albeit stripped of the references to the Tupolev Tu-144, LBJ, Ray Kurzweil, the 737 rudder reversal and Ruby stack backtraces that peppered (or perhaps polluted?) the actual talk...

I know that I've been quiet for a while, and I promise that I (or rather, we) are close to talking about what we've been up to for the past year, but I wanted to first pop my head up to to highlight out some exciting news: Ian Murdock has joined Sun.

I have always been impressed with Ian's decidedly pragmatic views of technology; he has been a supporter of OpenSolaris, and in particular he took a decisive and courageous stand against some absurd Debian-borne anti-Nexenta licensing FUD. Looking back, it's hard to imagine that our OpenSolaris/Debian fantasia was not even two years ago, and that it was just eighteen months ago that we first saw a prototype of what that Utopia might look like. Ian's arrival at Sun is a huge lurch forward towards the wide-spread productization of this ideal: it's great news for OpenSolaris and it's great news for long-time fans of Debian -- it looks like we're going to be able to apt-get our cake, and DTrace it too!

A little while ago, I blogged about DTrace on Rails. In particular, I promised that I would get diffs based on is-enabled probes out "shortly." In giving a guest lecture for a class at Berkeley yesterday, I was reminded that I still hadn't made this available. With my apologies for the many-months delay, the diff (against Ruby 1.8.2) is here.

And as long as I have your eyeballs, let me join Adam in directing you to Brendan Gregg's amazing Helper Monkey. Brendan's work is an indisputable quantum leap for what has become one of the most important, one of the most misunderstood, and certainly one of the most undebuggable platforms on the planet: JavaScript. If you do anything AJAX-ian that's evenly vaguely performance or footprint sensitive, you're wasting your time if you're not using Helper Monkey. (When Brendan first demo'd Helper Monkey to Adam and me, Adam's line was that Brendan had just propelled JavaScript debugging forward by "100,000 years" -- which is not to say that Helper Monkey is like debugging in the 1021st century, but rather that debugging JavaScript without Helper Monkey is like debugging in the late Pleistocene.)

So IBM has been on the warpath recently against OpenSolaris, culminating with their accusation yesterday that OpenSolaris is a "facade." This is so obviously untrue that it's not even worth refuting in detail. In fact, being the father of a toddler, I would liken IBM's latest outburst to something of a temper tantrum -- and as with a screaming toddler, the best way to deal with this is to not reward the performance, but rather to offer some constructive alternatives. So, without further ado, here are my constructive suggestions to IBM:

• Open source OS/2. Okay, given the tumultuous history of OS/2, this is almost certainly not possible from a legal perspective -- but it would be a great open source contribution (some reasonably interesting technology went down with that particular ship), and many hobbyists would love you for it. Like I said, it's probably not possible -- but just to throw it out there.

• Open source AIX. AIX is one of the true enterprise-class operating systems -- one with a long history of running business-critical applications. As such, it would be both a great contribution to open source and a huge win for AIX customers for AIX to go open source -- if only to be able to look at the source code when chasing a problem that isn't necessarily a bug in the operating system. (And I confess that on a personal level, I'm very curious to browse the source code of an operating system that was ported from PL/1.) However, as with OS/2, AIX's history is going to likely make open sourcing it tough from a legal perspective: its Unix license dates from the Bad Old Days, and it would probably be time consuming (and expensive) to unencumber the system to allow it to be open sourced.

Okay, those two are admittedly pretty tough for legal reasons. Here are some easier ones:

• Support the port of OpenSolaris to POWER/PowerPC. Sun doesn't sell POWER-based gear, so you would have the comfort of knowing that your efforts would in no way assist a Sun hardware sale, and your POWER customers would undoubtedly be excited to have another choice for their deployments.

• Support the nascent effort to port OpenSolaris to the S/390. Hey, if Linux makes sense on an S/390, surely OpenSolaris with all of its goodness makes sense too, right? Again, customers love choice -- and even an S/390 customer that has no intention of running OpenSolaris will love having the choice made available to them.

Okay, so those two are easier because the obstacles aren't legal obstacles, but there are undoubtedly internal IBM cultural issues that make them effectively non-starters.

So here's my final suggestion, and it's an absolutely serious one. It's also relatively easy, it clearly and immediately benefits IBM and IBM's customers -- and it even doesn't involve giving up any IP:

• Port DTrace to AIX. Your customers want it. Apple has shown that it can be done. We'll help you do it. And you'll get to participate in the DTrace community (and therefore the OpenSolaris community) in a way that doesn't leave you feeling like you've been scalped by Scooter. Hell, you can even follow Apple's lead with Xray and innovate on top of DTrace: from talking to your customers over the years, it's clear that they love SMIT -- integrate a SMIT frontend with a DTrace backend! Your customers will love you for it, and the DTrace community will be excited to have yet another system on which that they can use DTrace.

Now, IBM may respond to these alternatives just as a toddler sometimes responds to constructive alternatives ("No! No! NO! Mine! MINE! MIIIIIIIINE!", etc). But if cooler heads prevail at Big Blue, these suggestions -- especially the last one -- will be seen as a way to constructively engage that will have clear benefits for IBM's customers (and therefore for IBM). So to IBM I say what parents have said to screaming toddlers for time immemorial: we're ready when you are.

From WWDC here in San Francisco: Apple has just announced support for DTrace in Leopard, the upcoming release of Mac OS X! Often (or even usually?) announcements at conferences are more vapor than ware. In this case, though, Apple is being quite modest: they have done a tremendous amount of engineering work to bring DTrace to their platform (including, it turns out, implementing DTrace's FBT provider for PowerPC!), and they are using DTrace as part of the foundation for their new Xray performance tool. This is very exciting news, as it brings DTrace to a whole slew of new users. (And speaking personally, it will be a relief to finally have DTrace on the only computer under my roof that doesn't run Solaris!) Having laid hands on DTrace on Mac OS X myself just a few hours ago, I can tell you that while it's not yet a complete port, it's certainly enough to be uniquely useful -- and it was quite a thrill to see Objective C frames in a ustack action! So kudos to the Apple engineering team working on the port: Steve Peters, James McIlree, Terry Lambert, Tom Duffy and Sean Callanan.

It's been fun for us to work with the Apple team, and gratifying to see their results. And it isn't just rewarding for us; the entire OpenSolaris community should feel proud about this development because it gives the lie to IBM's nauseating assertion that we in the OpenSolaris community aren't in the "spirit" of open source.

So to Apple users: welcome to DTrace! (And to DTrace users: welcome to Mac OS X!) Be sure to join us in the DTrace community -- and if you're at the WWDC, we on Team DTrace will be on hand on Friday at the DTrace session, so swing by to see a demo of DTrace running on MacOS and to meet both the team at Apple that worked on the port and we at Sun who developed DTrace. And with a little luck, you might even hear Adam commemorate the occasion by singing his beautiful and enchanting ISA aria...

Update: For disbelievers, Adam posted photos -- and Mike went into greater detail on the state of the Leopard DTrace port, and what it might mean for the direction of DTrace.

A while ago, I blogged about the possiblity of a FreeBSD port of DTrace. For the past few months, John Birrell has been hard at work on the port, and has announced recently that he has much of the core DTrace functionality working. Over here at DTrace Central, we've been excitedly watching John's progress for a little while, providing help and guidance where we can -- albeit not always solicited ;) -- and have been very impressed with how far he's come. And while John has quite a bit further to go before one could call it a complete port, what he has now is indisputably useful. If you run FreeBSD in production, you're going to want John's port as it stands today -- and if you develop for the FreeBSD kernel (drivers or otherwise), you're going to need it. (Once you've done kernel development with DTrace, there's no going back.)

So this is obviously a win for FreeBSD users, who can now benefit from the leap in software observability that DTrace provides. It's also clearly a win for DTrace users, because now you have another platform on which you can observe your production software -- and a larger community with whom to share your experiences and thoughts. And finally, it's a huge win for OpenSolaris users: the presence of a FreeBSD port of DTrace validates that OpenSolaris is an open, innovative platform (despite what some buttheads say) -- one that will benefit from and contribute to the undeniable economics of open source.

So congrats to John! And to the FreeBSD folks: welcome to the DTrace community!

First I need to apologize for having been absent for so long -- I am very much heads-down on a new project. (The details of which will need to wait for another day, I'm afraid -- but suffice it to say that it, like just about everything else I've done at Sun, leverages much of what I've done before it.)

That said, I wanted to briefly emerge to discuss some recent work. A while ago, I blogged about DTrace and Ruby, using Rich Lowe's prototype DTrace provider. This provider represents a quantum leap for Ruby observability, but it suffers from the fact that we must do work (in particular, we must get the class and method) even when disabled. This is undesirable (especially considering that the effect can be quite significant -- up to 2X), and it runs against the DTrace ethos of zero disabled probe effect, but there has been no better solution. Now, however, thanks to Adam's work on is-enabled probes, we can have a Ruby provider that has zero disabled probe effect. (Or essentially zero: I actually measured the probe effect at 0.2% -- very much in the noise.) Having zero disabled probe effect allows us to deploy DTrace on Ruby in production -- which in turn opens up a whole new domain for DTrace: Ruby on Rails. And as I was reminded by Jason Hoffman's recent Scale with Rails presentation (in which he outlines why they picked Solaris generally -- and ZFS in particular), this is a hugely important growth area for Solaris. So without further ado, here is a (reasonably) simple script that relies on some details of WEBrick and Rails to yield a system profile for Rails requests:

#pragma D option quiet

self string uri;

syscall::read:entry
/execname == "ruby" && self->uri == NULL/
{
        self->fd = arg0;
        self->buf = arg1;
        self->size = arg2;
}

syscall::read:return
/self->uri == NULL && self->buf != NULL && strstr(this->str =
    copyinstr(self->buf, self->size), "GET ") == this->str/
{
        this->head = strtok(this->str, " ");
        self->uri = this->head != NULL ? strtok(NULL, " ") : NULL;
        self->syscalls = 0;
        self->rbcalls = 0;
}

syscall::read:return
/self->buf != NULL/
{
        self->buf = NULL;
}

syscall:::entry
/self->uri != NULL/
{
        @syscalls[probefunc] = count();
}

ruby$1:::function-entry
/self->uri != NULL/
{
        @rbclasses[this->class = copyinstr(arg0)] = count();
        this->sep = strjoin(this->class, "#");
        @rbmethods[strjoin(this->sep, copyinstr(arg1))] = count();
}

pid$1::mysql_send_query:entry
/self->uri != NULL/
{
        @queries[copyinstr(arg1)] = count();
}

syscall::write:entry
/self->uri != NULL && arg0 == self->fd && strstr(this->str =
    copyinstr(arg1, arg2), "HTTP/1.1") == this->str/
{
        self->uri = NULL;
        ncalls++;
}

END
{
        normalize(@syscalls, ncalls);
        trunc(@syscalls, 10);
        printf("Top ten system calls per URI serviced:\n");
        printf("---------------------------------------");
        printf("--------------------------------+------\n");
        printa("  %-68s | %@d\n", @syscalls);

        normalize(@rbclasses, ncalls);
        trunc(@rbclasses, 10);
        printf("\nTop ten Ruby classes called per URI serviced:\n");
        printf("---------------------------------------");
        printf("--------------------------------+------\n");
        printa("  %-68s | %@d\n", @rbclasses);

        normalize(@rbmethods, ncalls);
        trunc(@rbmethods, 10);
        printf("\nTop ten Ruby methods called per URI serviced:\n");
        printf("---------------------------------------");
        printf("--------------------------------+------\n");
        printa("  %-68s | %@d\n", @rbmethods);

        trunc(@queries, 10);
        printf("\nTop ten MySQL queries:\n");
        printf("---------------------------------------");
        printf("--------------------------------+------\n");
        printa("  %-68s | %@d\n", @queries);
}

Running the above while horsing around with the Depot application from Agile Web Development with Rails yields the following:

Top ten system calls per URI serviced:
-----------------------------------------------------------------------+------
  setcontext                                                           | 15
  fcntl                                                                | 16
  fstat64                                                              | 16
  open64                                                               | 21
  close                                                                | 25
  llseek                                                               | 27
  lwp_sigmask                                                          | 30
  read                                                                 | 62
  pollsys                                                              | 80
  stat64                                                               | 340

Top ten Ruby classes called per URI serviced:
-----------------------------------------------------------------------+------
  ActionController::CodeGeneration::Source                             | 89
  ActionController::CodeGeneration::CodeGenerator                      | 167
  Fixnum                                                               | 190
  Symbol                                                               | 456
  Class                                                                | 556
  Hash                                                                 | 1000
  String                                                               | 1322
  Array                                                                | 1903
  Object                                                               | 2364
  Module                                                               | 6525

Top ten Ruby methods called per URI serviced:
-----------------------------------------------------------------------+------
  Object#dup                                                           | 235
  String#==                                                            | 250
  Object#is_a?                                                         | 288
  Object#nil?                                                          | 316
  Hash#[]                                                              | 351
  Symbol#to_s                                                          | 368
  Object#send                                                          | 593
  Module#included_modules                                              | 1043
  Array#include?                                                       | 1127
  Module#==                                                            | 5058

Top ten MySQL queries:
-----------------------------------------------------------------------+------
  SELECT * FROM products  LIMIT 0, 10                                  | 2
  SELECT * FROM products WHERE (products.id = '7')  LIMIT 1            | 2
  SELECT count(*) AS count_all FROM products                           | 2
  SHOW FIELDS FROM products                                            | 5

While this gives us lots of questions we might want to answer (e.g., "why the hell are we doing 340 stats on every 'effing request?!"¹), it might be a little easier to look at a view that lets us see requests and the database queries that they induce. Here, for example, is a similar script to do just that:

#pragma D option quiet

self string uri;
self string newuri;

BEGIN
{
        start = timestamp;
}

syscall::read:entry
/execname == "ruby" && self->uri == NULL/
{
        self->fd = arg0;
        self->buf = arg1;
        self->size = arg2;
}

syscall::read:return
/self->uri == NULL && self->buf != NULL && (strstr(this->str =
    copyinstr(self->buf, self->size), "GET ") == this->str ||
    strstr(this->str, "POST ") == this->str)/
{
        this->head = strtok(this->str, " ");
        self->newuri = this->head != NULL ? strtok(NULL, " ") : NULL;
}

syscall::read:return
/self->newuri != NULL/
{
        self->uri = self->newuri;
        self->newuri = NULL;
        printf("%3d.%03d => %s\n", (timestamp - start) / 1000000000,
            ((timestamp - start) / 1000000) % 1000,
            self->uri);
}

pid$1::mysql_send_query:entry
/self->uri != NULL/
{
        printf("%3d.%03d   -> \"%s\"\n", (timestamp - start) / 1000000000,
            ((timestamp - start) / 1000000) % 1000,
            copyinstr(self->query = arg1));
}

pid$1::mysql_send_query:return
/self->query != NULL/
{
        printf("%3d.%03d   <- \"%s\"\n", (timestamp - start) / 1000000000,
            ((timestamp - start) / 1000000) % 1000,
            copyinstr(self->query));
        self->query = NULL;
}

syscall::read:return
/self->buf != NULL/
{
        self->buf = NULL;
}

syscall::write:entry
/self->uri != NULL && arg0 == self->fd && strstr(this->str =
    copyinstr(arg1, arg2), "HTTP/1.1") == this->str/
{
        printf("%3d.%03d <= %s\n", (timestamp - start) / 1000000000,
            ((timestamp - start) / 1000000) % 1000,
            self->uri);
        self->uri = NULL;
}

Running the above while clicking around with the Depot app:

# ./rsnoop.d `pgrep ruby`
  7.936 => /admin/edit/7
  7.955   -> "SELECT * FROM products WHERE (products.id = '7')  LIMIT 1"
  7.956   <- "SELECT * FROM products WHERE (products.id = '7')  LIMIT 1"
  7.957   -> "SHOW FIELDS FROM products"
  7.957   <- "SHOW FIELDS FROM products"
  7.971 <= /admin/edit/7
 20.881 => /admin/update/7
 20.952   -> "SELECT * FROM products WHERE (products.id = '7')  LIMIT 1"
 20.953   <- "SELECT * FROM products WHERE (products.id = '7')  LIMIT 1"
 20.953   -> "SHOW FIELDS FROM products"
 20.953   <- "SHOW FIELDS FROM products"
 20.954   -> "BEGIN"
 20.954   <- "BEGIN"
 20.955   -> "UPDATE products SET `title` = 'foo bar', `price` = 1.2, ...
 20.955   <- "UPDATE products SET `title` = 'foo bar', `price` = 1.2, ...
 20.989   -> "COMMIT"
 20.989   <- "COMMIT"
 21.001 <= /admin/update/7
 21.005 => /admin/show/7
 21.023   -> "SELECT * FROM products WHERE (products.id = '7')  LIMIT 1"
 21.023   <- "SELECT * FROM products WHERE (products.id = '7')  LIMIT 1"
 21.024   -> "SHOW FIELDS FROM products"
 21.024   <- "SHOW FIELDS FROM products"
 21.038 <= /admin/show/7

I'm no Rails developer, but it seems like this might be useful... If you want to check this out for yourself, start by getting Rich's prototype provider. (Using it, you can do everything I've done here, just with higher disabled probe effect.) Meanwhile, I'll work with Rich to get the lower disabled probe effect version out shortly. Happy Railing!

---

¹ Or if you're as savvy as the commenters on this blog entry, you might be saying to yourself, "why the hell is the 'effing development version running in production?!"

There are two ways to get DTrace for another operating system: you can try porting DTrace to the other system, or you can -- as Adam Leventhal describes -- use the new BrandZ framework to get that other system running under Solaris. Adam describes applying DTrace to a Linux binary -- top -- and using DTrace to find a (pretty serious) Linux-specific performance problem. Pretty eff'in cool...

If you haven't already seen it, ZFS is now available for download, marking a major milestone in the history of filesystems. Today is a way station down a long road: for as long as I have known Jeff Bonwick, he has wanted to solve the filesystem problem -- and about five years ago, Jeff set out to do just that, starting (as Jeff is wont to do) from a blank sheet of paper. I vividly remember Jeff describing some of his nascent ideas on my whiteboard; the ideas were radical and revolutionary, their implications manifold. I remember thinking "he's either missed something basic that somehow invalidates these ideas -- or this is the most important development in storage since RAID." As I recently recounted, Jeff is the reason that I came to Sun almost a decade ago -- and in particular, I was drawn by Jeff's contagious belief that nothing is impossible simply because it hasn't been done before. So I knew better than to doubt him at the time -- and I knew that the road ahead promised excitement if nothing else. Years after that moment, there is no other conclusion left to be had: ZFS is the most important revolution in storage software in two decades -- and may be the most important idea since the filesystem itself. That may seem a heady claim, but keep reading...

To get an idea of what ZFS can do, first check out Dan Price's awesome ZFS flash demo Then join me on a tour of today's ZFS blog entries, as ZFS developers and users inside Sun illustrate the power of ZFS: ease of administration, absolute reliability and rippin' performance.

• Administration. If you're an administrator, start your ZFS blog tour with Martin Englund's entry on ZFS from a sysadmin's view. Martin walks you through the ease of setting up ZFS; there are no hidden wires -- it really is that easy! And if, as a self-defence mechanism, your brain refuses to let you recall the slog of traditional volume management, check out Tim Foster's entry comparing ZFS management to Veritas management. (And have we mentioned the price?) For insight into the design principles that guided the development of the administration tools, check out Eric Schrock's entry on the principles of the ZFS CLI. Eric's entry and his design reflect the principles that we used in DTrace as well: make it simple to do simple things and make it possible to do complicated things. As you can imagine, this simplicity of management is winning fans both inside and outside of Sun. For some testimonials, check out Lin Ling's entry on the love for ZFS -- both Lin's and our Beta customers'. As Lin's entry implies, a common theme among ZFS users is "if I only had this when..."; check out James McPhearson's entry wishing he had ZFS back in the day.

  And if you think that the management of ZFS couldn't get any easier, check out Steve Talley's entry on managing ZFS from your browser. Steve's work highlights the proper role for GUI admin tools in a system: they should make something that's already simple even simpler. They should not be used to smear lipstick over a hideously over-complicated system -- doing so leads to an unresolvable rift between what the tool is telling you the system looks like, and what the system actually looks like. Thanks to the simplicity of ZFS itself, there is no second guessing about what the GUI is actually doing under the hood -- it's all just gravy!

  Speaking of gravy, check out the confluence of ZFS with another revolutionary Solaris technology in Dan Price's entry on ZFS and Zones -- thanks to some great integration work, local zone administrators can have the full power of ZFS without compromising the security of the system!

  For details on particular features of ZFS, check out Mark Maybee's entry on quotas and reservations in ZFS. Unlike some other systems, quotas and reservations are first-class citizens in ZFS, not bolted-on afterthoughts. Die, /usr/sbin/quota, die! And for details on another feature of ZFS, check out Mark Shellenbaum's entry on access control lists in ZFS, and Lisa Week's entry describing why ZFS adopted the NFSv4 ACL model. Like quotas and reservations, ACLs were a part of the design of ZFS -- not something that was carpet-bombed over the source after the fact.

• Reliability. Unlike virtually every other filesystem that has come before it, ZFS is designed around unreliable hardware. This design-center means that ZFS can detect -- and correct! -- errors that other filesystems just silently propagate to the user. To get a visceral feel for this, read Eric Lowe's entry on ZFS saving the day. Reading this entry will send a chill up your spine: Eric had a data-corrupting hardware problem that he didn't know he had until ZFS. How much data is being corrupted out there today because pre-ZFS filesystems are too trusting of faulty hardware? More to the point, how much of your data is being corrupted today? Yeah -- scary, ain't it? And not only can ZFS detect hardware errors, in a mirrored configuration it can correct them. Fortunately, you don't have to have busted hardware to see this: look at Tim Cook's entry demonstrating ZFS's self-healing by using dd to simulate date corruption.

  But if problems like Eric's are all over the place, how is anyone's data ever correct? The answer is pretty simple, if expensive: you pay for reliability by buying over-priced hardware. That is, we've compensated for dumb software by having smart (and expensive) hardware. ZFS flips the economics on its head: smart software allows for stupid (and cheap) hardware -- with ultra-high reliability. This is a profound shift; for more details on it check out Richard Elling's entry on the reliability of ZFS.

  ZFS is reliable by its architecture, but what of the implementation? As Bill Moore writes, testing ZFS was every bit as important as writing it. And testing ZFS involved many people, as Jim Walker describes in his entry on the scope of the ZFS testing effort.

• Performance. So fine: ZFS is a snap to administer, and it's ultra-reliable -- but at what performance cost? The short answer is: none, really -- and in fact, on many workloads, it rips. How can you have such features and still have great performance? Generally speaking, ZFS is able to deliver great performance because it has more context, a phenomenon that Bill Sommerfeld notes is a consequence of the end-to-end principle. To see how this unlocks performance, look at Bill Moore's entry on I/O scheduling; as Bill describes (and as I can personally attest to) ZFS is much smarter about how it uses I/O devices than previous filesystems. For another architectural feature for performance, look at Neil Perrin's entry on the ZFS intent log -- and chase it with Neelakanth Nadgir's entry taking you through the ZIL code.

  If you're looking for some performance numbers, check out Roch Bourbonnais's entry comparing the performance of ZFS and UFS. Or let Eric Kustarz take you to school, as you go to Filesystems Performance 101: Disk Bandwidth, Filesystems Performance 102: Filesystem Bandwidth and finally graduate to Filesystems Performance 201: When ZFS Attacks!

So given that ZFS is all that, when can we start forgetting about every other on-disk filesystem? For that, we'll need to be able to boot off ZFS. Bad news: this is hard. Good news: Tabriz Leman and the rest of the ZFS Boot team are making great progress, as Tabriz describes in her entry on booting ZFS. Once we can boot ZFS -- that is, once we can assume ZFS -- all sorts of cool things become possible, as Bart Smaalders brainstorms in his entry on the impact of ZFS on Solaris. As Bart says, this is just the beginning of the ZFS revolution...

Finally, this has been a long, hard slog for the ZFS team. Anyone who has worked through "crunch time" on a big project will see something of themselves in Noel Dellofano's entry on the final push. And any parent can empathize with Sherry Moore's entry congratulating the team -- and looking forward to having her husband once again available to help with the kids. So congratulations to everyone on the ZFS team (and your families!) -- and for everyone else, welcome to ZFS!

---

Technorati tags: OpenSolaris Solaris ZFS

About three months ago, I wrote about the exciting possibilities of combining Debian and OpenSolaris. To be honest, when I wrote that I assumed that such a Xanadu would be a couple of years off at least -- combining these systems is non-trivial. You can thus imagine my surprise last week when I first heard of the Nexenta project. If you haven't heard, this project is doing exactly what Jeff, Adam, Mike and I were wishing for: Debian package management on an OpenSolaris kernel. Of course, hearing about a new technology is one thing; seeing it is quite another -- I wanted to get my hands on actual bits before I got too excited. Well, today Nexenta released their ISOs. And, to make a pleasantly short story even shorter, I am writing this now running Nexenta on my Acer Ferrari 3400. But don't take my word for it; check out this screenshot -- and note in particular my use of DTrace to examine the package management system. Oh be still, my beating heart!