Jyri Virkki

Web Server and TLS/SSL

Well, noticed tonight on twitter that the recent SSL/TLS protocol vulnerability is being mentioned and information has been published here.

As far as our Sun Web Server 7 is concerned, we have been looking at this for a number of weeks now but the timing of the public discussion tonight is a bit of a surprise. I will point you at further information from here once officially available.

Posted at 09:28PM Nov 04, 2009 by jyri in WebServer  |  Comments[0]

Endless Night (Take Four)

Is it that time of the year again? I guess so!

Back in February I posted my (sort-of) biannual review of SFW build times so it has been six months. The SFW build continue to chug along towards collapsing under its own weight as I predicted two years ago. I can't claim much in the way of visionary powers for the observation since it is a rather obvious outcome of consolidating all sources into one tree. Unfortunately not obvious enough though since the practice still continues!

To review, refer to my original article on unconsolidating which exposes the problem with the Solaris build concept of building all applications together in one single source tree. I updated the data in June 2008 and later in February 2009.

Aside from the build times nothing has really changed so no other news. If you haven't read the previous articles check them out since I won't repeat the data here.

As of this month, SFW build produces 416 packages, takes 9.7 hours to build and a built workspace takes 19.4GB!

The time/space predictions for 5000 packages are within the range previously seen (current one in bold in above table: 116 hours and 233GB) so no big surprises.

Well the big surprise is we're still building OpenSolaris applications this way!

For the Web Stack project and components we are now looking into dropping out of SFW since this is unsustainable and is consuming too much of our limited resources. Hopefully my next biannual SFW update will be that there isn't one! ;-) Time will tell...

Posted at 10:20PM Aug 27, 2009 by jyri in WebStack  |  Comments[0]

Request Processing Capacity

Q: How many requests per second can the Web Server handle?

Short answer: It depends.

Long answer: It really depends on many factors.

Ok, ok.. sillyness aside, can we make any ballpark estimates?

The Web Server can be modeled as a queue. By necessity such modeling will be a simplification at best, but it may provide a useful mental model to visualize request processing inside the server.

Let's assume your web application has a fairly constant processing time^[1], so we'll model the Web Server as a M/D/c queue where c is the number of worker threads. In this scenario, the Web Server has a maximum sustainable throughput of c / (processing time).

To use some simple numbers, let's say your web app takes 1 second to process a request (that's a very slow web application!). If the Web Server has c=128 worker threads, that means it can indefinitely sustain a max request rate of:

128/1 = 128 requests per second

This makes a lot of sense if we think about it:

• At t = 0 seconds, 128 request come in and each one is taken by a worker thread, fully utilizing server capacity.
• At t = 1 second, all those requests complete and responses are sent back to the client and at the same time 128 new requests come in and the cycle repeats.

At this request rate we don't need a connection queue at all^[2] because all requests go straight to a worker thread. This also means that at this request rate the response time experienced by the end user is always 1 second.

To expand on that, the response time experienced by the end user is:

end user response time = (connection queue wait time) + (processing time)

Since we're not using the connection queue the end user response time is simply the same as the processing time^[3].

So far so good. Now, what happens if the incoming request rate exceeds the maximum sustainable throughput?

• At t = 10 seconds, 129 requests come in. 128 go straight to worker threads, 1 sits in wait in the connection queue.
• At t = 11 seconds, 128 requests come in. 128 (the one which was waiting + 127 of the new ones) go straight to worker threads, 1 sits in wait in the connection queue.

The connection queue absorbs the bumps in the incoming request rate, so connections are not dropped and worker threads can remain fully utilized at all times. Notice that now out of every 128 requests, one of them will have a response time of 2 seconds.

So what happens next?

If we go back to receiving a steady 128 requests per second, there will always be one requests in the connection queue.

If at some point we only receive 127 requests (or less), the server can "catch up" and the connection queue goes back to staying empty.

On the other hand, if the incoming request rate remains at 129 per second we're in trouble! Every second the connection queue waiting list will grow longer by one. When it reaches 129 entries, one end user will experience a response time of three seconds, and so on.

And of course, the connection queue is not infinite. If the max connection queue size is 4096 then 4096 seconds later it will fill up and from that point onwards, one incoming request will simply be dropped every second since it has no place to go. At this point the server has reached a steady state. It continues processing requests at the same rate as always (128 per second), it continues accepting 128 of the 129 new requests per second and dropping one. End users are certainly unhappy by now because they are experiencing response times of over 30 seconds (4096 / 128 = 32, so it takes 32 seconds for a new request to work its way through the queue. Almost like going to the DMV...

If the incoming request rate drops below the maximum sustainable rate (here, 128/sec) only then can the server start to catch up and eventually clear the queue.

In summary, while this is certainly a greatly simplified model of the request queue behavior, I hope it helps visualize what goes on as request rates go up and down.

Theory aside, what can you do to tune the web server?

• The single best thing to do, if possible, is to make the web app respond quicker!
• If you want to avoid dropped connections at all cost, you can increase the connection queue size. This will delay the point where the server reaches a steady state and starts dropping connections. Whether this is useful really depends on the distribution of the incoming requests. In the example above we've been assuming a very steady incoming rate just above the maximum throughput rate. In such a scenario increasing the connection queue isn't going to help in practice because no matter how large you make it, it will fill up at some point. On the other hand, if the incoming request rate is very bumpy, you can damp it by using a connection queue large enough to avoid dropping connections. However... consider the response times as well. In the example above your end user is already seeing 33 second response times. Increasing the connection queue length will prevent dropped connections but will only make the response times even longer. At some point the user is simply going to give up so increasing the connection queue any further won't help!
• Another option is to increase the number of worker threads. Whether this will help or hurt depends entirely on the application. If the request processing is CPU bound then it won't help (actually, if it were truly CPU bound, which is rare, then you'll probably benefit from reducing the number of worker threads unless your server has 128+ CPUs/cores...) If the web app spends most of its time just waiting for I/O then increasing the worker threads may help. No set answer here, you need to measure your application under load to see.

[1] In reality the response time can't be deterministic. At best it may be more or less constant up to the point where the server scales linearly but after that the response time is going to increase depending on load. On the flip side, cacheing might make some responses faster than expected. So M/D/c is certainly a simplification.

[2] Not true for several reasons, but it'll do for this simplified model and it helps to visualize it that way.

[3] Plus network transmission times but since we're modeling only the web server internals let's ignore that.

Posted at 01:48AM Aug 26, 2009 by jyri in WebServer  |  Comments[0]

Web Server 7 Request Limiting Revisited

Coincidentally last week I heard a couple related queries about check-request-limits from different customers. I haven't covered that feature in a while so it's a good time to revisit it for a bit.

To review, Web Server 7 has a feature (function) called check-request-limits which can be used to monitor and limit the request rate and/or concurrency of request which match some criteria. It can be used to address denial of service attacks as well as just to limit request rates to some objects or from some clients for other reasons (for example to reduce bandwidth or cpu usage).

I usually refer to 'matching requests' when speaking of this capability. Matching what? Probably the most common use case is to match the client IP address. This is useful when you wish to limit request rates coming from a given client machine. Here's a basic example of that scenario:

PathCheck fn="check-request-limits" max-rps="10" monitor="$ip"

The common theme to both customer requests I heard last week was whether it is possible to limit requests based on something other than the client IP?

Yes, certainly!

The monitor parameter above is set to "$ip" which expands to the client IP address but you can set it to anything that you prefer. In my introduction to check-request-limits article I gave examples of both "$ip" and "$uri" (and even both combined). You're not restricted to only these though, you can use any of the server variables available in WS7 as the monitor value.

You can also construct more complicated scenarios using the If expressions of Web Server 7. I gave a few examples of that in this article on check-request-limits.

To give a couple more examples, let's say your web server is behind a proxy and this the client $ip is always the same (the proxy IP). Clearly monitoring the $ip value isn't terribly useful in that case. Depending on how your application works you may be able to find other useful entries to monitor. For example if the requests contain a custom header named "Usernum" which contains a unique user number, you could monitor that:

PathCheck fn="check-request-limits" max-rps="1" monitor="$headers{'usernum'}"

Or maybe there's a cookie named customer which can serve as the monitor key:

PathCheck fn="check-request-limits" max-rps="1" monitor="$cookie{'customer'}" 

These two are made-up examples, you'll need to pick a monitor value which is suitable for your application. But I hope these ideas will help you get started.

By the way check-request-limits can also be used to limit concurrency.

Posted at 12:43AM Aug 25, 2009 by jyri in WebServer  |  Comments[0]

Time Allocation

Last week I wrote about the time spent dealing with email. While writing that entry I though it'd be nice to also visualize where all the time went, not just how much was spent on email. So tonight I went over the data to generate the following pie chart, showing relative allocation of working hours from early May until today, split into high level categories:

The 'Email' slice is self evident.. 'ARC' is the time I've spent in my role in Sun's Architecture Review Committee. 'Communications' includes conferences, presentation, blog entries, articles and other related work. 'Administrivia' is a catch-all category for all kinds of mindless unproductive overhead. Finally, 'Engineering' represents the time spent doing "real work".

About the only thing I can add is that this is about as concise a representation as we can get on why very large companies have trouble competing with agile startups. Part of my goal in this exercise is to find ways to grow that nice blue pie slice, but I realize there's a limit to what can be achieved in this environment. All those TPS^H^H^HPTL reports needs to be filed, after all.

Posted at 08:54PM Aug 18, 2009 by jyri in Other  |  Comments[0]

Let Me Check My Email

About two months ago I posted on attempting to keep email in check so it's a good time to review some statistics and results...

The following graph shows the percentage of time I spent reading email each day:

The average over the past three months is about 45% Wow.. So over the last quarter I've spent just under half of all working hours reading (and answering) email. No wonder it is hard to get concrete work done!

This is somewhat higher than the 37.5% (three hours a day out of eight) that I had predicted in the previous article a couple months ago. This is largely explained due to the recent release of Web Stack 1.5. Due to the impending release I found myself having to check email more often than scheduled to keep on top of last minute pre-release activities.

A few points worth noting out of the experiment so far...

• It is not easy to limit email activity to the scheduled two or three hours a day. Ideally the graph above should be mostly flat. While part of this is inevitably due to the release activities, I'll try harder going forward to stick to the scheduled email times.
• While the total times may have fluctuated more than I wanted, I did (mostly) manage to contain my email activities to bounded windows of time within the day, instead of checking emails every three minutes all day long. This has helped a great deal. Even while spending nearly half my hours on email, I've managed to get many other non-email tasks done more productively than before. This part has been a success and I highly recommend it. Shut down that email client!
• I found myself doing three (or even four) email sessions per day. This is too many. I need to more strictly limit myself to reading email only twice a day, at the beginning and end of the day. If these sessions need to be longer it is better to make them longer but stick with only two. Whenever I started inserting email tasks in the middle of the day, it fragmented my concentration too much, making the day less productive.
• I'm convinced the ideal arrangement is to do one single email session per day, at the end of the day. That way all the concentration disruption occurs after the days work is done, so it does no harm. The end of the day is also a good time to be entering new tasks into the to-do list so they'll be there tomorrow. Given our distributed time zones it is difficult to do only one email session per day, but that would be ideal. Maybe I'll try that at some point.

As a longer term goal I need to think of ways of reducing the time spent on email. Not sure how to do that yet but spending 45% or even "only" 37% of all working hours on email is totally insane. I suppose email overload is inevitable at a large company with tens of thousands of employees (all of whom, it seems at times, are emailing me) but there has to be a better way. I suppose I could cap my email time to an hour a day and let whatever goes unread just go unread. I'm sure people will be unhappy but will that unhappiness be greater than my productivity gain at doing real work? It's all about tradeoffs, after all. Hard to say what's worse.

Posted at 08:21AM Aug 12, 2009 by jyri in Other  |  Comments[3]

What's Taking So Long

While Sun's Web Server has a very nice threading model, once a worker thread is processing a specific request it will continue working on that request even if it takes a while or blocks.

This is rarely an issue. Static content is served very quickly and code which generates dynamic application content needs to be written so it responds promptly. If the application code takes a long time to generate response data the site has more problems than one, so the web application developers have a motivation to keep it snappy.

But what if you do have a bad application which occasionally does take a long time? As requests come in and worker thread go off to process them, each long-running request ties up another worker thread. If requests are coming in faster than the application code can process them, eventually the Web Server will have all its worker threads busy on existing connections.

As you can infer from Basant's blog entry, the server will still continue accepting new connections because the acceptor thread(s) are separate from the worker threads, so it is still accepting new connections. But there won't be any spare worker threads to take that new connection from the connection queue.

If you're the client making the request, you'll experience the server accepting your request but it won't answer for a (possibly long) while. Specifically, until one of the previous long-running requests finally completes and a worker thread frees up to take on your request (and of course, there may be many other pending requests piled up).

If this is happening with your application one option is to check the perfdump output and see which requests are taking a while. But, as these things are bound to do, it'll probably happen sporadically and never when you're watching.

So how can we easily gather a bit more info? It's been said countless times but always worth repeating.. dtrace really is the greatest thing since sliced bread (and I like bread). I can't imagine attempting to maintain a system without dtrace in this day and age, it would be limiting beyond belief! One of the many key benefits is being able to gather arbitrary data right from the production machine without any prior preparation (such as producing debug builds) or downtime or even any access to the sources you're diagnosing.

So in that spirit, I tried to gather a bit more data about the requests which appear to be taking a while using dtrace and without attempting to look at what the code is actually doing (well, also because I only had fairly limited time to dedicate to this experiment so didn't want to go looking at the code ;-). Although, I should mention, since Sun's Web Server is open source you certainly could go review the source code if you wish to know more detail.

So what am I looking for? Basically I'd like to know when the worker thread starts on a request and when it is done with it. If the time between those two grows "too long", I'd like to see what's going on. Sounds simple enough. Searching around a bit I saw Basant's article on dtrace and Web Server so using his pid$1::flex_log:entry as an exit point seems like a suitable thing to try. I didn't find (on a superficial search, anyway) a mention of an adequate entry point so instead I took a number of pstack snapshots and looked for something useful there and wound up selecting "pid$1::__1cLHttpRequestNHandleRequest6MpnGnetbuf_I_i_:entry" (ugly mangled C++ function name). With that, ran the following dtrace script on the Web Server process:

% cat log.d
#!/usr/sbin/dtrace -qs

pid$1::__1cLHttpRequestNHandleRequest6MpnGnetbuf_I_i_:entry
{
  self->begin = timestamp;
  printf("ENTER %d, %d to n\n", tid, self->begin);
}

pid$1::flex_log:entry
/self->begin/
{
  self->end = timestamp;
  printf("DONE %d, %d to %d\n", tid, self->begin, self->end);
  self->begin = 0;
}

This gets me entry/exit tick marks as the threads work their way through requests. On a mostly unloaded server it's easy enough to just watch that output, but then you're probably not experiencing this problem on an unloaded server. So we need a little bit of helper code to track things for us. Twenty minutes of perl later, I have

#!/usr/bin/perl

$PATIENCE = 9;                  # seconds - how long until complains start

$pid = shift @ARGV;
$now = 0;
$npat = $PATIENCE * 1000000000;

open(DD, "./log.d $pid |");
while (<DD>)
{
    chomp;
    ($op, $tid, $t1, $t2) = /(\S*) (\d*), (\d*) to (.*)/;
    if ($t1 > $now) { $now = $t1; }

    # dtrace output can be out of order so include start time in hash key
    $key = "$tid:$t1";          

    if ($op eq "ENTER") {
        if ($pending{$key} != -1) {
            $pending{$key} = $t1 + $npat; # value is deadline time
        }

    } else {
        $took = (($t2 - $t1) / 1000000000);
        if (!$pending{$key}) {
            $pending{$key} = -1; # if DONE seen before ENTER, just ignore it
        } else {
            delete $pending{$key};
        }
    }

    # Once a second, review which threads have been working too long
    # and do a pstack on those.
    # 
    # Note: we only reach here after processing each line of log.d output
    # so if there isn't any more log.d output activity we'll never get here.
    # A more robust implementation is left as an exercise to the reader.
    #
    if ($now > $nextlook) {
        $c = 0;
        foreach $k (keys %pending)
        {
            if ($pending{$k} != -1 && $pending{$k} < $now) {
                ($tid, $started) = $k =~ /(\d*):(\d*)/;
                $pastdue = ($now - $started) / 1000000000;
                print "=================================================\n";
                system("date");
                print "Thread $tid has been at it $pastdue seconds\n";
                system("pstack $pid/$tid");
                $c++;
            }
        }
        if ($c) { print "\n"; }
        $nextlook = $now + 1000000000;
    }
    
}

The perl code keeps track of the ENTER/DONE ticks (which may occasionally be out of order) and if too long (more than $PATIENCE) goes by, gives you pstack output what's going on.

I don't actually have a suitably misbehaving application so I'll leave it at that. If I had a real application issue, it'd be useful to fine tune the dtrace script to key off of more specific entry and exit points and it'd also be useful to trigger more app-specific data gathering instead of (or in addition to) the pstack call (for instance, checking database availability if you suspect a database response problem, or whatever is suitable for your concrete application).

dtrace is like lego blocks, there's a thousand and one ways of coming up with something similar. Care to try an alternative or more efficient approach? Please share it in the Web Server forum!

Posted at 09:43PM Aug 11, 2009 by jyri in WebServer  |  Comments[0]

The World of Web Stack (1.5)

As you may have seen, Web Stack 1.5 is out. Go ahead and give it a try!

One of the slightly confusing parts of the Web Stack distribution is that it varies by platform so there are several ways of "giving it a try". So it might be worth summarizing how and where to get it:

If you are on OpenSolaris 2009.06

Web Stack 1.5 is integrated into OpenSolaris 2009.06 out of the box. There is nothing to download. Simply install your favorite Web Stack components via the pkg(5) command. One shortcut is to install the 'amp' metapackage which will bring in a number of AMP-related Web Stack components (check the amp manifest to see precisely which ones it includes).

% pfexec pkg install amp

Note the 'amp' metapackage doesn't install all Web Stack components and you can just as well install only the ones you need, individually.

Yes, I realize OpenSolaris 2009.06 shipped last month! That means Web Stack 1.5 has been available in OpenSolaris for over a month now. So what is this week's announcement all about?

If you are on Solaris 10 or RedHat Linux

Well, unfortunately our shipping dates are a bit out of sync on different platforms (this is something I want to get aligned for the next time around) so what we're announcing this week is the availability of Web Stack 1.5 components on Solaris 10 and RedHat Linux.

Download link for Web Stack 1.5 for Solaris 10 and RedHat Linux

There are two different packaging formats you may download for these platforms: native packages and update center images.

The native packages (svr4 for Solaris 10 and rpm for RedHat Linux) are what you'd expect, similar to the Web Stack 1.4 packages (and yes, you can upgrade your previous install if you wish).

The update center image is new in this release and it is quite interesting. Instead of downloading actual Web Stack components you only download an IPS user image which you may unzip anywhere you like, such as in your home directory. From within this user image you will then invoke the pkg(5) CLI to download and install those components you wish to use (there is also a GUI, updatetool, if you're into that).

Yes, this means you can install Web Stack components into any location you like while running as your regular nonprivileged (non-root) user. This is quite nice for experimenting and development work. Please refer to the README and documentation for details. Give it a try and let us know how you like it.

The other brand new cool thing in 1.5 is the Enterprise Manager GUI, which provides a really nice monitoring interface to component statistics.

Others from my team have written in more detail about various features so I won't repeat that here, just check these out:

• Brian has a release announcement.
• CVR also has a release announcement and some notes on the update center packaging.
• Sriram wrote about the release, how to install it and the Enterprise Manager.
• Irfan wrote about the Enterprise Manager and its navigation panel.
• Jeff wrote about Apache httpd updates in 1.5.

Finally, please note that while the full marketing name this quarter is Glassfish Web Stack, the Web Stack product is separate and completely unrelated to the Glassfish Application Server! I realize this has been the source of much confusion lately, particularly at OSCON last week.

Hopefully this helps clarify a bit how and where to obtain Web Stack 1.5! With that out of the way, now go give it a try!

Posted at 08:06PM Jul 30, 2009 by jyri in WebStack  | 

Web Stack at OSCON

It's been so hectic here lately I actually forgot to blog about this before it happened but thanks to everyone who attended our ( CVR and myself) Web Stack session at OSCON 2009 this week.

As we mentioned, we should be announcing the availability of Web Stack 1.5 'real soon now', watch this space for the details...

Posted at 04:47PM Jul 24, 2009 by jyri in WebStack  | 

Web Server 7 Meets Slowloris

Lately there's been some noise about slowloris, a perl script which sends HTTP requests slowly. While there's nothing new about this technique, I've been asked about it a few times so I wanted to show how easy it is to protect against it if you're lucky enough to be using Sun's Web Server 7.

In a nutshell, the script opens a connection to the target web server and sends valid request headers and then continues to send more headers, slowly. Specifically, it first sends:

GET / HTTP/1.1
Host: $hostname
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.503l3; .NET CLR .0.4506.2152; .NET CLR 3.5.30729; MSOffice 12)  
Content-Length: 42
X-a: b

Then it continues to send:

X-a: b

after every $timeout delay. It has a default $timeout of 100 seconds but you can change this with -timeout switch.

Let's look at the more general cases here instead of just slowloris specifically.

The most rudimentary form of this attack is to open a connection to the web server and either don't send anything or send a partial request and nothing else after that (as described above, this is not what slowloris does).

You'll want your web server to eventually time out and close the connection if this happens. In Web Server 7 this is controlled by the io-timeout element in server.xml. The default value is 30 (seconds). Let's try it:

% date;telnet localhost 80;date
Mon Jun 29 19:05:43 PDT 2009
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection to localhost closed by foreign host.
Mon Jun 29 19:06:14 PDT 2009

As you can see, 31 seconds went by before the connection was closed. You can change io-timeout to be shorter if you wish:

  <http>
    <io-timeout>15</io-timeout> 
  </http>


% date;telnet localhost 8080;date
Mon Jun 29 19:15:12 PDT 2009
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection to localhost closed by foreign host.
Mon Jun 29 19:15:27 PDT 2009

Above I changed the io-timeout to 15 and indeed it took 15 seconds before closing the mute connection. Let's try the same thing but send a partial request:

% date;telnet localhost 8080;date
Mon Jun 29 19:14:38 PDT 2009
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.1
Host: localhost
HTTP/1.1 408 Request Timeout
Server: Sun-Java-System-Web-Server/7.0
Date: Tue, 30 Jun 2009 02:14:54 GMT
Content-length: 148
Content-type: text/html
Connection: close

<HTML><HEAD><TITLE>Request Timeout</TITLE></HEAD>
<BODY><H1>Request Timeout</H1>
The server timed out waiting for the client request.
</BODY></HTML>Connection to localhost closed by foreign host.
Mon Jun 29 19:14:54 PDT 2009

Ok, let's try to make the attack more interesting. Instead of just going silent, the client can continue sending more request data, just slowly. This is what slowloris does. As long as the client sends a little bit of valid request data often enough to not get disconnected by the timeout it can hold on to the connection.

Fortunately Web Server 7 also monitors the time it takes to receive all the request headers. This can be configured using the request-header-timeout element in server.xml. This can be used to defeat a slowloris-type attack. Even thought the slowloris request never actually completes (since it just keeps sending more headers forever), Web Server 7 will stop waiting and close it off after request-header-timeout seconds go by.

  <http>
    <request-header-timeout>5</request-header-timeout> 
  </http>

Of course, if you set request-header-timeout to 5s you could then run slowloris with a -timeout of less than 5 seconds. However, this quickly starts to defeat the premise of this style of attack. The idea behind a slowloris-style attack is to attempt to tie up the web server quietly without the client having to generate hundreds or more connections per second. For fun, I set my request-header-timeout to 1s and ran slowloris with a -timeout of 1s. The result is the client machine uses up all its CPU generating new connections while Web Server 7 continues to be happily responsive.

A variant of this attack is to send a POST request, send all the request headers and then start to send the body data, slowly. Note that slowloris does not implement this (the -httpready flag sends a POST instead of a GET, but it continues to send X-a:b request headers, not request body data). However it is easy enough to write a tool to do this instead.

If you encounter that scenario you're in luck because Web Server 7 also monitors the time for the request body to arrive and you can set a timeout on that as well, using the request-body-timeout element:

  <http>
    <request-body-timeout>5</request-body-timeout> 
  </http>

That's all there is to it to protect against slowlaris and similar slow-client attacks if you're using Sun Web Server 7! Enjoy!

Posted at 10:52PM Jun 29, 2009 by jyri in WebServer  |  Comments[8]

Email Backlog

It's no secret that email overload is a problem these days, here's just a few of many articles on the topic:

• http://www.nytimes.com/2008/06/14/technology/14email.html?_r=1
• http://www.thepersonalfinancier.com/2008/11/email-becoming-distraction-heres-how-to.html

A quote from the second article above is particularly interesting (or scary):

   In this study Dr Jackson found that it takes an average of 
   64 seconds to recover your train of thought after interruption 
   by email. So people who check their email every five minutes 
   waste 8 and 1/2 hours a week figuring out what they were 
   doing moments before.

In nearly 20 years (wow!) of reading lots of email daily this has never been much of a problem for me though. I always managed to keep my inbox almost empty from day to day (I long considered 100 emails to be the maximum threshhold to ever have pending in the inbox).

Thinking back, I'd say historically the bulk of my incoming email has been either

• Administrivia (meeting announcements and such): quickly dispatched without thought or mental interruptions
• Engineering content, directly related to whatever I'm working on that day: these take time to read and process but since the emails are relevant to the current project they don't cause a mental context switch and may even help further the project at hand so there is a net win

As resources get tighter and I find myself filling more and more roles simultaneously the dynamic has changed in the last 6-9 months or so. From a perpetually clean inbox I've now gone to a significant backlog. Even more annoying is that I find there are many days where all I get to do is read email!

After some months it is clear this is not a temporary crunch, so I need to change strategies from what has worked in the past. I spent some time monitoring my email activity to figure out what is different. It's not really quantity, I've always received lots of email but it hasn't been a problem. The key difference appears to be that now I'm involved in many projects each one with many unrelated trains of discussion.

As emails arrive, each one is more often than not unrelated to the previous ones and also unrelated to what I'm actually trying to get done at the moment. And thus, I find myself facing the case made in the Dr. Jackson study quoted above.

As each email arrives I read it and start thinking of that particular project/problem for a few moments (a few seconds to a minute or two). It is not enough time to solve or address the issue, just enough to get distracted. Hoping to get back to the real work I was doing instead of spending more time on this new train of though, I don't actually process the email, so it remains in the queue.

By then, several other emails have arrived so I repeat the cycle with each one. By the time I finally get back to what I was actually working on, that project is so many mental context switches behind I no longer have any idea what I was doing and need to spend several minutes getting back into it. By which time, of course, ten more emails have arrived... and the cycle repeats all day.

So I need to address the interruption and context switch problem. A few weeks ago I started to allocate limited time to email. Specifically:

• Only read email in specific blocks of time preallocated for email on that day.
• If I can answer or resolve the issue in less than 5 minutes, do so right then, within the time allocated for email handling.
• If it's going to take any longer than that, add a task to the bottom of my to-do list and move on.
• The rest of the time, quit mutt and resist all urges to go look at email.

I started by allocating two hours a day to email, one in the morning and one in the afternoon. Quickly it became apparent this is not enough to keep up, so I increased it to three hours. I'll gather more data before settling on the final timing but looks like it'll have to be a bit over three hours a day for email processing.

Here's a graph, showing only a few days from last month. I'll post another one with much more data once some more time goes by so I have more numbers. The yellow area is my current email backlog and the blue line is the number of minutes a day spent processing email.

Posted at 12:22AM Jun 16, 2009 by jyri in Other  |  Comments[2]

Web Stack Deep Dive at CommunityOne

In addition to the unconference, Web Stack will also have an afternoon-long deep dive session at CommunityOne. It will be on Wednesday (June 3rd). Here is a link to the deep dive sessions page.

As of today, the agenda shown on the above page is fairly preliminary and we'll be tailoring it depending on the schedule of the speakers as well as feedback we get in the next few days on topic interest (so feel free to let us know if there is anything in particular you want explored in depth - webstack-discuss@opensolaris.org).

It'll be informative and a good chance to meet several of the Web Stack component developers and maintainers, so I hope to see you all there!

Posted at 09:42AM May 15, 2009 by jyri in WebStack  |  Comments[2]

Web Stack Unconference (CommunityOne)

At the Web Stack webinar this week I mentioned the free unconference on the Sunday before CommunityOne (Sunday May 31st). Here is the signup sheet:
http://wikis.sun.com/display/GlassFishConferences/GlassFish+2009+unconference+planning

As noted in the above page, you don't need to be a participant in CommunityOne nor JavaOne in order to attend the unconference. Just sign up on the above page (add your name to the table) and show up! If you have an interest on any particular Web Stack topic areas please add them to the bottom of the page.

See you there!

Posted at 08:39PM May 14, 2009 by jyri in WebStack  | 

Web Stack at the MySQL Conference

Kind of a last minute announcement, but for those at the MySQL Conference today, CVR and myself will be hosting a Web Stack BoF tonight. We'll cover Web Stack in general but of course with an eye on MySQL on OpenSolaris in particular. We should have a few demos from Sriram and some slides, but mostly just open to any Web Stack discussion. Join us at 7:30pm!

Posted at 10:30AM Apr 22, 2009 by jyri in WebStack  | 

Web Stack Road Trip (2)

A couple new opportunities to hear about Web Stack are coming up this month..

Next week (March 10th) as part of the 'Glassfish Boot Camp' here in Santa Clara we'll have a hands-on lab session on Web Stack. We'll do some quick demos and I expect there to be open time for questions, discussion and trying anything you might be curious about. Unfortunately (only if you are not signed up ;-) looks like the event is sold out by now.

The week after (March 18th) I'll be at CommunityOne East in New York City where we'll have a session titled 'OpenSolaris and the Web Stack: Apache, MySQL, PHP, NetBeans PHP IDE and Dtrace Integration'. I'll give a quick intro/background to Web Stack and then we'll have a number of demos. Check it out!

Posted at 04:13PM Mar 06, 2009 by jyri in WebStack  |  Comments[2]