If I could just get the stupid thing to build: The blog of Amanda Waite

I think that this blog is turning into a list of my shortcomings, but these kind of silly problems need to be stamped out and you can only do that by documenting them. 

I was testing our Web 2.0 application on Lighttpd/PHP/MySQL and my throughput tailed off at around 800 users, I did all of the usuals, tuned MySQL a little, enabled XCache, used the private interfaces instead of the ones hooked up to our internal network. Still I was seeing the same tail off - I was getting 160 ops/sec for 900 and 1000 users and < 160 ops/sec for > 1000 users.

As I was just quickly ramping this test up I wasn't monitoring iostat during the run, so I enabled it (Faban allows you to specify a whole bunch of tools that you can run on each system in the rig) and ran it again. iostat showed that the root disk was 87% busy on average with avg. service times at around 80ms which is way too high, particularly for a disk that wasn't doing anything. I realised then that either PHP or Lighttpd were writing file upload data temporarily to /var/tmp, the default for both of them. It was Lighttpd in fact, I just added the line (which I really thought I had already done):

server.upload-dirs     = ( "/tmp")

to the Lighttpd config file and restarted.  On Solaris, /tmp is an in-memory file system so you can get real benefits from writing temporary data there over writing to /var/tmp (which by default is on a filesystem). Just make sure that you don't run out of memory or you'll start swapping (writing memory pages to disk, probably your applications pages) and don't use /tmp for ordinary files. UFS and ZFS both have caches which will benifit read/write access to existing files and all of the data in /tmp is lost when you reboot.

The config file option for PHP that causes it to store file upload data to /tmp is:

upload_tmp_dir = /tmp

You can also cause session data to be saved to /tmp with:

session.save_path = "/tmp"