Mostly Harmless

Dingy corners of file i/o

ufs bears in its wake a rat-pack of low-life ioctls. One of these - DIRECTIO_ON allows you to force directio per-file and is used potentially by databases if their app tunable is so set. The idea is that with a properly sized memory area the db can manage its own buffering and avoid the overhead of copying to kernel buffers.

I was asked about dd with small blocksize being extremely slow on some database files but fast on others. The question of why anyone would want to dd a database file and the dangers of un-sparsing such files etc. are another subject The interesting thing is that the directio setting persisted long after the db had closed the file.

The flag is in the inode's i_flag and hangs around in the cached inodes hanging off the dnlc ready to surprise the next program which opens it. Basically only databases should perform io on database files. But if people use per-file directio on other files for other reasons it is worth remembering to clean up afterwards.

Here are the noddies I tested this with:

# cat dio.c
#include <stdio.h>
#include <sys/types.h>
#include <sys/fcntl.h>

int directio(int fildes, int advice);
static char *Usage = "Usage: dio <filename> 1|0";

void
bomb(char *mess)
{
        fprintf(stderr,"%s\n",mess);
        exit(1);
}

main(int argc, char **argv)
{
        int fd, adv;

        if (argc != 3) {
                bomb(Usage);
        }

        if ((fd = open64(argv[1],O_RDWR)) < 0) {
                bomb("Can't open file");
        }

        switch ((int)(*argv[2])) {
            case '0':
                adv = DIRECTIO_OFF;
                break;
            case '1':
                adv = DIRECTIO_ON;
                break;
            default:
                bomb(Usage);
        }

        if (directio(fd,adv) < 0) {
                bomb("could not set advice");
        }

        close(fd);
        exit(0);
}



# cat flu.c
#include <stdio.h>
#include <sys/types.h>
#include <sys/fcntl.h>
#include <sys/filio.h>

int directio(int fildes, int advice);
static char *Usage = "Usage: flu <filename>";

void
bomb(char *mess)
{
        fprintf(stderr,"%s\n",mess);
        exit(1);
}

main(int argc, char **argv)
{
        int fd;

        if (argc != 2) {
                bomb(Usage);
        }

        if ((fd = open64(argv[1],O_RDWR)) < 0) {
                bomb("Can't open file");
        }

        if (ioctl(fd,_FIOISBUSY) < 0) {
                bomb("could not ioctl");
        }

        close(fd);
        exit(0);
}

_FIOISBUSY flushes a file from the dnlc. That's more infomative than just doing "dio foo 0".

Tests

# ls -l
total 10032
-rwxrwxrwx   1 root     other    5120015 Mar 24 16:22 foo
# timex dd if=foo of=/dev/null count=10000
10000+0 records in
10000+0 records out

real        0.09
user        0.04
sys         0.05

# dio foo 1
# timex dd if=foo of=/dev/null count=10000
10000+0 records in
10000+0 records out

real        2.41     ## Whoa! 10000 latencies = 26 times slower
user        0.03
sys         0.31

# flu foo            ## Simulate eventual purging of the inode
# timex dd if=foo of=/dev/null count=10000
10000+0 records in
10000+0 records out

real        0.10     ## Back to normal
user        0.01
sys         0.09

Posted at 08:18PM Sep 20, 2006 by John Alderson in Code  |  Comments[2]