Trond Norbye's Weblog

Not using CAS? disable them and save 8 bytes pr. item in your cache

The memcached server stores an 8-byte big unique id for each item that is stored (or updated), so that you can ask the server to do some operations on the items if and only if the cache has the value you provide. If your application doesn't use this feature, you are wasting 8 bytes pr object. You may think that eight bytes is not much on modern hardware (even my laptop has 4GB of memory), but the more items you put in the cache the more memory you "waste".

Earlier today I created a patch that allows you to remove the allocation of these bytes if you start memcached with -C. You might ask yourself why I didn't just make this a compile-time-setting, and there is two reasons for that:

1. Compile-time-settings are a pain if you would like to ship a binary, because then you need to ship two versions and the users have to select the correct one
2. It would cause problems for the work we are doing with the storage engines. If we have two different item structures depending if you compile with or without cas support, you need also to create two different engine implementations.

If you look at the diffs in my github repository you will see that it is actually a pretty simple change, but if you don't want to read diffs I'll just highlight the important changes here. The old item structure looked like:

typedef struct _stritem {
    struct _stritem *next;
    struct _stritem *prev;
    struct _stritem *h_next; /* hash chain next */
    rel_time_t time; /* least recent access */
    rel_time_t exptime; /* expire time */
    int nbytes; /* size of data */
    unsigned short refcount;
    uint8_t nsuffix; /* length of flags-and-length string */
    uint8_t it_flags; /* ITEM_* above */
    uint8_t slabs_clsid;/* which slab class we're in */
    uint8_t nkey; /* key length, w/terminating null and padding */
    uint64_t cas_id; /* the CAS identifier */
    void * end[];
    /* then null-terminated key */
    /* then " flags length\r\n" (no terminating null) */
    /* then data with terminating \r\n (no terminating null; it's binary!) */
} item;

What I did was to move the cas_id into the variable part of the structure, so that the struct now looks like:

typedef struct _stritem {
    struct _stritem *next;
    struct _stritem *prev;
    struct _stritem *h_next; /* hash chain next */
    rel_time_t time; /* least recent access */
    rel_time_t exptime; /* expire time */
    int nbytes; /* size of data */
    unsigned short refcount;
    uint8_t nsuffix; /* length of flags-and-length string */
    uint8_t it_flags; /* ITEM_* above */
    uint8_t slabs_clsid;/* which slab class we're in */
    uint8_t nkey; /* key length, w/terminating null and padding */
    void * end[];
    /* if it_flags & ITEM_CAS we have 8 bytes CAS */
    /* then null-terminated key */
    /* then " flags length\r\n" (no terminating null) */
    /* then data with terminating \r\n (no terminating null; it's binary!) */
} item;

This means that we don't have a cas_id member anymore, so I created two "functions" to get and set the value:

#define ITEM_get_cas(i) ((uint64_t)(((i)->it_flags & ITEM_CAS) ? \
                                    *(uint64_t*)&((i)->end[0]) : 0x0))
#define ITEM_set_cas(i,v) { if ((i)->it_flags & ITEM_CAS) { \
                          *(uint64_t*)&((i)->end[0]) = v; } }

The next thing to do was to replace all usage of the cas_id-member with these accessory functions. Since the old offset of the variable data in the structure is no longer true, I had to update the macro's we use to get the other variable parts of the structure:

#define ITEM_key(item) (((char*)&((item)->end[0])) \
         + (((item)->it_flags & ITEM_CAS) ? sizeof(uint64_t) : 0))
 
#define ITEM_suffix(item) ((char*) &((item)->end[0]) + (item)->nkey + 1 \
         + (((item)->it_flags & ITEM_CAS) ? sizeof(uint64_t) : 0))
 
#define ITEM_data(item) ((char*) &((item)->end[0]) + (item)->nkey + 1 \
         + (item)->nsuffix \
         + (((item)->it_flags & ITEM_CAS) ? sizeof(uint64_t) : 0))
 
#define ITEM_ntotal(item) (sizeof(struct _stritem) + (item)->nkey + 1 \
         + (item)->nsuffix + (item)->nbytes \
         + (((item)->it_flags & ITEM_CAS) ? sizeof(uint64_t) : 0))

And that was basically it :-)

Posted at 02:33AM Nov 26, 2008 by trond in Memcached  |  Comments[0]