/var/adm/blog
Two wrongs do not make a right ... but two 'set's can make a 'delete'
I've spent some time the past couple days looking through the memcached server code and I noticed something interesting in how we process set commands when the server is configured not to evict items when low on memory(-M). For a split second, the behavior I was seeing in the code seemed wrong to me, but after giving it a bit of thought I realized that the behavior made perfect sense. Without getting into the details upfront, let me show you an example.
Lets start a memcached server. The behavior in question happens when the server is not able to fulfill a set request due to memory constraints, so to make it easier to achieve this condition, lets limit our cache size to 1MB.
smacky:memcached elambert$ ./memcached -m 1 -M |
Now that we have our server up and running, I want to add some data. On my system, I have file called 'afile', its 524,288 byte file that consists of nothing but the letter A. Lets store the contents of this file and associate it with the key 'key1'.
smacky:~ elambert$ ls -ld afile
-rw-r--r-- 1 elambert staff 524288 Mar 25 14:58 afile
smacky:~ elambert$ val=`cat ./afile`; printf "set key1 0 0 524288\r\n$val\r\n" | nc 127.0.0.1 11211
STORED
smacky:~ elambert$
Now, some time passes, and I want to insert the content of file 'bfile' (a 524,288 byte file filled with the letter B) into the cache, again under the key 'key1' (essentially replacing the old entry for 'key1').
smacky:~ elambert$ ls -ld bfile
-rw-r--r-- 1 elambert staff 524288 Mar 25 14:59 bfile
smacky:~ elambert$ val=`cat ./bfile`; printf "set key1 0 0 524288\r\n$val\r\n" | nc 127.0.0.1 11211
SERVER_ERROR out of memory storing object
smacky:~ elambert$
Well, it looks like we don't have enough room in the cache and since we told the server not to evict entries we get an out-of-memory error. So what does this mean for our 'key1' entry. Well, lets retrieve it and find out.
smacky:~ elambert$ printf "get key1\r\n" | nc 127.0.0.1 11211 |
Wow, thats interesting. The cache says it does not have an entry for 'key1', yet we were able to store the 'afile' contents under the key 'key1' and we told the server not to evict an items. Shouldn't the original entry still be in the cache? Well, actually, no. The server did exactly what it should have, which is delete the entry when the second set operation failed.
For some of you this may be obvious, but I suspect others are finding this behavior a little counter-intuitive and are wondering why a failed set would result in delete, especially when we tell the server not to evict items.
But to understand this behavior, we need to realize that memcached is ... well ... a cache. It is not designed to be the sole source of truth in your system. Memcached is just an 'optimization' that aides you in scaling your system and that it should always operate in tandem with a reliable store (might I suggest MySQL for this purpose :-) ). It is that store which acts as 'truth' for your system. And in a well designed system, updates must first be applied to this backend store before being applied to the cache. So even though the second set operation fails, the mere fact that an attempt was made to modify the value of the entry tells the server that the entry for 'key1' is most likely no longer in sync with the our backend --otherwise, why else would we attempt to change the value. Armed with this knowledge, the worst thing the server could do is leave the entry for 'key1' as is. If it were to leave the original entry, than any proceeding get request would get a stale value. A better strategy is to delete the item. By deleting the entry, clients that request the key will incur a cache-miss and simply go to the backend for the correct value.
Another point of confusion is the presence of the -M (don't evict) flag. Some people may interpret this flag to instruct the server to never evict an item. If you read the description of the flag, you'll see that is not the case. This flag only has meaning when the cache has exhausted its memory. If the server is out of space and we have specified the -M flag, the server will not evict an entry to make room, but it is still free to evict entries for many other reasons, including the example discussed above.
Posted at 04:32PM Mar 25, 2009 by Eric Lambert in Memcached | Comments[0]