Trond Norbye's Weblog
Friday July 10, 2009
Memcached 1.4.0 released!
We released memcached 1.4.0 earlier today! It is more than two years since we started the work on one of the most important part of this release, the binary protocol. I guess a lot of users doesn't really care about the binary protocol, but it makes the implementation of other features easier (the replication in libmemcached is only available if you use the binary protocol).
Other highlights in 1.4.0:
- Performance improvements (removed lock contention)
- Run without privileges on Solaris
- More statistics information
Check out the full release notes at http://code.google.com/p/memcached/wiki/ReleaseNotes140.
Posted at 12:18PM Jul 10, 2009 by trond in Memcached | Comments[3]
Monday July 06, 2009
Scale beyond 8 cores?
If you try to benchmark a memcached server, you will see that it scales relatively good up to 4 threads, and if you go beyond 8 threads the throughput will start to drop. People have been talking about the scalability problems of the memcached server on the mailing lists, IRC and on the hackathons with various solutions to the problems. If you look at the implementation of the memcached server, you will see that it use only a limited set of locks:
| Lock | Purpose |
|---|---|
slabs_lock | The internal memory management of memcached use one single lock to protect access to the internal memory allocator. People tend to want to split this mutex into a separate mutex per slab class, but I don't think that this will give you any measurable performance wins. Why? well the _only_ time we will try to access this lock is during an operation that will modify an entry in the cache (add, set, replace, append, prepend, incr, decr). If this lock really is a performance bottleneck, your GET load is lower than your SET load and that is not the typical use pattern of memcached (and I would like to optimize for the common case...) |
stats_lock | In 1.2.x all modifications of statistical information was protected by this single lock. plockstat revealed that we had mutex contention on this mutex, so in 1.4 most of the statistics are collected per thread, and aggregated when you call the stats command. |
conn_lock | Access to the list of connection structures is guarded by this lock. If this lock comes up when you try to benchmark your memcached server, you would be way better off by reusing your connections to the memcached server. |
cache_lock | All access to the internal hash table is protected by this lock, and this is the lock I'm going to talk a bit more about! |
If you look at the implementation of the memcached server, it stores all of the items in one large hash map. Every time we need to insert, delete or search the hash table, we will try to get exclusive access to the entire hash table. And this is pretty much all that memcached does ;-) (You send a request, it looks up the item, and sends it back to you).
So how can we easily fix this problem without rewriting everyting? Well we could partition up the hash into multiple partitions and only lock down a subset of the hash instead of the complete hash. To avoid extra locking to update the LRU (if it spans two partitions) list I decided to let each partition have it's own LRU list. This sounded like a pretty easy fix to implement, so I just grabbed my laptop and implemented it while watching C.S.I on TV with my girlfriend one night :-)
So how did it scale? Zoran Radovic wrote the following blog entry: Scaling Memcached: 500,000+ Operations/Second with a Single-Socket UltraSPARC T2. If you are interested in the code, you can get it from http://github.com/trondn/memcached/tree/partition.
Posted at 01:10PM Jul 06, 2009 by trond in Memcached | Comments[5]
This is a personal weblog, I do not speak for my employer.
