Alan Hargreaves' Weblog
The ramblings of an Australian STSC* Staff Engineer
* Systems Technical Support Centre - The group I work forTags
(update 1) acoustic blues bugs cec2007 china contention cringley dogs dtrace earthquake encumbered-binaries funny google guitar halloween huron install kids linux locking music mysql niagra openjava opensolaris oracle patches percussion redhat secondlife security solaris sun sxcr t2 t2000 ufs upgrade youtube
Tuesday Nov 20, 2007
Mutex Contention vs number of cpus
I've had a few cases recently that have brought this issue to the fore.
It's amazing how many people think that the answer to all performance issues is to simply throw more cpu at the problem.
Let's work through this thought (and this holds true for other queuing type locks too).
- On Solaris, if the mutex holder is on proc, the waiter spins instead of blocking and sleeping
- If we have sufficient threads wanting the same mutex, we can quickly utilize all cpus in a box
- The more threads we have in the queue for a mutex, logically the longer any given thread will take to progress through this queue to obtain it.
What does this tell you about what is likely to happen if you add more cpus into the mix?
It's relatively obvious now, isn't it.
- More cpus stuck spinning in kernel space
- Longer mutex queues
- Longer average time to obtain mutex for each thread that wants it.
The obvious consequence being that adding more cpus can actually have the effect of making the problem worse.
Sigh.
Posted at 11:23AM Nov 20, 2007 by Alan Hargreaves in OpenSolaris | Comments[2]
Subscribe!
Is there a way to improve the locking in the case you've hit ? For example, replace the plain mutex by a version which allows multiple readers concurrently (may drain the queue faster) ... or somehow re-arrange the locking...
... offtopic: Does Solaris allow memory barriers in the kernel ?
Posted by Roland Mainz on November 21, 2007 at 03:07 AM EST #
Yes, you can use mem barriers in the kernel. Yes, the solution to this type of problem is to re-work the code in question to avoid contention, use a different locking approach, ...
Posted by Nico on November 21, 2007 at 05:49 AM EST #