Alan Hargreaves' Weblog

The ramblings of an Australian SaND TSC* Principal Field Technologist

* Solaris and Network Domain Technology Support Centre - The group I work for

Tags

(update 1) acoustic bind birthday blues bugs cec cec2007 cec2008 china cmt contention cringley debugging dogs dtrace earthquake encumbered-binaries extra flash funny google guitar halloween huron install kids linux liveupgrade locking mdb music mysql newyear niagra openjava opensolaris oracle patches patents percussion performance redhat secondlife security solaris sru sun support sxcr t2 t2000 timeslider ufs upgrade virtualbox windows youtube zfs
« Making a Stomp Box | Main | Listening to Ian... »
pageicon Tuesday Nov 20, 2007

Mutex Contention vs number of cpus

I've had a few cases recently that have brought this issue to the fore.

It's amazing how many people think that the answer to all performance issues is to simply throw more cpu at the problem.

Let's work through this thought (and this holds true for other queuing type locks too).

  1. On Solaris, if the mutex holder is on proc, the waiter spins instead of blocking and sleeping
  2. If we have sufficient threads wanting the same mutex, we can quickly utilize all cpus in a box
  3. The more threads we have in the queue for a mutex, logically the longer any given thread will take to progress through this queue to obtain it.

What does this tell you about what is likely to happen if you add more cpus into the mix?

It's relatively obvious now, isn't it.

  1. More cpus stuck spinning in kernel space
  2. Longer mutex queues
  3. Longer average time to obtain mutex for each thread that wants it.

The obvious consequence being that adding more cpus can actually have the effect of making the problem worse.

Sigh.

Posted at 11:23AM Nov 20, 2007 by Alan Hargreaves in OpenSolaris  |  Comments[2]

Comments:

Is there a way to improve the locking in the case you've hit ? For example, replace the plain mutex by a version which allows multiple readers concurrently (may drain the queue faster) ... or somehow re-arrange the locking...
... offtopic: Does Solaris allow memory barriers in the kernel ?

Posted by Roland Mainz on November 21, 2007 at 03:07 AM EST #

Yes, you can use mem barriers in the kernel. Yes, the solution to this type of problem is to re-work the code in question to avoid contention, use a different locking approach, ...

Posted by Nico on November 21, 2007 at 05:49 AM EST #

Post a Comment:
Comments are closed for this entry.