Keith Bierman's Weblog

Caches Considered Harmful For what seems like forever, designers have been adding more and more cache to systems to reduce latency to memory. This has been successful, but it hasn't been the only approach, but it has been the most typical.

But has it been Good?

• Caches are very energy intensive (essentially large amount of SRAM close to the CPU). The larger they are, the more energy wasteful they are.
• Caches, on average, produce a benefit on the order of sqrt(size), so the heat outpaces the benefit.
• Of course, with heat you pay several times. You pay for the electricity to create the heat, you pay in the system design to cool the device, you pay in the data center to cool the entire system, and you typically pay a price in RAS because heat kills.
• Notably, adding cores (providing enough memory bandwidth has been provided) provides nearly linear improvement in throughput.
• And for cache experts, increasing the associativity increases their effectiveness.
  
• Caches help us avoid dealing with the underlying issue of doing useful work while waiting for memory. Putting off the harder work of innovation, or at least limiting the innovation to the process level rather than the architecture level is a form of laziness.
• When the data one wants isn't in the cache, it's often worse than it would have been had there been no cache (so fancy non-cache polluting loads and stores may be added to the ISA, and compiler, etc.
  

So what are the alternatives to caching?

As in the citations above, the key observation is that if one has additional "threads" ready to do useful work, that work can be done while awaiting the data to be returned (from memory, from cache, from disk, ... wherever) rather than keeping all that hot and possibly expensive iron (silicon) hot. And that's precisely what UltraSPARC T1 does  ).

So when you hear someone making a spurious claim about the UST1 being cache starved, ask them how big a cache they think it should have, and why? What level of associativity? What's the downside? And, of course, point out that the application performance is what counts, and it doesn't support the contention that the UltraSPARC T1 family systems are cache starved.


NB: of course, caches aren't all bad. If you are focused on minimum latency (fastest response time for a single thread) they can be very effective. But if your goal is the most aggregate work for the least power, they are certainly not your friend.

To learn more about caches


(2005-12-08 16:57:41.0) Permalink Comments [2]