CPU Caps: Control those CPU Pigs     I am thrilled to say that probably I am one of those few lucky souls to get an early experience of the latest and arguably coolest resource control mechanism on Solaris. CPU Caps released a month back by Andrei on opensolaris. Its fresh. I feel 'privileged' ( well I don't like this word when I am setting resource controls in the zone configuration. RFE 6222025 ) :) :)
    I am not one of those experienced senior sys admins, but even then it was a cakewalk to install CPU Cap BFUs and use the two new resource controls, zone.cpu-cap and project.cpu-cap. My observations and remarks here may not be technically perfect, or could even be ridiculed as amature thoughts, but you know what.. I don't care.

At a first glance: it's a simple equation. 1 CPU cap = 1/100 th of one CPU. And it gells well with the Solaris Containers. Let's check out how CPU caps and containers work together.
Let me switch off one cpu from my two way box for a moment.

# psradm -f 0
# psrinfo 
0       off-line  since 05/01/2006 20:23:09
1       on-line   since 05/01/2006 20:23:10

    Just so that the calculations become easy

# prctl -r -n zone.cpu-cap -v 10 -i zone newzone
# prctl -t privileged -n zone.cpu-cap -v 20 -i zone global

    And start nspins in global as well as the non global zone.

# prstat -Z -n4,2
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
348747 root     1176K  768K run     22    0   0:00:43  19% nspins/1
348805 root     1176K  768K run      1    0   0:00:07  10% nspins/1
116413 root       15M   12M sleep   59    0   0:17:40 0.2% java/35
348754 root     3336K 3024K cpu1    46    0   0:00:00 0.2% prstat/1
ZONEID    NPROC  SIZE   RSS MEMORY      TIME  CPU ZONE
     0       78 7931M 7483M    99%   2:38:38  20% global
     2       29  103M   42M   0.5%   0:01:08  10% newzone

    With the CPU Caps on, they just refuse to go up & take the entire CPU which they would have done otherwise. Now after I bring back my CPU 0, within seconds I get the results in prstat

# psradm -n 0
# prstat -Z -n4,2
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
348747 root     1176K  768K cpu1     1    0   0:01:26 9.6% nspins/1
348805 root     1176K  768K wait     1    0   0:00:32 5.4% nspins/1
116413 root       15M   12M sleep   59    0   0:17:41 0.2% esd/1
109924 noaccess  220M  182M sleep    8    0   0:17:56 0.1% java/35
ZONEID    NPROC  SIZE   RSS MEMORY      TIME  CPU ZONE
     0       78 7931M 7483M    99%   2:39:27  11% global
     2       29  103M   42M   0.5%   0:01:33 5.4% newzone

This is so cool.

• When it gets onto all our lab machines, first thing I would do is create projects for those who mess with me, and put a cap 1. ;) ;) Well, I guess there are probably better things to be done.
• Let me show this to all those datacenter admins handling clients who need to be given fractional CPUs based on their investment.
• I feel this is going to be very useful in performance tuning related experiments, benchmarking etc.
• I guess I should get local lab admins interested in this. They can use it a number of ways.
• I remember one QA who had to test our app on a machine with 450 MHz CPUs. But all he had were some latest sun blade 2500 with 1280 MHz or more or less. I am wondering should I show this magic to him ? Will that be appropriate ? He can set a cap at a fractional value of 1280 to get 450 equivalent performance !! haw haw
• In some situations where one needs to divide the host's CPU power into smaller partitions in terms of one CPU units, Pools is a solution with hard resource boundaries. But In some situations, caps can be preferred over pools as discussed in Pools Vs CPU Caps section below.
• In order to use caps, processes don't need to belong to a particular sched class. ( Well I don't count RT here )
• I talked to one of the senior guys here who is an experienced sysadmin, and I was able to explain it faster as compared to the cpu shares which provide minimum guaranties in a totally utilized environment. Well, personally I find cpu shares nearly equally cool if not better
• CPU Caps are absolute values. switching off a CPU and getting % increase in prstat is fun ! Also, projects don't get their project.cpu-cap out of the zone.cpu-cap. This is dissimilar to zone.cpu-shares and project.cpu-shares relationship.
• Moving around processors across pools doesn't impact cpu caps ( as long as the value is less than poolsize*100 )
• Pools, Shares and CPU Caps:
  Given a situation to restrict the CPU usage below a limit, shares can be used both ways, as lower and upper limits if the cpu cycles available are being fully utilized. An appropriate way is to create pools which set hard boundaries, although in terms of complete CPU numbers but on a resource rich box with 20+ 30+ CPUs, pools with few CPUs create small fractions of the entire CPU power. Now, CPU caps appear to be an interesting solution there.
  • It can go upto 1/100 th of a CPU. An imp advantage over the pools.
  • But most importantly, since it can put virtual limit in units of fractional CPUs without having to create hard boundaries, the processes can be scheduled on all the CPUs if they are designed to do so. This is very useful on something like a two way box. e.g. If the requirement is to run an application that shouldn't utilize more than 50% of the CPU power on a 2 way box, the "pool" solution will require creating a pool with size 1 as a result, all the processes will be scheduled on a single CPU. The same goal can be achieved with project.cpu-cap=100 so that if the app is in that project, it's processes can be scheduled on both the CPUs, at the same time the app will not use more than half of the 2 way box. This is awesome !!
  • And the best part is, caps can be clubbed together with pools / shares if need be.
  It's important to understand these three and use them effectively. Shares are relative numbers that provide minimum guarantees when all are competing for cpu cycles. Pools with psets, in a true sense provide both, a reservation as well as insurance shield. An exclusive area with hard borders, no sharing of the cpu resources with those not hooked on to it. CPU Caps, on the other hand provide the upper limit, and allow sharing.

I guess I better put a cap on my thoughts now...

Posted by chiplunkar ( May 05 2006, 10:10:00 PM IST ) Permalink Comments [0]