« Previous day (Sep 11, 2007) | Main | Next day (Sep 12, 2007) »
20070912 Wednesday September 12, 2007

Resource control observability using kstats

One of the things I sometimes miss when using resource controls, is a simple way to see what the current usage of a particular resource control by a project or zone is. While finding out the limit for the rctl is no problem (for that we have prctl(1)), getting the actual usage requires work and implementation knowledge.

For instance, we could get the amount of System V shared memory used by a project using ipcs -Jam and some parsing of its output. Or fire up mdb(1) and lookup the value for kpd_shmmax in the project's kproject_t struct. And, if we wanted to get the usage of another resource control (say the number of lwps), we'd need to use another tool (prstat -LJc) or know that the number of lwps is kept in the kpj_nlwps member. Hardly usable for more than the occasional peek. Plus that relying on kernel implementation details such as these structure members is highly inadvisable as they may change in the future (they probably won't, but they are not stable interfaces so don't rely on them).

The addition of the swap and locked memory resource controls by PSARC 2006/598 Swap resource control; locked memory RM introduced a number of kstats for observability:

  • caps:{zoneid}:swapresv_zone_{zoneid}
  • caps:{zoneid}:lockedmem_zone_{zoneid}
  • caps:{zoneid}:lockedmem_project_{projid}

These kstats have a 'value' statistic for the current limit and a 'usage' statistic that holds the current usage: 

$ kstat -c zone_caps -n swapresv_zone_0
module: caps                            instance: 0     
name:   swapresv_zone_0                 class:    zone_caps
        crtime                          0
        snaptime                        102512.50351337
        usage                           532168704
        value                           18446744073709551615
        zonename                        global

Exposing these values as kstats gives us exactly what is needed, a simple, well defined method to get the limit and usage for a resource control.

To satisfy my curiosity and to see what changes would be needed, I spent some evenings creating a prototype that adds kstats for all project.* and zone.* resource controls. The following extra kstats are available in the prototype:

  • caps:{projid}:contracts_project_{projid}
  • caps:{projid}:msgids_project_{projid}
  • caps:{zoneid}:msgids_zone_{zoneid}
  • caps:{projid}:nlwps_project_{projid}
  • caps:{zoneid}:nlwps_zone_{zoneid}
  • caps:{projid}:ntasks_project_{projid}
  • caps:{projid}:semids_project_{projid}
  • caps:{zoneid}:semids_zone_{zoneid}
  • caps:{projid}:shmids_project_{projid}
  • caps:{zoneid}:shmids_zone_{zoneid}
  • caps:{projid}:shmmem_project_{projid}
  • caps:{zoneid}:shmmem_zone_{zoneid}

Getting a list of the current usage of all resource controls is now as simple as typing: 

$ kstat -p caps:::usage
caps:0:contracts_project_0:usage        33
caps:0:contracts_project_1:usage        2
caps:0:contracts_project_101:usage      0
caps:0:cryptomem_project_0:usage        0
...
caps:5:nlwps_project_0:usage    108
caps:5:nlwps_zone_5:usage       108
caps:5:ntasks_project_0:usage   15
caps:5:semids_project_0:usage   0
caps:5:semids_zone_5:usage      0
caps:5:shmids_project_0:usage   1
caps:5:shmids_zone_5:usage      1
caps:5:shmmem_project_0:usage   172032
caps:5:shmmem_zone_5:usage      172032
caps:5:swapresv_zone_5:usage    95178752

And now that we have the numbers as kstats, we can use any tool to massage the numbers into a form that suits us. The screenshot below is from a hacked up version of one of the JKstat demo programs and shows a graph of the number of LWPs in all projects and zones during boot and shutdown of a Zone.

T: ( Sep 12 2007, 09:29:14 PM CEST ) Permalink Comments [3]