Resource controls and agent-influenced design

Eric mentioned the agent LWP technique a few days ago. I thought I would point out that the resource controls facility was expressly designed with the agent's capabilities in mind. This choice lets us have an interface like:

int setrctl(const char *controlname, rctlblk_t *old_blk, rctlblk_t *new_blk, uint_t flags);

where the action of setting a resource control value applies to the current process or a process collective that encloses it (like a task, project, or zone). (As opposed to an interface that requires an (idtype_t, id_t) pair to specify its target.) Making this choice means we don't end up having a lot of difficult locking scenarios, as we just insert an agent LWP into a victim process to modify the settings on its task.

Resource controls, when they were introduced in S9, brought in some new useful controls, like task.max-lwps and project.cpu-shares, as well as handling both ulimit(2) and setrlimit(2) calls in a compatible fashion. But S10 has seen the introduction of a number of very useful controls: for instance, the System V IPC tunables have been replaced by a set of resource controls, so that you can dynamically change these settings (and apply different constraints to different projects) without a reboot. Dave Powell deserves the credit for this much appreciated piece of work. New controls associated with cryptographic capabilities, zones, and general resource handling have also appeared. And Steve Lawrence spent a fair bit of time looking at making prctl(1) output actually human-readable.

So it's all much improved from the initial work.

Assembling a collective viewpoint

Bryan fielded some questions at USENIX today, and in doing so augments some of Andy's first comments on why we are pursuing an open source development model for Solaris.

Quiet in the office

With the DTrace folks at the USENIX Technical Conference, a contingent at JavaOne, and the remainder focussed on finishing their work for Solaris 10, it's pretty quiet at the office. Oh, except for the fact that my team's multi-year project is wending its way through the final stages of the processes surrounding Solaris integration. Checklist, checklist, test transcript, architectural updates, performance results, status meetings. And maybe one or two more checklists. But we're almost there.

Resource pools and XML/XSLT gymnastics

Since I mentioned resource pools in my previous post, I thought I should also point out that Gary Pennington has been presenting some resource pool tricks. Since resource pools uses XML as a marshallable configuration format, you can apply the many "X*" technologies to the document to realize alternate views of an active configuration.

Why predictable?

I titled my blog "Predictable" more out of genuine interest about making systems more so, than any particular cynicism about how things (any things) turn out. We have a pretty established language about desirable system attributes: reliability, availability, serviceability, and performance. Predictability is a courtier of all of these, but really doesn't dominate any of them--it's almost a separate quality.

We've been working on resource management for a while now, with an eye to allowing the construction of more predictable servers. Most of this work has been adding various mechanisms to the OS to prevent resource denial of service opportunities, or mechanisms that reserve or fairly schedule resources among competing services, such that the system can respond fairly smoothly until we reach some level of overcommitment. Now that we've got the basic mechanisms (and more are coming), we can start to examine how we can layer automation on top of them, and push out the boundary where true overcommitment occurs, without administrator intervention.

The current edition of Solaris Express contains the first of these features, dynamic resource pools, which you can use in combination with zones to have your consolidated systems smoothly react and reassign processors based on system load and relative importance. (You can also use the fair share scheduler with zones, if you don't need to reserve or cap some absolute amount of processing capability for each zone).

We now have a system that can respond elastically across a wider range of load scenarios, without compromising some minimal expectations regarding quality of service with respect to one resource. Our solution, however, can handle multiple resources, and we'll see how that "predictability product" allows you to run your systems closer to maximal utilization in the future.

There are lots of problems in this space, connected with specific operating system resources (and the notion of a resource itself), how resources map from one layer in a stack to the next, how end-to-end scenarios play out within a participating host, ... But it's clear that predictability engineering is a piece of operating systems development--and, if you can get some time to think about it, it's a lot of fun.

A little quiet

There's now at least two good reasons for my hiatus.

Questions for the audience: visible project leaders

One of the interesting attributes of open source projects is that, once they reach a certain critical size, they still may or may not have a single, visible leader. I suppose by a visible leader I mean an individual that the project's community views as having authority over the project's direction, architecture, and implementation. For other projects, there is a public or private group that possesses that authority. We've also seen projects, either at the time of a fork or in their normal course, transition from one leadership class to the other.

The primary question that we might formulate is "does the presence of a single, visible leader affect the chances for success of an open source project?". But this doesn't really probe the issue very deeply, as success for an open source project can be defined very differently by the project, by an external observer, or by a customer or investor in the organization(s) that use or sponsor the project. We could define success by size of the user community, increase in likelihood of the project's longevity, introduction of desired features or elimination of defects, or grant renewals, and so forth.

If we look at the first of these, we might conclude that the size of the user community, although connected to project utility and quality, is probably also influenced by communications about the project. Since stories about individuals are usually more compelling than those about committees, perhaps it's easiest to take advantage of the comfortable "leader-as-project" viewpoint and let those stories--be they formal news coverage or blogging or word of mount--circulate. But it's less clear that the identification of the individual with the project ("I am the state.") and the authority for project direction need to travel together.

What do you think? Does the leadership structure (or even the celebrity of the author) associated with a project influence your choice of that software, open source or otherwise? Are certain structures more appealing from your perspective?

Thanks.

A bug from a blog

blogs.sun.com has already improved Solaris a bit: Moazam Raja mentioned that which(1) isn't doing what one might expect. The newly minted bug, 5061831, will give us a place to discuss how we might fix this issue, gives one suggested fix, and a workaround using ksh(1)'s whence builtin.

Introductions

Although I've been blogging privately and on the web statically for a while now, it's pleasant to have a location to discuss Solaris and Sun specific information. I work on a couple of different projects, of varying sizes; in the past, I've done (I believe) interesting work in resource management, routine work fixing various bugs, and foolish things like rewriting sort(1) from scratch. (I'm particularly proud of the fact that I (with much support) integrated Perl into Solaris 8, and then found someone much more capable to make that a particular competency of Solaris.)

How Sun participates, specifically via its technical staff, in community development situations is of much interest to me; trust me when I say I'll revisit this topic later.

Small talk: Presently, I'm an engineer in the Solaris Kernel group. I moved from Providence, RI to the Bay Area when I started work for Sun in 1997. (I was studying computational high energy physics at Brown, but decided not to pursue the series of post-doc positions traditional at the time.) My undergraduate degree is in engineering science at the University of Toronto. I was born, raised, and primarily and secondarily educated in southwestern Ontario.