Liane Praza's Weblog

There was a bit of an internal push for folks to externally blog about CEC2005, the Sun internal Customer Engineering Conference happening this weekend. I don't engineer customers, but I hope I engineer (I use the term loosely here -- software developers are not Professional Engineers) for customers. Still, the sales and service engineers who run the conference will allow me the opportunity to speak about SMF at some really early hour Monday morning.

Still, with all of the technical-content talks that happen at CEC, it's too bad we don't have an external conference to focus on Sun-specific, or Solaris-specific technical information. Would people go? Certainly JavaOne is a raging success -- but the product I work on isn't Java-specific or even Java-focused. I'd be a fish out of water there. In the meantime, we'll keep pushing to attend and submit papers for usual non-Sun conferences. Hopefully a few SMF developers will be able to attend USENIX and LISA this year.

In the meantime, it's good that some CEC2005 attendees will be taking time to blog about the information presented there. Keep an eye out for posts on blogs.sun.com.

I hadn't logged into my Orkut account for a while, but it was time to accept connections of a few more friends over lunch. Poking around at my scraps revealed that there is an entire community there of "Liane"s. In Portugese. Seems that what I thought was an uncommon name is not entirely rare among Brazilian women. Nobody will ever mistake me for Brazilian (too bad, really), it's been a few years since I visited Brazil (highly recommended, by the way -- Rio is a gorgeous, friendly, vibrant city), and my Portugese is nonexistant. Still, maybe I'll pull out the Portugese-English dictionary again and join the forum.

We'll never compete in numbers with the vast Dave conspiracy ¹, but the idea of a Liane-specific support network is still amusing.

¹If you work in tech in the US, there's always at least one Dav[e|id] in close proximity. Usually multiple. I currently work regularly with six of them, rendering the usual technique to differentiate between two useless. Only one of them blogs, further validating the hypothesis of a vast underground Dave network. Clearly they're not ready to go public yet.

After 2 years of burn-the-candle-at both-ends effort on smf, I'm now trying to catch up on at least a little of my neglected vacation time. I had a few days in London (prior to some internal Solaris 10 training), which I'll try to remember to blog about once all my pictures are back. This weekend, my boyfriend and I made a short hop to Salt Lake City for my first time skiing in Utah.

Sunday was beautiful and sunny at Alta and Snowbird, even if the snow was a little heavy. Monday was a bit windy and clouded over, but with the reward of consistent fresh snow. Despite some altitute sickness, we spent the day making figure eights in the powder at Snowbird. Nowhere near good enough to play competitively, but a pretty nice valentine's day nonetheless. I wanted to take a few pictures of our tracks from the lift, but the snow was consistent enough to cover them up every time. A small price to pay for fresh tracks nearly every run.

I'll catch up with a new smf entry here soon.

Some time ago I had to stop using my beloved 1982 Fiat Spider as a daily driver. It is an aging Italian sportscar, so a minor tantrum was to be expected from it once a year or so. But, getting to work was becoming an increasingly stressful endeavor. So, a hunt for a new car began.

Many hours were spent talking me out of waiting for the less-than-reasonable new Elise about to be released in the US. Eventually, I had to cede to the logic of a compromise car. Practical, reliable, maintainable, and even used. I've never been a fan of the Miata's handling, and while the Honda S2000 had the sweetest little engine you've ever heard, its lackluster steering feel would have always left me wanting. I've always loved the BMW M-coupe's looks, and a test drive confirmed everything I'd read about its performance. Lots of power and perfectly predictable to handle. But, no convertible. That was a non-starter.

After a month or two of foot-dragging, I managed to swallow my pride and climb into an M-roadster. Ok, it can probably be forgiven for looking like the Z3. All the power of the M-coupe, but plenty of body twist just waiting to jump out and bite you at the most inopportune moments. What a brute! Still, that's a lot of the appeal, and there were a few reasonably priced low-mileage examples to be found. So, we jumped in and bought one of those examples.

What prompted this useless anectdote? A friend sent along a link to a review from the Car Talk guys. They've, as usual, got it pretty spot on. Fortunately, I haven't run afoul of the law with the beast yet.

Folks are starting to write service manifests for various pieces of software. Figured it might be useful to collect a few here. If I've missed something useful (and I'm certain I have), feel free to drop me a line and it'll be included.

So far, one of my favorites is an article by Peter Tribble and Geoff Gibbs about creating an smf service for postfix. Coolest part is that they actually noticed what we intended and created postfix as an instance of the svc:/network/smtp service. It contains a great step-by-step set of instructions in addition to the manifests and method files. Nice.

Chris Gerhard posted this description for postgres.

Trevor Watson's done some work with squid. [2/11/2005] Wow, and he's already done another one for samba.

Michael Hunter contributed an entire package for freeradius to blastwave including a service manifest if you're running it on Solaris 10. If you're a Solaris user who hasn't heard of blastwave, check it out. It's an invaluable resource.

Most the questions I get about smf(5)'s continued execution of init.d scripts are along the lines of "for how long will you continue to support them?". The answer to that common question is: for the foreseeable future. We haven't yet, and have no near-term plans to file the EOF that would be necessary to remove support for init.d startup scripts. But, one reader (reader? Who am I? Dear Abby? Sheesh, sorry guys.) sent me mail to ask:

> With all of the goodness in smf, why do we still have "legacy"
> Solaris services?
> 
> I can understand having both systems so that there can be a
> transition from the old to the new for those who have invested a
> lot of work into their init scripts.
> 
> I'm afraid I must be missing something.

Not really. We did keep around compatibility for the legacy init.d scripts so that Solaris customers and ISVs don't have to do a bunch of work (it really should be no work, given the Solaris compatibility guarantees) to have their software work on Solaris 10. Those who want to realize the benefits of smf(5) can write a simple service description for their software. Those who can't yet fit it into their schedules don't have to. As mentioned in the quote, it eases the transition.

But, why isn't all of /etc/rc?.d empty for Solaris 10 as Sun delivers it? Honestly, because we just didn't have the time before Solaris 10 shipped. Our team was pretty small, and we didn't get the word out as well as we'd have liked within Sun. We did a bunch (well over 100) ourselves, but we also often need help from the people who own the specific services. Those of you on the Solaris Express train probably noticed smf(5) was a pretty late addition to the release. But, now that it is in the release, our job in selling the benefits and helping folks to convert is a lot easier.

That said, any of you out there under a support contract can certainly help us improve. If you've got a favorite Sun-delivered service that isn't under smf(5) control that you think would benefit, let us know through your standard channels! While I'm happy to file RFEs (that's Request For Enhancement) to get this work done, actual reports from customers carry (justifiably!) more weight than internal requests. You can also comment on this blog entry to let me know about any services in Solaris that you think are particularly important to get under the aegis of smf(5), and I'll see what I can do.

A recurring concern about smf(5) is the configuration repository. To some folks, it resembles the Windows registry too much for comfort. Rather than trying to contrast with the Windows registry or other registries such as GConf, I thought I'd talk about the design choices we made when deciding how to implement the smf(5) repository. Below is the high-level list of design criteria. It may not be complete, but captures a reasonable amount of what we were thinking when designing the repository.

1. Transactional.

   All of smf(5) is designed to be completely restartable from the ground up. Do you doubt? First, try killing all user processes (kill -9 -1) on a non-critical Solaris 9 system -- one that nobody's using, please:

         wands console login: root
         Password:
         Feb  7 13:26:30 wands login: ROOT LOGIN /dev/console
         Last login: Tue Feb  1 14:44:40 on console
         Sun Microsystems Inc.   SunOS 5.9       Generic January 2003
         # ptree
         59    /usr/lib/sysevent/syseventd
         73    /usr/lib/picl/picld
         130   /usr/sbin/in.routed
         149   /usr/sbin/rpcbind
         152   /usr/sbin/keyserv
         162   /usr/lib/netsvc/yp/ypbind -broadcast
         178   /usr/sbin/inetd -s
         199   /usr/lib/nfs/lockd
         201   /usr/lib/nfs/statd
         202   /usr/lib/autofs/automountd
         214   /usr/sbin/syslogd
         222   /usr/sbin/cron
         227   /usr/sbin/nscd
         240   /usr/lib/power/powerd
         251   /usr/lib/utmpd
         263   /usr/sadm/lib/smc/bin/smcboot
           270   /usr/sadm/lib/smc/bin/smcboot
           271   /usr/sadm/lib/smc/bin/smcboot
         268   /usr/lib/im/htt -port 9010 -syslog -message_locale C
           275   htt_server -port 9010 -syslog -message_locale C
         285   /usr/lib/sendmail -bd -q15m
         286   /usr/lib/sendmail -Ac -q15m
         311   /usr/dt/bin/dtlogin -daemon
         312   /usr/lib/snmp/snmpdx -y -c /etc/snmp/conf
           315   mibiisa -r -p 15488
         323   /usr/lib/dmi/dmispd
         324   /usr/lib/dmi/snmpXdmid -s wands
         329   /usr/sbin/vold
         336   /usr/lib/saf/sac -t 300
           339   /usr/lib/saf/ttymon
         337   -sh
          1234  ptree
         340   /usr/lib/ssh/sshd
         # kill -9 -1
   
         wands console login: root
         Password:
         Last login: Mon Feb  7 13:26:30 on console
         Sun Microsystems Inc.   SunOS 5.9       Generic January 2003
         # ptree
         1235  /usr/lib/saf/sac -t 300
           1238  /usr/lib/saf/ttymon
         1236  -sh
           1242  ptree
         # 
         

   You've really little chance of recovering your system without a reboot. Note that even init(1M) has disappeared. Restarting it manually won't even do the trick, as it doesn't maintain its process table in a persistent place in Solaris 9.

   Now try the same kill -9 -1 on a non-critical Solaris 10 system. Again, all user processes are killed, including init, svc.startd, svc.configd, inetd, and everything else. You'll be logged out, but log back in and poke around. You'll see that nearly the entire system comes back (services started by their legacy init.d scripts won't, though). Based on doing this experiment now, I'll be filing bugs against a few services, but the experiment is generally successful. All of the core daemons I mentioned dying have returned (we kill them individually as part of our standard testing), and we give restarting all services a good college try.

   In order to implement restart from the kernel up completely, we needed a transactional backing store for all of our service information, including things like service state. If any of our daemons die halfway through an operation, they need to pick up where they left off when they return. Thus, the repository must be transactional to allow us to implement recoverability.

2. Typed.

   We want to be able to validate that configuration information is at least of the appropriate form. In the future, we expect to be able to do even further validation than just on the type.

3. Single point of access.

   We wanted all configuration and runtime data access through a single API that can be maintained across release boundaries. Flat file administration tools usually allow access through multiple mechanisms -- e.g. editing the file directly or using an admin GUI which edits the file. This type of access reduces the ability to write event-based APIs -- e.g. "tell me if this service has changed configuration or state". While we don't have many of those APIs yet, they're coming. If we allowed vi as a tool to manipulate the repository, there's no precise way to provide the notification API. A single API also decreases the time to write layered administrative tools.

4. Access control.

   Allow a subset of configuration changes to be safely delegated to non-root users, without requiring that all configuration changes be allowed by those users. We, however, didn't include provisions for configuration data to be hidden from unprivileged users or applications. While modification is protected, reading is not.

5. Layerable

   Our configuration store must be designed to support a mix of configuration (with overrides) at the network level (shared among many machines) as well as at the local machine. This isn't here in Solaris 10, but we've designed to allow configuration to span multiple machines. It was important our initial implementation didn't impede that goal. It's easy to imagine that our underlying storage format for local data might not be the same as that for network data.

6. Service/instance model specific.

   This is more of a non-goal, but, we didn't want to design for a general data storage model. We wanted to constrain ourselves to the general service/instance schema that we've designed for smf(5). That isn't to say the schema couldn't be abused for other data, but we didn't design to make it easy. I realize that perhaps nobody's done the service/instance split blog entry yet. I mentioned it in passing in the service developer intro, but will try to write a dedicated entry later.

7. Rollback.

   Allow administrators to easily revert to previous configuration versions. This is sometimes solved manually with a revision control system (e.g. SCCS) and flat text files.

8. Checkable consistency.

   We should be able to confirm on startup that at least the format of the system's configuration data looks sane. Obvious filesystem corruption should be flagged explicitly rather than parsed as lack of or incorrect configuration.

9. Fast.

   It's pretty tricky to implement a structured, typed, and transactional common store as flat files that's still quick enough for the state-change updates we need to do. Binary format files is usually the way to go. Some other projects doing parallel startup only use a binary cache of plain text files, but that doesn't handle the other design criteria we had. I'm sure there will be comments telling me more precise projects that have solved this problem using plain text files, but the ability to leverage other code can decrease development time.

10. Endian-neutral export.

    Allow export of all configuration data in an endian-neutral format, so that configuration can easily be moved from machine to machine, regardless of architecture. An easy way to marshall the data out of a machine-specific format and into a standard format (e.g. XML) was considered sufficient.

11. Embeddable.

    Any open-source solution used must be in a commercial product without licensing/royalty issues. Obviously, writing something ourselves easily gets around this constraint, but that would be a pretty significant additional investment over the implementation we already needed to do for smf(5).

Based on this set of design considerations, we decided on a four-part scheme.

1. Service Manifests: XML service descriptions provide a transportable way to deliver individual services. No knowledge of the underlying data format nor the full service creation API is necessary for simple service delivery.

2. libscf: A library provides the fundamental API which all tools can build on. In addition to providing transactional create/change/delete semantics, this also allows us to write tools which also dump the repository in a standard format. Try svccfg archive (re-direct the output to a file) to dump existing configuration of all services and instances in our standard XML manifest format. While it doesn't contain things like snapshot information, it does provide all the information that's necessary to restore a system to its current configuration.

3. svc.configd: A daemon to manage the data store, providing a single point of access to underlying data for security, layering, etc.

4. Repository/SQLite: A back-end transactional database to provide file-level storage for smf(5) configuration data. To be precise, we've got 2 backing databases. One is for persistent property groups which, well, persist across system restart. It's located in /etc/svc/repository.db. The other is the non-persistent properties, such as states, which don't need to be kept across system restart. The non-persistent database is kept in /etc/svc/volatile/svc_nonpersist.db

We decided to use SQLite for the local-repository implementation because there was simply no need to re-invent the wheel and implement a transactional database ourselves. SQLite fit all of our other design criteria. However, we haven't exposed that implementation in the interfaces. If SQLite no longer fulfills all of our requirements, we'll change to using a different underlying implementation. Existing code based on libscf(3LIB) or svcprop(1) will continue to work unmodified. That's the nice thing about hiding the data format behind a set of standard interfaces.

By the way, now that I've pulled back the covers on our implementation I should give the warning: direct access to the underlying repository is completely unsupported. If you scrog your repository using direct (sqlite) access, you're on your own. If you'd like to take a copy of repository.db and poke around in it, go for it! But, don't muck with the running copy lest you end up with a 'repository corrupt' message.

A number of questions about smf(5) milestones have been surfacing lately, so I'll try to give a summary of the topic and answer a few common questions here.

An smf(5) milestone is really nothing more than a service which aggregates a bunch of service dependencies. Usually, a milestone does nothing useful itself, but declares a specific state of system-readiness which other services can depend upon. One example is the name-services milestone. It simply depends upon the possible name services you might be running:

   $ svcs -d name-services
   STATE          STIME    FMRI
   disabled       Jan_04   svc:/network/rpc/nisplus:default
   disabled       Jan_04   svc:/network/dns/client:default
   disabled       Jan_04   svc:/network/ldap/client:default
   online         Jan_04   svc:/network/nis/client:default

and has no useful actions to perform during the start or stop method:

   $ svcprop -p start name-services
   start/exec astring :true
   start/timeout_seconds count 3
   start/type astring method

   $ svcprop -p stop name-services 
   stop/exec astring :true
   stop/timeout_seconds count 3
   stop/type astring method

The name-services milestone is considered online as long as any name services which are enabled are running. There's also nothing different about these milestones to smf(5), it just sees them as yet-another-service.

We've implemented standard Unix system run-levels in smf(5) using milestones. The single-user, multi-user, and multi-user-server milestones correspond to run-levels S, 2, and 3, respectively. In addition to the runlevel milestones, there are the all and none keywords. These aren't actual services, but shorthand for either the graph with no services, or the graph with all services. This set of five special milestones can either be booted directly to (boot -m milestone=) or reached by running svcadm milestone. As mentioned in a previous entry, the way we reach a limited milestone (any special milestone but all) is to temporarily disable all services which aren't part of the milestone's subgraph.

A common question is why the console-login service is disabled if you boot to a milestone that isn't all. This can easily be determined by looking at console-login's dependents.

   $ svcs -D console-login
   STATE          STIME    FMRI

As there are no milestones which have console-login as one of their dependencies, it won't be started as part of any milestone but all. Fortunately, we'll always start an sulogin(1M) prompt if a login service can't be reached.

So, why are milestones useful then? The most useful milestone is none, for the recovery/exploration scenario I described here. The other use is when doing service development. You can use svcadm milestone to transition to limited milestones then back up without rebooting the system.

There's a large omission in my description of milestone use above. I don't mention system maintenance or patching anywhere. A very common question is: Should I stop using init s, boot -s, and my other standard procedures to change runlevels and perform standard system maintenance? Emphatically, no! Your old favorite commands continue to work as they always have. There's no need to change procedures. There's no reason to retrain your fingers with a much longer-to-type command when init s works just fine. The init invocations will work just like they always have, where svcadm milestone won't. For example, running svcadm milestone svc:/milestone/single-user:default won't change the run-level of the system (as described by who -r). Running init s will.

While I'll stay out of the window manager wars (I run gnome, but that's primarily because as a Solaris developer, I consider it part of my job to shake out bugs in the thing Sun ships out-of-the-box), illustrious OpenSolaris community member Ben Rockwood took a screenshot of his favoured working environment. I'm just tickled to see some svcs output featured in an entry titled "The Way OpenSolaris Should Look!".