2010.next: a better netbook for meetings

I recently borrowed one of the 10" netbooks that came out this year, and tested Build 123 on it. Here's a shot of Toshiba's sleek black NB205 displaying a clean Device Driver Utility run.

The NB205 has a keyboard on par with the r500 I use regularly—large for a netbook—and has a superior trackpad. Suspend-resume is reliable; compiz runs well; the extra vertical inch means fewer application compromises over last year's netbook crop. (For instance, rather than rearranging panel to the sides, use of gnome-panel's auto-hide feature is enough to let applications run well when maximized.)

If I don't have to give it back, it'll be my new "netbook for meetings".

Netbook styling credit to Dave Powell.

[ T: OpenSolaris netbook Atom Toshiba ]

2008.11: A roundup

With the launch over, I took a few minutes to read through today's blog entries, and pulled out a list of the either technical or personally significant entries. If you're looking for the highlights, Calum's overview seems to be a great place to start. He's got links to the best summaries and screencasts.

• First, if you're running 2008.05 and considering upgrading—recommended strongly—please watch Markus's screencast. There are one or two tricky steps to pay attention to. (We got caught offguard by some misinteractions between GRUB, ZFS, and early startup; won't happen again.)
• Erwann's screencast is a great introduction to Time Slider, and I'm pretty confident no turkeys were harmed during filming.
• Peter has screen captures of top(1) on a couple of older laptops, including one with 447 MiB available to the OS. Dave and the Install team worked hard to keep 512 MiB systems booting on the release; here's evidence.
• Doug's pkgfactory/roboporter discussion and Nico's explanation of the repositories show an extremely promising way to make sure there's a wide variety of software available for OpenSolaris—make the computers do the grunt work. Doug and Nico, and Jim Walker, have been pushing hard on improving porting efficiencies.
• Knut explains using JavaDB as a managed service on 2008.11. Since Knut's post combines pkg(5) and smf(5) invocations, it easily wins a spot in any summary I might write. (And so it appears in this one.)
• Jerry describes the foundational work for integrating zones, Snap Upgrade, and pkg(5). The changes to filesystem layout to support that integration place limitations on zones across the 2008.05 to 2008.11 transition, so be sure to review Jerry's post if you've been experimenting with ipkg zones on the release.
• John reviews 2008.11 as a suitable paravirtualized guest on xVM, with some careful discussion of how to handle the graphical console.
• Shawn's the sole member of the image packaging team to get a technical entry up. Shawn talks about the depot refacing work he's done and planning on doing for 2009.next. (Shawn modestly omits the fact that he also implemented "pkg history" and operational intent in the client.)
• Finally, I'm giving Fintan the nod for best screenshot, even if I prefer to see this release on the metal.

There are a lot of entries under tagged with opensolaris; you'll find coverage of many other aspects of the release.

[ T: OpenSolaris 2008.11 ]

2008.next: Recovering a dropped data pool

As you update your pkg(5)-based system to build 100a, you might see a message on console during startup. It will look something like

WARNING: pool 'zeta1' could not be loaded as it was last accessed by
another system (host: rosseau hostid: 0x581d45e). See:
http://www.sun.com/msg/ZFS-8000-EY

and in /var/adm/messages, you'll find it again:

Oct 27 14:55:21 rosseau zfs: [ID 427000 kern.warning] WARNING: pool
'zeta1' could not be loaded as it was last accessed by another system
(host: rosseau hostid: 0x581d45e). See:
http://www.sun.com/msg/ZFS-8000-EY

zeta1 is a data pool on my system—the boot pool, rpool, upgraded normally—and recent changes in the hostid implementation mean that we have to import (or reimport) any non-boot pools on the system. That is,

$ pfexec zpool import -f zeta1

and our zeta1-based filesystems are again online.

[ T: OpenSolaris zfs pkg ]

2008.next: Easy install for docutils

It's been a busy summer, mostly trying to manage conflicting schedules at home while staying loyal to the six month release schedule for OpenSolaris at work. Schedules everywhere, it seems.

I've been probing deeper into the ecosystem around Python, and have been trying to determine whether restructured text might have some value in the workflow around OpenSolaris development: it can output to a variety of formats, including XML, LaTeX, and HTML, and doesn't have an onerous list of dependencies.

I've scars from building AsciiDoc 8.

docutils isn't available via image packaging yet, but it's easily retrievable using easy_install. That means the question is: how do I get easy_install? I'm running Build 98, so I'll restrict my search to packages on the development branch for that build:

$ pkg search -r easy_install | grep 0.98
basename  file  usr/bin/easy_install pkg:/SUNWpython-setuptools@0.5.11-0.98

Okay, easy_install is part of setuptools—you probably knew that already. Let's install the appropriate package using pkg(1):

$ pfexec pkg install SUNWpython-setuptools
DOWNLOAD                                    PKGS       FILES     XFER (MB)
Completed                                    1/1       81/81     0.30/0.30 

PHASE                                        ACTIONS
Install Phase                                  93/93 
PHASE                                          ITEMS
Reading Existing Index                           9/9
Indexing Packages                                1/1
$ which easy_install
/usr/bin/easy_install

Having setuptools in place, we can now attempt installs of any Python module available in the Python Package Index (pypi), via the easy_install command. (easy_install --help will display a usage message for easy_install.) Let's install the docutils package:

$ pfexec easy_install docutils 
Searching for docutils
Reading http://pypi.python.org/simple/docutils/
Reading http://docutils.sourceforge.net/
Best match: docutils 0.5
Downloading http://prdownloads.sourceforge.net/docutils/docutils-0.5.tar.gz?download
Processing docutils-0.5.tar.gz
Running docutils-0.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-57yD32/docutils-0.5/egg-dist-tmp-Ip1xgp
"optparse" module already present; ignoring extras/optparse.py.
"textwrap" module already present; ignoring extras/textwrap.py.
zip_safe flag not set; analyzing archive contents...
docutils.parsers.rst.directives.misc: module references __file__
docutils.writers.html4css1.__init__: module references __file__
docutils.writers.pep_html.__init__: module references __file__
docutils.writers.s5_html.__init__: module references __file__
docutils.writers.newlatex2e.__init__: module references __file__
Adding docutils 0.5 to easy-install.pth file
Installing rst2xml.py script to /usr/bin
Installing rst2pseudoxml.py script to /usr/bin
Installing rst2html.py script to /usr/bin
Installing rst2s5.py script to /usr/bin
Installing rst2latex.py script to /usr/bin
Installing rstpep2html.py script to /usr/bin
Installing rst2newlatex.py script to /usr/bin

Installed /usr/lib/python2.4/site-packages/docutils-0.5-py2.4.egg
Processing dependencies for docutils
Finished processing dependencies for docutils
$ which rst2html.py
/usr/bin/rst2html.py

And now I can start experimenting with restructured text.

[ T: OpenSolaris Python pkg docutils setuptools ]

Mail droppage

It looks like I missed making a configuration change last update, and opensolaris.org has been dropping my mail messages for the past week as a result. Sorry—if you're waiting for mail from me, you may need to ping me again. Otherwise, I'll try to reinsert myself in various threads...

CommunityOne: here and there

Sunday I spent at Moscone, teaching laptops and projectors to get along. Saturday, Nathaniel, Benjamin, and I dropped in for lunch at the Developer Summit. I managed to talk with a few people before the boys found large whacking sticks, and then it seemed best to drive to Pescadero for some beach time.

I'll be busy for the morning of CommunityOne. For Rich's keynote, I'll be running some of the less violent demos. Almost immediately after that, lead modernizer David Comay and I will be going into more detail in our session

S297399 Getting Started with OpenSolaris™; New Features & Building OpenSolaris™ Packages; 11:00 a.m., Moscone South/Esplanade 300.

I'm hoping to have some time for questions during the session but, if not, I'll be circulating during the afternoon and happy to talk to people about 2008.05, image packaging, or whatever. And, of course, there will be time to talk at the party after the day's sessions. In any case, I should be easy to spot: I have a new tie.

I'm told you can still register on-site—it's not too late.

[ T: OpenSolaris pkg CommunityOne ]

OpenSolaris: Bug dependencies and release management

Right now, if you're subscribed to any of the Installation and Packaging community group or project lists, you'll see a lot of commit notifications as the various teams attempt to fix various bugs noted since the second Developer Preview release. We've been using the trial Bugzilla instance—which is becoming the default defect tracker for opensolaris.org it appears—and trying out various features.

For tracking release completeness, we're using "blocker bugs" or "tracker bugs", which are synthetic bug entries that we mark various important bugs ("stoppers") as blocking. That means that we end up creating a little dependency graph that shows what unfixed bugs are stopping us from reaching some initial set of release criteria. We have two tracker bugs

• 571 Blocker bug for Indiana 3
• 1190 Packaging bugs to fix for May (tracker bug)

Bugzilla has two nice summaries for showing this information, in addition to the default bug status page. I'll use 571 as the example tracker bug, since we've made 1190 block it—which leads to a more nteresting graph. The tree view is a useful and succinct representation, where indentation shows dependencies. The graph view is a bit unwieldy for this bug, but might be useful if the tree view became too long.

A useful technique if you're trying to bring a release together.

[ T: OpenSolaris Bugzilla pkg ]

OpenSolaris: Finding work(ers)

A relatively new forum on opensolaris.org is opensolaris-jobs, a community-moderated list where OpenSolaris-related positions can be posted and which should give access to the people most passionate, most expert, and most interested in making OpenSolaris work their job. In the past few weeks, there's been two postings you might find interesting:

• Networking summer interns, Sun Microsystems, and
• Senior kernel developers, Intel.

Have a look, or—if you're hiring for OpenSolaris knowledge—get in touch with great candidates.

[ T: OpenSolaris jobs ]

Indiana: VESA if you need it

Over the past week, as we kept reassembling the distro constructor, image packaging, and slim install, we tested installs on a bunch of laptops. This manual operation let me rehearse how to get things working if the Preview LiveCD doesn't have a graphics driver that will work. So: here's the VESA workaround.

1. Boot text mode. One of the non-default lines in the GRUB menu should say "text"—pick that one.
2. Login as jack. User is "jack", password is "jack".
3. Generate a representative xorg.conf. Become root and have Xorg generate an initial configuration. The root password is "opensolaris". (Enjoy that `root`'s default shell is 'bash' for a bit.)

   $ su
   Password:
   # /usr/X11/bin/Xorg -configure
   ...
   

   This procedure should create /jack/xorg.conf.new.
4. Change the driver. As root, you'll continue by editing that xorg.conf.new file in vi(1). Search for the Device section, and modify the Driver line to

       Driver  "vesa"
   

5. Make it go. Stay root and hand-launch GNOME.

   # /usr/X11/bin/xinit /usr/bin/dbus-launch gnome-session -- \
           /usr/X11/bin/Xorg -config /jack/xorg.conf.new :0
   

6. Launch the installer. Bring up a terminal, and invoke the installer, by typing install-lan &.

If your install fails, please file a bug report at defect.opensolaris.org. If it succeeds, but graphics doesn't work, you can follow the same steps, but you'll have to edit the GRUB entry. (Roughly, type 'e', add "-s" to the end of the line with kernel/unix, and log in with the new root password you set during install. I haven't tested this portion—let me know if it fails.)

Last night, I had to do this for my trusty VAIO T370P, but tonight it's fine:

Please let us know how it goes.

[ T: OpenSolaris Solaris indiana pkg ]

pkg(5): Talking in the redwoods, talking on the beach, ...

Danek and I are at the 2007 OpenSolaris Developer Summit at UC Santa Cruz, talking about and, I hope, later demonstrating the image packaging prototype so far. I gave a brief overview of the project's goals and status, and mentioning some of the unmentionables we've encountered—some particularly undisciplined configuration files, some apparently important but encumbered drivers, and so on.

View the slides

[ T: OpenSolaris pkg UCSC opensolaris_summit_2007 ]

pkg(5): project opens, development continues

I asked my teammates to take a brief break from prototyping, and we jumped from a collection of systems inside Sun to the clean project hosting at opensolaris.org.

As the migration registers, you should be able to access

• the project web site at http://opensolaris.org/os/project/pkg/,
• the Mercurial repository via

      $ hg clone ssh://anon@hg.opensolaris.org/hg/pkg/gate pkg-gate
  

• the same source via OpenGrok (using the 'pkg' project as a filter) at http://src.opensolaris.org/, and
• have a discussion on pkg-discuss@opensolaris.org.

If you're interested, please come check it out.

[ T: OpenSolaris Solaris pkg ]

OpenSolaris: Defect tracking relationships

Just prior to the Board election, we ran a poll of the core contributors to get some sense of what one active subset felt were the five most pressing obstacles to open development. Dan just issued an initial Beta of a webrev-based approach, derived from his earlier experiment on http://cr.grommit.com, so that's a starting point for Priority 3. The Board is tackling, in full public view on ogb-discuss, Priority 4: there's OGB/2007/001, Project creation enhancements and OGB/2007/002, Community and Project Reorganization as two significant chunks of a stablized reorganization. Priority 2, the deployment of a "request to integrate" system, is somewhat gated on ON and sister consolidations' switch to Mercurial, being pursued in the scm-migration project—it's an aspect of workflow that isn't required by all of the hosted consolidations. Priority 5, the deployment of an opensolaris.org-hosted wiki, is in a requirements gathering phase over on website-discuss.

That leaves Priority 1, the deployment of a public bug tracking system. Bug tracking has loomed over the OpenSolaris effort for pretty much its entire implementation phase; we've known that aspects of the current bug tracking methodologies impact many parts of Sun's business, and that any solution will require the identification of which entanglements are strategic—meaning that there's a requirement for any new system—and which are accidental—meaning that there's only some transition cost, as the entangled system can be adjusted to consume information in some designed fashion. So, as part of getting a set of draft requirements together for discussion, I thought I would work through some of the issues facing defect tracking as we move to open development. Most of these points are about the developer–distribution (or upstream–downstream) interface that a defect tracking system (DTS) or defect tracking architecture represents.

The primary requirement that a distribution has on their DTS is some ability to maintain confidential data associated with each tracked defect. Let's call that database a proprietary annotation system (PAS)—the data within it capture the customer histories associated with various defects or collections of defects ("products"?) represented in the system. The DTS, meanwhile, is meant to be neutral across all participants, developers and distribution assemblers, and unconcerned with non-technical characteristics of the defect.

This contrast allows us to postulate a set of relationships among various active DTSs and PASs for an open development community.

The association of confidential information with an existing defect looks something like

Of course, in the OpenSolaris case, SMI's annotations become just one potential PAS; other distributions may also choose to annotate publicly known ("community tracked") defects:

One requirement that we've heard during preliminary discussions with the various teams is there must be some ability to search the entirety of the product-relevant portion of the DTS. One possibility is that each PAS operator builds a search corpus that combines the upstream DTS with the PAS content. A potentially more economic alternative would be to allow the the associations to be bidirectional (so that an indexer with authorizations allowing it to access one or more proprietary annotation systems can present a complete defect corpus). Making the existence of the annotations public does not seem like a significant leak of proprietary information, while the existence of annotations might be a useful measure of defect significance. (It is probably worth explicitly stating that having a complete defect corpus for searching does not imply that use of a single DTS is the only means of obtaining such a corpus.)

These associations are more complex objects than the current See Also links in typical monolithic DTSes, in that they carry one or more mappings of status and responsibility between the DTS and each PAS. Potentially, this capability could lead to more precise handling of release readiness, in that a query involving a group of PAS-tracked defects could indicate that one or more stoppers are blocking the release of a certain version of the tracked product. Of course, prior to open development, the Solaris organization has historically managed that fairly well for its products, so why is it worth discussing?

Well, for OpenSolaris today, the set of defects in our own DTS isn't the complete set of relevant defects for product release. In fact, as examples, Solaris tracks the defects, branches, and patches for GNOME, Mozilla Firefox, and Perl, among others. That is, Sun's current combined DTS/PAS, Bugster, is in the following relationships with an upstream source

when the product is affected by an upstream-sourced component, and

when a customer is affected by an upstream-sourced component. There's also an awkward local cost since upstream state is usually tracked manually (or possibly via custom scripting).

We can choose to apply this tracking need to our second defect/annotation figure above, in two ways: we can track the OpenSolaris consolidation defects as a peer of other upstream DTSs

or, more interestingly, we can allow community management of the relevance of upstream defects, like the following

(A distribution may not choose to annotate the public DTS record on an upstream bug, but may instead choose to annotate on the upstream DTS record directly. Ignoring the community signal when it has a mismatch with distribution priorities seems likely; using it as a guide when no conflicting principle exists seems like a safe course as well.)

I think that introducing these kinds of associations allows us to solve a large class of defect tracking problems that are currently impacting the OpenSolaris space. (Obviously there are issues, too--for instance, one could introduce association cycles (that clients must detect).) I would expect that the confidentiality issue impacts all of the distributions pulling from OpenSolaris consolidations (and probably, more generally, all groups looking at open development): I believe every distribution has customers of some kind. The importance of upstream relationships may presently be operating environment-specific, although I know other groups are also dependent on some number of OSS components.

[ T: OpenSolaris defects spam ]

Busy, stopping email

The past couple of months have been busy: I've been working on different deliverables for OpenSolaris, SMF, and SFW. I'll try to give summaries on each of these items as they complete in the coming weeks, but I thought I should mention the consequences of some anti-spam provisions I implemented on opensolaris.org in February. Since these restrictions can cause mail to OpenSolaris lists to be dropped, you may wish to read on.

At the time, we had essentially no anti-spam on opensolaris.org's mail pipeline, and we relying on Mailman (which handles our 120+ lists) to reject spam directly, or for list moderators to do so manually. We knew that some lists were promptly moderated, and others less so, but figured things were working, for the most part. This state of affairs then changed.

It was public news that there was a jump in spam volume the previous fall. On opensolaris.org, spam hit all of our addresses, including administrative aliases for list subscription and removal. Mailman requires time to process each request, and with the rate of inflow, finally built to a backlog of almost 35 000 unprocessed Mailman administrative queries in early February. For a potential OpenSolaris list subscriber, this translated to a multi-hour wait for subcription requests—no way to make it easy to get in on a conversation.

To stop the -subscribe and -unsubscribe addresses from getting pounded, I started activating the basic anti-spam features of Postfix (which we have run on opensolaris.org since the initial setup). Presently, we expect a host to have legitimate DNS entries and to not be on various blacklists, and we expect envelopes to be well-formed. The backlog started to clear as these controls were in place, and was completed with no dropped requests a day or two later. (We had one hiccough: I had initially written the configuration so that Postfix didn't trust the Mailman outbound mail, which meant that we did both anti-spam on the inbound submission and also on the dozens or hundreds of outbound messages to the list subscribers. Once we straightened that out, things have been much better.)

Since we imposed these requirements after initial subscriptions, it is possible that a valid subscriber to a Mailman list can no longer send mail acceptable for opensolaris.org. This problem is particularly likely to be related to a host lacking a valid MX record. If you've sent mail to opensolaris.org that hasn't appeared in the lists, please check that you have a valid MX record. You can use one of the web forms offering MX lookups, but on an OpenSolaris-based system it's easy to do this using nslookup(1M):

$ nslookup

> set querytype=mx
> rosseau.eng.sun.com
Server:         192.168.1.2
Address:        192.168.1.2#53

** server can't find rosseau.eng.sun.com: NXDOMAIN

> rosseau.sfbay.sun.com
Server:         192.168.1.2
Address:        192.168.1.2#53

Non-authoritative answer:
rosseau.sfbay.sun.com   mail exchanger = 10 btmx1.sun.com.
rosseau.sfbay.sun.com   mail exchanger = 10 btmx3.sun.com.
rosseau.sfbay.sun.com   mail exchanger = 10 btmx4.sun.com.
rosseau.sfbay.sun.com   mail exchanger = 20 mx3.sun.com.
rosseau.sfbay.sun.com   mail exchanger = 20 mx4.sun.com.

Authoritative answers can be found from:
sun.com nameserver = ns1.sun.com.
....
>

(In fact, rosseau lost its eng.sun.com MX record in the past couple of weeks, so I, too, got caught by the new policy.)

Both our Mailman instance and our many moderators are much happier now, processing many fewer, mostly legitimate list postings. If you've a valid MX record, aren't blacklisted, and are still having problems, website-discuss is the place to ask.

[ T: OpenSolaris Postfix Mailman spam ]

OpenSolaris: Restricted builds through the ages

Last month, Stephen Harpster announced that there would be a two-build sequence with restricted integration conditions. This sort of restriction isn't new for Solaris development, as I'll show below, but the means of arriving at the restriction and the manner in which we shared it shows we are still learning what an "open process" must do—and when we miss, to make sure we learn from those mistakes.

So Stephen's announcement noted that Build 54 and 55 would be "only bug fixes". That's not new; we can look at the build-by-build restrictions on ON integrations for Solaris 8

Solaris 9,

and Solaris 10,

Here, each box represents a cycle of the two-week build clock. Solid boxes indicate a strong "bugfix only" integration policy (like "stopper only" or integrations requiring the consolidation's technical lead's approval); outlined boxes indicate a weak "bugfix only" integration policy (meaning no specially approved projects). I've worked out these integration policy digests from the ON schedules of past releases; policies will vary for other consolidations, but should match up roughly with ON's history.

Solaris 8 and 9 show the classic "(Development Complete–)Beta–Beta Refresh–Ship" pattern that characterized Solaris releases. In Solaris 10, the Solaris Express program gets reflected in that, although there were identified Beta and Beta Refresh builds, the shift from early release to late release really takes place with the Beta Refresh build, which was the first in a series of restricted builds, until Ship.

Thus, for current Nevada, ON looks a lot like it did for the release that became Solaris 10:

Inferences you might make are at your own risk.

Of course, the freezes we took in the past were easily achieved as there was only one distribution derived from the ON sources. Now that we are a community with a collection of active distributions, the notion of periodic and public restricted builds may still be attractive, so that distributions could choose to issue their latest version from a known stabilized build. Certainly current distributions (or emerging ones) might consider using the stabilized build 55—however we got here—as an opportune version to base their next release upon.

Of course, there is a tradeoff between letting each distribution stabilize and a consolidation-wide restriction. Perhaps a topic for a future post?

We, as a community, will need to have a position on restricted builds. Please think about it, and then feel free to comment here, and mail me privately. Public discussions, I suspect, are more valuable: opensolaris-code would be a good forum for the technical tradeoffs, possible frequency of stabilized builds, and so on, while cab-discuss would be the best place for inter-distribution and coordination issues.

[ T: OpenSolaris Solaris distribution consolidation ]

opensolaris.org: Prevent accidental robotomies

We've been working through bits and pieces of opensolaris.org, looking for the troubling portions where regular human touches are currently required.

These are troubling because (a) someone forgets to touch when someone else is on vacation, sick, or whatever, and then (b) someone depending on that resource being touched gets annoyed/loses time/sends us Kafkaeque email about transforming into an automaton.

Recently, Steve pulled a little Python script together and now we can smoothly offer direct DVD downloads of Solaris Express Community Release across each incremental release. In honour of his script, a new download button:

http://opensolaris.org/sxcr_dvd should work, without problem, for the duration of Nevada. But pipe up on website-discuss if it fails.

[ T: OpenSolaris Solaris DVD Downloads ]