Mail droppage

It looks like I missed making a configuration change last update, and opensolaris.org has been dropping my mail messages for the past week as a result. Sorry—if you're waiting for mail from me, you may need to ping me again. Otherwise, I'll try to reinsert myself in various threads...

CommunityOne: here and there

Sunday I spent at Moscone, teaching laptops and projectors to get along. Saturday, Nathaniel, Benjamin, and I dropped in for lunch at the Developer Summit. I managed to talk with a few people before the boys found large whacking sticks, and then it seemed best to drive to Pescadero for some beach time.

I'll be busy for the morning of CommunityOne. For Rich's keynote, I'll be running some of the less violent demos. Almost immediately after that, lead modernizer David Comay and I will be going into more detail in our session

S297399 Getting Started with OpenSolaris™; New Features & Building OpenSolaris™ Packages; 11:00 a.m., Moscone South/Esplanade 300.

I'm hoping to have some time for questions during the session but, if not, I'll be circulating during the afternoon and happy to talk to people about 2008.05, image packaging, or whatever. And, of course, there will be time to talk at the party after the day's sessions. In any case, I should be easy to spot: I have a new tie.

I'm told you can still register on-site—it's not too late.

[ T: OpenSolaris pkg CommunityOne ]

OpenSolaris: Bug dependencies and release management

Right now, if you're subscribed to any of the Installation and Packaging community group or project lists, you'll see a lot of commit notifications as the various teams attempt to fix various bugs noted since the second Developer Preview release. We've been using the trial Bugzilla instance—which is becoming the default defect tracker for opensolaris.org it appears—and trying out various features.

For tracking release completeness, we're using "blocker bugs" or "tracker bugs", which are synthetic bug entries that we mark various important bugs ("stoppers") as blocking. That means that we end up creating a little dependency graph that shows what unfixed bugs are stopping us from reaching some initial set of release criteria. We have two tracker bugs

• 571 Blocker bug for Indiana 3
• 1190 Packaging bugs to fix for May (tracker bug)

Bugzilla has two nice summaries for showing this information, in addition to the default bug status page. I'll use 571 as the example tracker bug, since we've made 1190 block it—which leads to a more nteresting graph. The tree view is a useful and succinct representation, where indentation shows dependencies. The graph view is a bit unwieldy for this bug, but might be useful if the tree view became too long.

A useful technique if you're trying to bring a release together.

[ T: OpenSolaris Bugzilla pkg ]

OpenSolaris: Finding work(ers)

A relatively new forum on opensolaris.org is opensolaris-jobs, a community-moderated list where OpenSolaris-related positions can be posted and which should give access to the people most passionate, most expert, and most interested in making OpenSolaris work their job. In the past few weeks, there's been two postings you might find interesting:

• Networking summer interns, Sun Microsystems, and
• Senior kernel developers, Intel.

Have a look, or—if you're hiring for OpenSolaris knowledge—get in touch with great candidates.

[ T: OpenSolaris jobs ]

Indiana: VESA if you need it

Over the past week, as we kept reassembling the distro constructor, image packaging, and slim install, we tested installs on a bunch of laptops. This manual operation let me rehearse how to get things working if the Preview LiveCD doesn't have a graphics driver that will work. So: here's the VESA workaround.

1. Boot text mode. One of the non-default lines in the GRUB menu should say "text"—pick that one.
2. Login as jack. User is "jack", password is "jack".
3. Generate a representative xorg.conf. Become root and have Xorg generate an initial configuration. The root password is "opensolaris". (Enjoy that `root`'s default shell is 'bash' for a bit.)

   $ su
   Password:
   # /usr/X11/bin/Xorg -configure
   ...
   

   This procedure should create /jack/xorg.conf.new.
4. Change the driver. As root, you'll continue by editing that xorg.conf.new file in vi(1). Search for the Device section, and modify the Driver line to

       Driver  "vesa"
   

5. Make it go. Stay root and hand-launch GNOME.

   # /usr/X11/bin/xinit /usr/bin/dbus-launch gnome-session -- \
           /usr/X11/bin/Xorg -config /jack/xorg.conf.new :0
   

6. Launch the installer. Bring up a terminal, and invoke the installer, by typing install-lan &.

If your install fails, please file a bug report at defect.opensolaris.org. If it succeeds, but graphics doesn't work, you can follow the same steps, but you'll have to edit the GRUB entry. (Roughly, type 'e', add "-s" to the end of the line with kernel/unix, and log in with the new root password you set during install. I haven't tested this portion—let me know if it fails.)

Last night, I had to do this for my trusty VAIO T370P, but tonight it's fine:

Please let us know how it goes.

[ T: OpenSolaris Solaris indiana pkg ]

pkg(5): Talking in the redwoods, talking on the beach, ...

Danek and I are at the 2007 OpenSolaris Developer Summit at UC Santa Cruz, talking about and, I hope, later demonstrating the image packaging prototype so far. I gave a brief overview of the project's goals and status, and mentioning some of the unmentionables we've encountered—some particularly undisciplined configuration files, some apparently important but encumbered drivers, and so on.

View the slides

[ T: OpenSolaris pkg UCSC opensolaris_summit_2007 ]

pkg(5): project opens, development continues

I asked my teammates to take a brief break from prototyping, and we jumped from a collection of systems inside Sun to the clean project hosting at opensolaris.org.

As the migration registers, you should be able to access

• the project web site at http://opensolaris.org/os/project/pkg/,
• the Mercurial repository via

      $ hg clone ssh://anon@hg.opensolaris.org/hg/pkg/gate pkg-gate
  

• the same source via OpenGrok (using the 'pkg' project as a filter) at http://src.opensolaris.org/, and
• have a discussion on pkg-discuss@opensolaris.org.

If you're interested, please come check it out.

[ T: OpenSolaris Solaris pkg ]

OpenSolaris: Defect tracking relationships

Just prior to the Board election, we ran a poll of the core contributors to get some sense of what one active subset felt were the five most pressing obstacles to open development. Dan just issued an initial Beta of a webrev-based approach, derived from his earlier experiment on http://cr.grommit.com, so that's a starting point for Priority 3. The Board is tackling, in full public view on ogb-discuss, Priority 4: there's OGB/2007/001, Project creation enhancements and OGB/2007/002, Community and Project Reorganization as two significant chunks of a stablized reorganization. Priority 2, the deployment of a "request to integrate" system, is somewhat gated on ON and sister consolidations' switch to Mercurial, being pursued in the scm-migration project—it's an aspect of workflow that isn't required by all of the hosted consolidations. Priority 5, the deployment of an opensolaris.org-hosted wiki, is in a requirements gathering phase over on website-discuss.

That leaves Priority 1, the deployment of a public bug tracking system. Bug tracking has loomed over the OpenSolaris effort for pretty much its entire implementation phase; we've known that aspects of the current bug tracking methodologies impact many parts of Sun's business, and that any solution will require the identification of which entanglements are strategic—meaning that there's a requirement for any new system—and which are accidental—meaning that there's only some transition cost, as the entangled system can be adjusted to consume information in some designed fashion. So, as part of getting a set of draft requirements together for discussion, I thought I would work through some of the issues facing defect tracking as we move to open development. Most of these points are about the developer–distribution (or upstream–downstream) interface that a defect tracking system (DTS) or defect tracking architecture represents.

The primary requirement that a distribution has on their DTS is some ability to maintain confidential data associated with each tracked defect. Let's call that database a proprietary annotation system (PAS)—the data within it capture the customer histories associated with various defects or collections of defects ("products"?) represented in the system. The DTS, meanwhile, is meant to be neutral across all participants, developers and distribution assemblers, and unconcerned with non-technical characteristics of the defect.

This contrast allows us to postulate a set of relationships among various active DTSs and PASs for an open development community.

The association of confidential information with an existing defect looks something like

Of course, in the OpenSolaris case, SMI's annotations become just one potential PAS; other distributions may also choose to annotate publicly known ("community tracked") defects:

One requirement that we've heard during preliminary discussions with the various teams is there must be some ability to search the entirety of the product-relevant portion of the DTS. One possibility is that each PAS operator builds a search corpus that combines the upstream DTS with the PAS content. A potentially more economic alternative would be to allow the the associations to be bidirectional (so that an indexer with authorizations allowing it to access one or more proprietary annotation systems can present a complete defect corpus). Making the existence of the annotations public does not seem like a significant leak of proprietary information, while the existence of annotations might be a useful measure of defect significance. (It is probably worth explicitly stating that having a complete defect corpus for searching does not imply that use of a single DTS is the only means of obtaining such a corpus.)

These associations are more complex objects than the current See Also links in typical monolithic DTSes, in that they carry one or more mappings of status and responsibility between the DTS and each PAS. Potentially, this capability could lead to more precise handling of release readiness, in that a query involving a group of PAS-tracked defects could indicate that one or more stoppers are blocking the release of a certain version of the tracked product. Of course, prior to open development, the Solaris organization has historically managed that fairly well for its products, so why is it worth discussing?

Well, for OpenSolaris today, the set of defects in our own DTS isn't the complete set of relevant defects for product release. In fact, as examples, Solaris tracks the defects, branches, and patches for GNOME, Mozilla Firefox, and Perl, among others. That is, Sun's current combined DTS/PAS, Bugster, is in the following relationships with an upstream source

when the product is affected by an upstream-sourced component, and

when a customer is affected by an upstream-sourced component. There's also an awkward local cost since upstream state is usually tracked manually (or possibly via custom scripting).

We can choose to apply this tracking need to our second defect/annotation figure above, in two ways: we can track the OpenSolaris consolidation defects as a peer of other upstream DTSs

or, more interestingly, we can allow community management of the relevance of upstream defects, like the following

(A distribution may not choose to annotate the public DTS record on an upstream bug, but may instead choose to annotate on the upstream DTS record directly. Ignoring the community signal when it has a mismatch with distribution priorities seems likely; using it as a guide when no conflicting principle exists seems like a safe course as well.)

I think that introducing these kinds of associations allows us to solve a large class of defect tracking problems that are currently impacting the OpenSolaris space. (Obviously there are issues, too--for instance, one could introduce association cycles (that clients must detect).) I would expect that the confidentiality issue impacts all of the distributions pulling from OpenSolaris consolidations (and probably, more generally, all groups looking at open development): I believe every distribution has customers of some kind. The importance of upstream relationships may presently be operating environment-specific, although I know other groups are also dependent on some number of OSS components.

[ T: OpenSolaris defects spam ]

Busy, stopping email

The past couple of months have been busy: I've been working on different deliverables for OpenSolaris, SMF, and SFW. I'll try to give summaries on each of these items as they complete in the coming weeks, but I thought I should mention the consequences of some anti-spam provisions I implemented on opensolaris.org in February. Since these restrictions can cause mail to OpenSolaris lists to be dropped, you may wish to read on.

At the time, we had essentially no anti-spam on opensolaris.org's mail pipeline, and we relying on Mailman (which handles our 120+ lists) to reject spam directly, or for list moderators to do so manually. We knew that some lists were promptly moderated, and others less so, but figured things were working, for the most part. This state of affairs then changed.

It was public news that there was a jump in spam volume the previous fall. On opensolaris.org, spam hit all of our addresses, including administrative aliases for list subscription and removal. Mailman requires time to process each request, and with the rate of inflow, finally built to a backlog of almost 35 000 unprocessed Mailman administrative queries in early February. For a potential OpenSolaris list subscriber, this translated to a multi-hour wait for subcription requests—no way to make it easy to get in on a conversation.

To stop the -subscribe and -unsubscribe addresses from getting pounded, I started activating the basic anti-spam features of Postfix (which we have run on opensolaris.org since the initial setup). Presently, we expect a host to have legitimate DNS entries and to not be on various blacklists, and we expect envelopes to be well-formed. The backlog started to clear as these controls were in place, and was completed with no dropped requests a day or two later. (We had one hiccough: I had initially written the configuration so that Postfix didn't trust the Mailman outbound mail, which meant that we did both anti-spam on the inbound submission and also on the dozens or hundreds of outbound messages to the list subscribers. Once we straightened that out, things have been much better.)

Since we imposed these requirements after initial subscriptions, it is possible that a valid subscriber to a Mailman list can no longer send mail acceptable for opensolaris.org. This problem is particularly likely to be related to a host lacking a valid MX record. If you've sent mail to opensolaris.org that hasn't appeared in the lists, please check that you have a valid MX record. You can use one of the web forms offering MX lookups, but on an OpenSolaris-based system it's easy to do this using nslookup(1M):

$ nslookup

> set querytype=mx
> rosseau.eng.sun.com
Server:         192.168.1.2
Address:        192.168.1.2#53

** server can't find rosseau.eng.sun.com: NXDOMAIN

> rosseau.sfbay.sun.com
Server:         192.168.1.2
Address:        192.168.1.2#53

Non-authoritative answer:
rosseau.sfbay.sun.com   mail exchanger = 10 btmx1.sun.com.
rosseau.sfbay.sun.com   mail exchanger = 10 btmx3.sun.com.
rosseau.sfbay.sun.com   mail exchanger = 10 btmx4.sun.com.
rosseau.sfbay.sun.com   mail exchanger = 20 mx3.sun.com.
rosseau.sfbay.sun.com   mail exchanger = 20 mx4.sun.com.

Authoritative answers can be found from:
sun.com nameserver = ns1.sun.com.
....
>

(In fact, rosseau lost its eng.sun.com MX record in the past couple of weeks, so I, too, got caught by the new policy.)

Both our Mailman instance and our many moderators are much happier now, processing many fewer, mostly legitimate list postings. If you've a valid MX record, aren't blacklisted, and are still having problems, website-discuss is the place to ask.

[ T: OpenSolaris Postfix Mailman spam ]

OpenSolaris: Restricted builds through the ages

Last month, Stephen Harpster announced that there would be a two-build sequence with restricted integration conditions. This sort of restriction isn't new for Solaris development, as I'll show below, but the means of arriving at the restriction and the manner in which we shared it shows we are still learning what an "open process" must do—and when we miss, to make sure we learn from those mistakes.

So Stephen's announcement noted that Build 54 and 55 would be "only bug fixes". That's not new; we can look at the build-by-build restrictions on ON integrations for Solaris 8

Solaris 9,

and Solaris 10,

Here, each box represents a cycle of the two-week build clock. Solid boxes indicate a strong "bugfix only" integration policy (like "stopper only" or integrations requiring the consolidation's technical lead's approval); outlined boxes indicate a weak "bugfix only" integration policy (meaning no specially approved projects). I've worked out these integration policy digests from the ON schedules of past releases; policies will vary for other consolidations, but should match up roughly with ON's history.

Solaris 8 and 9 show the classic "(Development Complete–)Beta–Beta Refresh–Ship" pattern that characterized Solaris releases. In Solaris 10, the Solaris Express program gets reflected in that, although there were identified Beta and Beta Refresh builds, the shift from early release to late release really takes place with the Beta Refresh build, which was the first in a series of restricted builds, until Ship.

Thus, for current Nevada, ON looks a lot like it did for the release that became Solaris 10:

Inferences you might make are at your own risk.

Of course, the freezes we took in the past were easily achieved as there was only one distribution derived from the ON sources. Now that we are a community with a collection of active distributions, the notion of periodic and public restricted builds may still be attractive, so that distributions could choose to issue their latest version from a known stabilized build. Certainly current distributions (or emerging ones) might consider using the stabilized build 55—however we got here—as an opportune version to base their next release upon.

Of course, there is a tradeoff between letting each distribution stabilize and a consolidation-wide restriction. Perhaps a topic for a future post?

We, as a community, will need to have a position on restricted builds. Please think about it, and then feel free to comment here, and mail me privately. Public discussions, I suspect, are more valuable: opensolaris-code would be a good forum for the technical tradeoffs, possible frequency of stabilized builds, and so on, while cab-discuss would be the best place for inter-distribution and coordination issues.

[ T: OpenSolaris Solaris distribution consolidation ]

opensolaris.org: Prevent accidental robotomies

We've been working through bits and pieces of opensolaris.org, looking for the troubling portions where regular human touches are currently required.

These are troubling because (a) someone forgets to touch when someone else is on vacation, sick, or whatever, and then (b) someone depending on that resource being touched gets annoyed/loses time/sends us Kafkaeque email about transforming into an automaton.

Recently, Steve pulled a little Python script together and now we can smoothly offer direct DVD downloads of Solaris Express Community Release across each incremental release. In honour of his script, a new download button:

http://opensolaris.org/sxcr_dvd should work, without problem, for the duration of Nevada. But pipe up on website-discuss if it fails.

[ T: OpenSolaris Solaris DVD Downloads ]

Tip: Mercurial atop ssh

On opensolaris.org, we host Mercurial and Subversion via their ability to tunnel through an SSH connection. A slight gotcha with Mercurial is that it turns off compression when tunnelled through SSH, as described in hg(1):

$ man hg
....
    - Mercurial doesn't use its own compression via SSH; the right thing
      to do is to configure it in your ~/.ssh/ssh_config, e.g.:
        Host *.mylocalnetwork.example.com
          Compression off
        Host *
          Compression on
    
      Alternatively specify "ssh -C" as your ssh command in your hgrc or
      with the --ssh command line option.
....

For pulls of a large repository, such as ON, compression changes an approximately 51 minute operation to a 17 minute operation. So you'll want something like

$ cat ~/.ssh/config
....
Host hg.opensolaris.org
    Compression yes
    ...
....

[ T: OpenSolaris Solaris Mercurial hg ]

OpenSolaris: Documentation tutorial buffet

Each of the community groups that make up OpenSolaris is experimenting with different styles of introductory materials (or procedures), so that potential contributors don't get blocked on the details and can focus on their real intention—contributions. The Documentation community group has been energetically pursuing a complete offering: they have a large document tree, they're using a "scratchpad" on the Genunix wiki, and they have at least one mailing list. They even have a Flash animated tutorial, illustrating how to use a DocBook XML toolchain. Slick.

[ T: OpenSolaris Solaris documentation DocBook ]

OpenSolaris Day, Buenos Aires

I've been back from Buenos Aires for a week now, and just wanted to comment on my trip, and show the few pictures I managed to take. Like its sister events, Buenos Aires TechDays was a densely packed three day conference, with multiple tracks available on OpenSolaris, Java, NetBeans, SunStudio, Glassfish, and other technologies and platforms. I updated the "Building and Deploying OpenSolaris" talk, to cover the convenient Starter Kit, which has the tools and source (and OS images) and to mention the ONNV Mercurial repository on opensolaris.org.

The talk was simultaneously translated into Spanish, using some slick IR emitter arrays and receivers. Russ, I think, was much more effective, as he gave his presentations in Spanish.

The event was held at the Hilton Buenos Aires, which is located in the recently redeveloped Puerto Madero area. When Teresa and I walked around on the first day, we saw activity in the harbour, and paused to watch (along with a few hundred other pedestrians), as this tugboat

helped this coast guard ship

turn out of the relatively narrow channel.

Back at TechDays, we enjoyed the chance to have further technical discussions, with many resolutions

to study new problems, reengage with old ones, or just return to Argentina.

The Inquirer's Fernando Cassia has more on the NetBeans portion of the event.

In Argentina, but missed TechDays? There's an OpenSolaris user group—check it out.

[ T: Solaris OpenSolaris TechDays Buenos_Aires ]

OpenSolaris: Gary's in Brussels

Gary Pennington is attending EuroOSCON, being held this week in Brussels. He's reporting on the various sessions. Worth a read, particularly if you've not been to an open source conference in a while...

[ T: OpenSolaris Solaris EuroOSCON Brussels ]