2008.next: Easy install for docutils

It's been a busy summer, mostly trying to manage conflicting schedules at home while staying loyal to the six month release schedule for OpenSolaris at work. Schedules everywhere, it seems.

I've been probing deeper into the ecosystem around Python, and have been trying to determine whether restructured text might have some value in the workflow around OpenSolaris development: it can output to a variety of formats, including XML, LaTeX, and HTML, and doesn't have an onerous list of dependencies.

I've scars from building AsciiDoc 8.

docutils isn't available via image packaging yet, but it's easily retrievable using easy_install. That means the question is: how do I get easy_install? I'm running Build 98, so I'll restrict my search to packages on the development branch for that build:

$ pkg search -r easy_install | grep 0.98
basename  file  usr/bin/easy_install pkg:/SUNWpython-setuptools@0.5.11-0.98

Okay, easy_install is part of setuptools—you probably knew that already. Let's install the appropriate package using pkg(1):

$ pfexec pkg install SUNWpython-setuptools
DOWNLOAD                                    PKGS       FILES     XFER (MB)
Completed                                    1/1       81/81     0.30/0.30 

PHASE                                        ACTIONS
Install Phase                                  93/93 
PHASE                                          ITEMS
Reading Existing Index                           9/9
Indexing Packages                                1/1
$ which easy_install
/usr/bin/easy_install

Having setuptools in place, we can now attempt installs of any Python module available in the Python Package Index (pypi), via the easy_install command. (easy_install --help will display a usage message for easy_install.) Let's install the docutils package:

$ pfexec easy_install docutils 
Searching for docutils
Reading http://pypi.python.org/simple/docutils/
Reading http://docutils.sourceforge.net/
Best match: docutils 0.5
Downloading http://prdownloads.sourceforge.net/docutils/docutils-0.5.tar.gz?download
Processing docutils-0.5.tar.gz
Running docutils-0.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-57yD32/docutils-0.5/egg-dist-tmp-Ip1xgp
"optparse" module already present; ignoring extras/optparse.py.
"textwrap" module already present; ignoring extras/textwrap.py.
zip_safe flag not set; analyzing archive contents...
docutils.parsers.rst.directives.misc: module references __file__
docutils.writers.html4css1.__init__: module references __file__
docutils.writers.pep_html.__init__: module references __file__
docutils.writers.s5_html.__init__: module references __file__
docutils.writers.newlatex2e.__init__: module references __file__
Adding docutils 0.5 to easy-install.pth file
Installing rst2xml.py script to /usr/bin
Installing rst2pseudoxml.py script to /usr/bin
Installing rst2html.py script to /usr/bin
Installing rst2s5.py script to /usr/bin
Installing rst2latex.py script to /usr/bin
Installing rstpep2html.py script to /usr/bin
Installing rst2newlatex.py script to /usr/bin

Installed /usr/lib/python2.4/site-packages/docutils-0.5-py2.4.egg
Processing dependencies for docutils
Finished processing dependencies for docutils
$ which rst2html.py
/usr/bin/rst2html.py

And now I can start experimenting with restructured text.

[ T: OpenSolaris Python pkg docutils setuptools ]

2008.05: Who's got it

Well, I don't know exactly who's running a copy of 2008.05, but I do know what IP addresses have interacted with pkg.opensolaris.org from the server logs. I used MaxMind's GeoIP Lite to map these back to countries, and Google's chart API to make a little map:

If 2008.05 is your first experience with the technology developed at opensolaris.org, welcome. (You're not alone—the map suggests that, in many places, there might even be a fellow user nearby.) Don't have it? Try a convenient mirror site.

Maybe we need a TechDays in Greenland next year?

[ T: OpenSolaris pkg nationalism ]

Mail droppage

It looks like I missed making a configuration change last update, and opensolaris.org has been dropping my mail messages for the past week as a result. Sorry—if you're waiting for mail from me, you may need to ping me again. Otherwise, I'll try to reinsert myself in various threads...

CommunityOne: here and there

Sunday I spent at Moscone, teaching laptops and projectors to get along. Saturday, Nathaniel, Benjamin, and I dropped in for lunch at the Developer Summit. I managed to talk with a few people before the boys found large whacking sticks, and then it seemed best to drive to Pescadero for some beach time.

I'll be busy for the morning of CommunityOne. For Rich's keynote, I'll be running some of the less violent demos. Almost immediately after that, lead modernizer David Comay and I will be going into more detail in our session

S297399 Getting Started with OpenSolaris™; New Features & Building OpenSolaris™ Packages; 11:00 a.m., Moscone South/Esplanade 300.

I'm hoping to have some time for questions during the session but, if not, I'll be circulating during the afternoon and happy to talk to people about 2008.05, image packaging, or whatever. And, of course, there will be time to talk at the party after the day's sessions. In any case, I should be easy to spot: I have a new tie.

I'm told you can still register on-site—it's not too late.

[ T: OpenSolaris pkg CommunityOne ]

2008.05: More ways to get it

As I did for the preview releases, I'll collect links to mirror sites here. These will also get links on the various download pages out there.

Bart and I just finished updating the package repository with the new packages we'd received the past few weeks, and that means 2008.05 is out the door. (Thanks to everybody who tried the release candidate, filed a bug, shuffled a package, wrote or proofread a document, or just spent energy anticipating the bits.) You can get the ISO image, suitable for burning to a 700 MiB CD or immediate use in virtual environment, directly from the following locations:

• via HTTP from dlc.sun.com, and
• via HTTP from genunix.org,
• via HTTP from ftp.df.lth.se,
• via FTP from ftp.df.lth.se,
• via HTTP from mirrors.xservers.ro,
• via HTTP from gulus.usherbrooke.ca,
• via FTP from ftp.nluug.nl,
• via HTTP from ftp.nluug.nl,
• via rsync from ftp.nluug.nl,
• via FTP from ftp.surfnet.nl,
• via HTTP from ftp.surfnet.nl,
• via rsync from ftp.surfnet.nl, or
• via BitTorrent.

Reading the logs, and talking with some of our mirror sites, we know we all served out a lot of downloads for the previews; if you're interested in being a mirror, please let me know. (2008.05 remains completely redistributable.) We're using a bigger download complex this time, but every mirror helps.

If you already downloaded and installed Preview 2, it's more complete, easier, and probably faster to update directly using image packaging: see the update guidelines. These instructions involve a small script to safely update a clone of your installed system, and then switch to that on a system reboot. (If you were running Preview 1, you should update that to Preview 2, and then go on to 2008.05.)

Update 1: My thanks to Tobias Lundquist, who's once again mirrored via FTP and HTTP (Internet 2) in Sweden.

Update 2: My thanks to Luca, who's put up an HTTP mirror in Romania.

Update 3: Simon Valiquette has put up an HTTP mirror (Internet 2) at the Université de Sherbrooke in Québec, Canada. Merci, Simon.

Update 4: Bart Muijzer has arranged for mirrors by the Netherlands Unix Users Group (NLUUG) and surfnet.nl. Thanks, Bart.

These links are for the gzipped CD image, which contains the 12 "primary languages". It installs quite a bit faster, particularly on systems with slower CPUs. There is also an LZMA-compressed image, which has localization support for 42 languages, including those primary ones. It's available from dlc.sun.com, genunix.org, ftp.df.lth.se, mirrors.xservers.ro, and as a torrent. (Consult the language lists for specifics.)

[ T: OpenSolaris indiana ]

Live chat on #os200805

We're talking about OpenSolaris 2008.05 over on IRC, using the #os200805, from now until 2 p.m. Pacific Time. If you haven't got an IRC client, you can use freenode's Java applet, which worked great (until Danek kickbanned me). Enter a nickname, and then ask some questions.

Apparently the load from 60 000 users blew out the chat system, so we moved to IRC.

OpenSolaris: Bug dependencies and release management

Right now, if you're subscribed to any of the Installation and Packaging community group or project lists, you'll see a lot of commit notifications as the various teams attempt to fix various bugs noted since the second Developer Preview release. We've been using the trial Bugzilla instance—which is becoming the default defect tracker for opensolaris.org it appears—and trying out various features.

For tracking release completeness, we're using "blocker bugs" or "tracker bugs", which are synthetic bug entries that we mark various important bugs ("stoppers") as blocking. That means that we end up creating a little dependency graph that shows what unfixed bugs are stopping us from reaching some initial set of release criteria. We have two tracker bugs

• 571 Blocker bug for Indiana 3
• 1190 Packaging bugs to fix for May (tracker bug)

Bugzilla has two nice summaries for showing this information, in addition to the default bug status page. I'll use 571 as the example tracker bug, since we've made 1190 block it—which leads to a more nteresting graph. The tree view is a useful and succinct representation, where indentation shows dependencies. The graph view is a bit unwieldy for this bug, but might be useful if the tree view became too long.

A useful technique if you're trying to bring a release together.

[ T: OpenSolaris Bugzilla pkg ]

pkg(5): Reverse proxying your depot with Apache HTTPD

As part of the changes to get Developer Preview 2 ready, we decided to rejigger the HTTP handling on pkg.opensolaris.org so that we could have more options as more people attempt to use the early versions of image packaging. Previously, we ran pkg.depotd directly on port 80, in its read-only mode; now we use Apache HTTPD to listen on port 80, and use mod_proxy to proxy those incoming requests to a pkg.depotd instance listening on a separate port. With a couple of different approaches

Proxying and rewriting is one of those endlessly fun activities that somehow actually ends up being productive. Last time, proxying fun led Steve and I to fiddling around such that we ended up with proxy and rewrite patterns to enable the country portals for opensolaris.org.

If you want to share the top-level component of your URL space, you'll need to watch pkg(5) developments, as you have to map the list of operations one-by-one—and I know there are some new operations forthcoming. That would involve adding something like the following to a VirtualHost directive in your Apache configuration.

ProxyRequests On

Redirect /index.html http://pkg.opensolaris.org/status

ProxyPass /abandon http://pkg.opensolaris.org:10000/abandon
ProxyPass /add http://pkg.opensolaris.org:10000/add
ProxyPass /catalog http://pkg.opensolaris.org:10000/catalog
ProxyPass /close http://pkg.opensolaris.org:10000/close
ProxyPass /feed http://pkg.opensolaris.org:10000/feed
ProxyPass /file http://pkg.opensolaris.org:10000/file
ProxyPass /filelist http://pkg.opensolaris.org:10000/filelist
ProxyPass /manifest http://pkg.opensolaris.org:10000/manifest
ProxyPass /open http://pkg.opensolaris.org:10000/open
ProxyPass /search http://pkg.opensolaris.org:10000/search
ProxyPass /versions http://pkg.opensolaris.org:10000/versions

ProxyPass /css http://pkg.opensolaris.org:10000/css
ProxyPass /logo http://pkg.opensolaris.org:10000/logo
ProxyPass /icon http://pkg.opensolaris.org:10000/icon

ProxyPass /status http://pkg.opensolaris.org:10000/

Configuring your server in this fashion allows you to mix an image packaging server in with your other site content. You can easily deliver static content alongside your depot, for example.

If you don't mind pushing your package repository down one level in your URL space, then the above simplifies to

ProxyRequests On

ProxyPass /pkg/ http://pkg.opensolaris.org:10000/

(which should be a hint on how to create a repository farm under a single URL). To use the latter, you would use pkg(1)'s image-create subcommand

$ pkg image-create -F -a mypkgs.com=http://www.myserver.com/pkg /path/to/image

to connect your image to your reverse-proxied packaging depot.

In the two examples above, you should of course replace machine names like pkg.opensolaris.org and port numbers like 10000 with values appropriate to your own installation.

Happy proxied package serving!

Feel free to share your alternative configurations or approaches with other HTTP servers here, or on pkg-discuss@opensolaris.org.

[ T: OpenSolaris Solaris pkg httpd ]

Indiana: More ways to get to Preview 2

As I did for Preview 1, I'll collect links to mirror sites here, as well as on the project page.

So we just released the second Indiana preview ISO, which you can get from the following locations:

• via HTTP from dlc.sun.com, and
• via HTTP from genunix.org,
• via FTP from ftp.df.lth.se, or
• via HTTP from osol.acm.jhu.edu.
• via BitTorrent.

We had a lot of downloads for Preview 1; if you're interested in being a mirror, please let me know. (Indiana remains completely redistributable.) We're using a bigger download complex this time, but every mirror helps.

If you already downloaded and installed Preview 1, it's more complete, easier, and probably faster to update directly using image packaging: see the update guidelines on opensolaris.org. These instructions use ZFS and pkg(5) to safely update a clone of your installed system, and then switch to that on a system reboot.

Update 1: My thanks to Tobias Lundquist, who's once again mirrored via FTP (Internet 2) in Sweden.

Update 2: My thanks to trisk, who's once again put up an HTTP mirror (Internet 2) on the East Coast of the USA.

[ T: OpenSolaris Solaris indiana ]

OpenSolaris: Finding work(ers)

A relatively new forum on opensolaris.org is opensolaris-jobs, a community-moderated list where OpenSolaris-related positions can be posted and which should give access to the people most passionate, most expert, and most interested in making OpenSolaris work their job. In the past few weeks, there's been two postings you might find interesting:

• Networking summer interns, Sun Microsystems, and
• Senior kernel developers, Intel.

Have a look, or—if you're hiring for OpenSolaris knowledge—get in touch with great candidates.

[ T: OpenSolaris jobs ]

pkg(5): Fueling the next steps

Most Fridays, I spend the two hours before lunch assisting at our co-operative daycare. The hour and a half before are pretty good thinking hours, most of which lately have been spent on packaging. As the year passes, and we move into fall here in Northern California, the thinking's been best assisted by having a brief, and warming, snack.

The notes being written are about some of the points raised about the use of hashing, but then I buckled down and wrote an outline for our ARC inception materials, which will probably take more than one or two Friday sessions.

Location: Café Borrone, Menlo Park, California.

[ T: OpenSolaris pkg ]

Indiana: More ways to get it

So there are a few ways to get a copy of the Indiana preview ISO:

• via HTTP from dlc.sun.com,
• via HTTP from genunix.org,
• via HTTP from osol.acm.jhu.edu,
• via FTP from ftp.df.lth.se, or
• via BitTorrent.

Both our mirrors are in Northern California, so—since it's a completely redistributable set of software—if you're interested in being a mirror (even in California), please let me know. If you're a torrent fan, come join me and seed.

Update 1: My thanks to trisk, who's put up a fast mirror (Internet 2) on the East Coast of the USA.

Update 2: My thanks to Tobias Lundquist, who's put up an FTP mirror (Internet 2) in Sweden.

Icon Design by IconBuffet (from "Durango Office" and "Durango Research").

[ T: OpenSolaris Solaris indiana ]

Indiana: VESA if you need it

Over the past week, as we kept reassembling the distro constructor, image packaging, and slim install, we tested installs on a bunch of laptops. This manual operation let me rehearse how to get things working if the Preview LiveCD doesn't have a graphics driver that will work. So: here's the VESA workaround.

1. Boot text mode. One of the non-default lines in the GRUB menu should say "text"—pick that one.
2. Login as jack. User is "jack", password is "jack".
3. Generate a representative xorg.conf. Become root and have Xorg generate an initial configuration. The root password is "opensolaris". (Enjoy that `root`'s default shell is 'bash' for a bit.)

   $ su
   Password:
   # /usr/X11/bin/Xorg -configure
   ...
   

   This procedure should create /jack/xorg.conf.new.
4. Change the driver. As root, you'll continue by editing that xorg.conf.new file in vi(1). Search for the Device section, and modify the Driver line to

       Driver  "vesa"
   

5. Make it go. Stay root and hand-launch GNOME.

   # /usr/X11/bin/xinit /usr/bin/dbus-launch gnome-session -- \
           /usr/X11/bin/Xorg -config /jack/xorg.conf.new :0
   

6. Launch the installer. Bring up a terminal, and invoke the installer, by typing install-lan &.

If your install fails, please file a bug report at defect.opensolaris.org. If it succeeds, but graphics doesn't work, you can follow the same steps, but you'll have to edit the GRUB entry. (Roughly, type 'e', add "-s" to the end of the line with kernel/unix, and log in with the new root password you set during install. I haven't tested this portion—let me know if it fails.)

Last night, I had to do this for my trusty VAIO T370P, but tonight it's fine:

Please let us know how it goes.

[ T: OpenSolaris Solaris indiana pkg ]

pkg(5): Talking in the redwoods, talking on the beach, ...

Danek and I are at the 2007 OpenSolaris Developer Summit at UC Santa Cruz, talking about and, I hope, later demonstrating the image packaging prototype so far. I gave a brief overview of the project's goals and status, and mentioning some of the unmentionables we've encountered—some particularly undisciplined configuration files, some apparently important but encumbered drivers, and so on.

View the slides

[ T: OpenSolaris pkg UCSC opensolaris_summit_2007 ]

pkg(5): project opens, development continues

I asked my teammates to take a brief break from prototyping, and we jumped from a collection of systems inside Sun to the clean project hosting at opensolaris.org.

As the migration registers, you should be able to access

• the project web site at http://opensolaris.org/os/project/pkg/,
• the Mercurial repository via

      $ hg clone ssh://anon@hg.opensolaris.org/hg/pkg/gate pkg-gate
  

• the same source via OpenGrok (using the 'pkg' project as a filter) at http://src.opensolaris.org/, and
• have a discussion on pkg-discuss@opensolaris.org.

If you're interested, please come check it out.

[ T: OpenSolaris Solaris pkg ]