Transitive

Enabling pkg(5) Availability

With the putback for the following issues in revision 1504:

9969 client support for multiple origins desired
11715 ImageConfig does not handle None for publisher correctly
11793 image-create example partially disagrees with the usage

...the client now supports multiple origins for publisher repositories.  The purpose of this change was to enable greater pkg (5) server availability and redundancy.

What is an origin?

An origin is simply a location of a package repository that contains package file, manifest, and catalog data; such as: http://pkg.opensolaris.org/dev.  The default mode for the pkg.depotd process is to run as an 'origin'.

A mirror, in contrast, only contains package file data and is currently provided by a pkg.depotd server running in --mirror mode.

What does this change mean?

This change means that redundancy can now more easily be built into package server infrastructure.  For example, you could have http://pkg1.example.com/, http://pkg2.example.com/, and http://pkg3.example.com/.

A client, once configured, would have pkg publisher output similar to the following:

PUBLISHER                TYPE   STATUS URI
example.com  (preferred) origin online http://pkg1.example.com/
example.com  (preferred) origin online http://pkg2.example.com/
example.com  (preferred) origin online http://pkg3.example.com/

During package operations, the client will automatically attempt to select and use the best repository origin if any data needs to be retrieved for that publisher.

In addition, ipkg branded zone creation and attach has been updated to use this additional origin and mirror information when configuring zones.

How do I use this?

For consistency, and to support this new client functionality, the image-create and set-publisher subcommands for pkg(1) have changed as follows:

image-create
 New Options:
  [-g|--origin ...] [-m|--mirror ...]

set-publisher
 New Options:
  [-g origin_to_add | --add-origin=origin_to_add ...]
  [-G origin_to_remove | --remove-origin=origin_to_remove ...]

 Deprecated Options:
  [-O uri]

So, as an example, to create a new, full image, with publisher example.com, that also has an additional mirror, two additional origins and that is stored at /aux0/example_root:

$ pkg image-create -F -p example.com=http://pkg.example.com:10000 \
  -g http://alternate1.example.com:10000/ \
  -g http://alternate2.example.com:10000/ \
  -m http://mirror.example.com:10000/ \
  /aux0/example_root

Alternatively, to add a new publisher or update an existing publisher with the above information:

$ pkg set-publisher \
  -g http://pkg.example.com:10000 \
  -g http://alternate1.example.com:10000/ \
  -g http://alternate2.example.com:10000/ \
  -m http://mirror.example.com:10000/ \
  example.com

Are there are any client compatibility concerns?

No.  Clients upgrading from older versions of the pkg client software to this version of the client software will not experience any issues.  In addition, some care has been taken to ensure older clients that understand the current image format are compatible with this new configuration, although they will be limited to using the first origin for a repository.

Do I need to update my scripts or client usage?

While the '-O' option has been deprecated for the set-publisher command and is no longer documented, it will continue to work exactly as it has in the past so no change is strictly necessary.  The command-line option compatibility for -O is currently planned to be maintained through the next major OpenSolaris release (2010.x), but will be dropped after that.

However, if you are using any scripts or programs that parse the output of the 'pkg publisher' command, and only expect a single origin to be listed for each publisher, they will need to be updated to account for this.

Comments or concerns should be sent to the pkg-discuss mailing list on opensolaris.org.

Posted at 02:26PM Nov 18, 2009 by Shawn Walker in pkg  |  Comments[0]

Do you want to go faster?

Please read this if you publish packages or run a pkg.depotd server!

Overview

The short version?  These changes make the client take a lot less time to perform common operations (info, list, etc.).  Although, a greater benefit is only realized when the client is configured to get package data from a repository that is being served by the new pkg.depotd server in this putback.

The Nitty Gritty

Revision 1431:62b6033670e4 in the pkg(5) gate integrated the following fixes and enhancements:

• 10416 server catalog v1 support desired

• 243 need localized descriptions, other metadata at catalog level

• 2424 Need to use UTC consistently everywhere

• 3092 messaging api/framework needed for pkg clients (cli, gui, etc.)

• 7063 "pkg list -a -s" needs performance improvement

• 7163 manifests are modified by client before being written to disk

• 8217 package fmri should be added to manifest during publishing

• 9061 importer should not refresh indexes

• 9446 traceback for cfg_cache operations if read-only filesystem

• 10415 client catalog v1 support desired

• 11094 Client transport for catalog v1

• 11523 only permit FMRIs from same publisher for network repositories

• 11831 server api version incompatible templates can cause traceback

• 11832 depot needs ability to seed / alter repository configuration

• 11954 importer shows zero packages processed unless debug enabled

• 12006 merge utility should have a test suite

Client Software Considerations

The pkg(1) client now requires that the name (prefix) that you have configured your publisher with matches that reported by its repository (origin) when the package repository provides a version 1 catalog.

For example, if you have your client configured this way:

---

$ pkg publisher
PUBLISHER       TYPE     STATUS URI
dev (preferred) origin   online http://pkg.opensolaris.org/dev

---

Then the next time a pkg refresh is performed, you will see the following warning:

---

$ pkg refresh
Refreshing catalog 1/1 test
The catalog retrieved for publisher 'dev' only contains package data for these publisher(s): opensolaris.org.  To resolve this issue, update this publisher to use the correct repository origin, or add one of the listed publishers using this publisher's repository origin.

To correct the repository origin, execute the following command as a privileged user:

pkg set-publisher -O <url> dev

To add a new publisher using this publisher's repository origin, execute the following command as a privileged user:

pkg set-publisher -O http://pkg.opensolaris.org/dev <publisher>

After the new publisher has been added, this one should be removed by executing the following command as a privileged user:

⁞pkg unset-publisher test

---

Please note that this means that available packages from that publisher (excluding already installed packages) will not be shown in pkg list, pkg info, etc.

This change is the first step in a set of changes to not require that a user specify the publisher's name (prefix) when using image-create, set-publisher, etc. and to further simplify configuration.

Disk space usage will also increase for the client when it is configured to use a publisher repository that contains catalog v1 data.  As an example, when pkg.opensolaris.org/dev's depot software is upgraded, it now requires roughly 36MB of disk space (uncompressed) for local metadata storage (51K+ packages).  This information does compress extremely well (around 7MB gzip'd).

Disk space usage will continue to slightly increase for each new build until historical and obsolete packages are implemented, at which point it will significantly decrease again.

Depot Software Considerations

As a result of these changes, if you use the new depot server with an existing repository without using the --readonly option it will be automatically upgraded to a new repository format.

If you use an older repository with the newer depot software, but want to use it with the newer depot software, you must specify --readonly and --writable-root.  If you do that, a new catalog will be written to the --writable-root directory based on the contents of the repository.

In addition, there is now a hard requirement that you specify a default publisher prefix for the repository.  Carefully consider what you specify here as it will be used during publication if the FMRI specified at publication time does not include a publisher prefix.  The publisher prefix of any published packages will be placed into package manifests, the catalog, etc. and so changing it once set will not update the existing package data inside the repository as it is only a default.

Once a repository has been upgraded to the new format, you will be unable to use older pkg(5) depot server software, pksend, or pkgrecv with the new repository.  If you attempt to use an older client with the upgraded repository, it will fail.

As a reminder, parallel publication is not supported at this time via file://, only with http://.  Various import tests have been performed to verify that publication performance is roughly the same as it was before.

Finally, please note that this upgrade behaviour also applies if you are using 'file://' locations for destination repositories with pkgrecv or pkgsend.

Thanks

Enough thanks cannot be given to team members that helped complete this phase of the catalog v1 project over the past month.  This was truly a team effort.  Transport work provided by our resident guru: johansen.

Posted at 06:51PM Oct 23, 2009 by Shawn Walker in pkg  | 

What's in a name? That which we call an Authority...

The pkg(5) team is always looking for ways to improve the user experience for those using the pkg(5) system and to promote a better understanding of it.  One of the ways to do that is to ensure that clear, consistent terminology is used to describe ideas and concepts.  That is why it was recently decided that a name change was in order for one of the core concepts in the pkg(5) system: authorities.

An authority is a forward or reverse domain name that identifies an organization (or group of individuals) responsible for providing one or more packages.  However, the name that was chosen for that concept caused some confusion among our users.  This was also further exacerbated by the interchangeable usage of the words "Authority" and "Repository" which are decidedly different in meaning.  As a result, starting with revision 926 of the pkg gate code,  an "Authority" is now called a "Publisher".

Before I move on to the details of the changes involved, I felt it was important to summarise what the current overall vision is for the structure of publishers and repositories and how the client is intended to use them:

• Publisher - a forward or reverse domain name (e.g. "opensolaris.org" or "org.opensolaris") that can be used to identify an organization that is responsible for one or more packaging repositories.  While a publisher may have (publish to) one or more repositories, pkg(5) clients are only intended to access one of them at a time (such as the "development" or "release" repositories for OpenSolaris).  Please note that for compatibility reasons, the client is currently restricted to a single repository per publisher; this will change after the 2009.06 release of OpenSolaris.
• Repository - a location where clients can publish and retrieve package content (files contained within the package such as programs, documents, etc.) and / or metadata (information about the package such as its name, description, etc.).  It has the following characteristics:
  
  • may have one or more origins (URIs) used for publication and retrieval of package metadata and content (the client currently only supports one of these)
    
  • may have zero or more mirrors (URIs) for retrieval of package content
  • can only contain packages from a single publisher

Now that you know what a publisher and a repository is, here is a high-level summary of the changes that were needed to make this happen:

• All references to "authority" were replaced with "publisher".
  
• A publisher management and client history logging interface was added to pkg.client.api so that all pkg(5) clients can consistently and easily manage publisher and repository information.
  
• The output of the "pkg publisher" subcommand (aka "pkg authority") was changed to include mirrors in the default listing.
• The cli and gui clients included with the pkg(5) system were updated to use the new publisher management interfaces.
• Support for generating and parsing .p5i data (pkg(5) publisher and package information files) was added to the api and to the gui client.  This functionality will eventually allow users to trigger installation of new software just by clicking a link in their web browser.
• Fixed SSL Certificate handling to be more robust so that the client won't fail with a Python traceback when performing network operations (such as downloading package data from a repository) and so that the user will know why the failure occurred (expired certificate, etc.).
• Fixed "pkg publisher" command to return a partial failure code of 3 if the information for one of the publishers or repositories couldn't be read (such as the SSL Certificate).
  

As a result of these changes the following commands have been renamed and altered as follows:

• pkg authority -> pkg publisher
  
  • The output of this command also changed.  Before, executing "pkg authority" would have given you a list of authorities and the origin URIs for each repository.  Now it shows repository mirrors as well and indicates their status and type and ensures that the preferred publisher is always listed first and the rest in alphabetical order by name.  Here is a before and after comparison from my own system:
    

    $ pkg authority
    AUTHORITY                           URL
    opensolaris.org                     http://pkg.opensolaris.org/dev/
    ipkg (preferred)                    http://ipkg.sfbay/dev/
    extra                               https://pkg.sun.com/opensolaris/extra/
    

    $ pkg publisher
    PUBLISHER                             TYPE     STATUS   URI
    ipkg                     (preferred)  origin   online   http://ipkg.sfbay/dev/
    extra                                 origin   online   https://pkg.sun.com/opensolaris/extra/
    opensolaris.org                       origin   online   http://pkg.opensolaris.org/dev/
    

• pkg set-authority -> pkg set-publisher
• pkg unset-authority -> pkg unset-publisher

All of the man pages and other technical documentation provided with the packaging system has been updated, and the IPS User Guide is in the process of being updated.  Please note that the old commands are still recognised for the purposes of backwards compatibility, but they will be removed after the 2009.06 release of OpenSolaris.

For those that are interested, the technical details are listed below:

Revision 926 / 6ee411c9026a

• 5871 publisher apis desired
• 3682 pkg publisher should show mirrors in default listing
• 5071 provide direct way to change publisher to not use certificate
• 5373 pkg commands shouldn't traceback when client can't read a key/cert file
• 6843 problems with empty string for ssl_cert, ssl_key
• 6897 "authority" should be "publisher"
• 7046 client intent is not sent if same api object used for multiple operations / reasons

Posted at 06:37PM Mar 10, 2009 by Shawn Walker in pkg  | 

"Direct" package publishing

Who says good things don't happen on Friday the 13th?

For several weeks now, I've been working on improvements to our publishing process.  The primary one, among those, is the ability to publish using pkgsend or solaris.py directly to disk without running the pkg.depotd server.  Here's a summary of the major changes:

Revision 873 / b95d76c53b5e

This revision integrated enhancements and fixes for the following issues:

• 2691 ability to publish packages to local disk repository
• 850 pkgsend traceback if invalid or unavailable repository url specified
• 894 pkgsend should do more action verification
• 1236 pkgsend open should not allow both -e and -n
• 2430 pkgsend add of non-existent file causes traceback
• 5088 server can still create a repo when in —readonly mode
• 5294 pkgsend doesn't provide server response when operations fail
• 5944 pkgsend traceback if invalid or no action type specified
• 6566 catalog is always rebuilt when server config is not readonly
  

As a result of these changes, the following changes in behaviour should be noted:

• pkg.depotd will no longer automatically create or populate the inst_root specified using pkg/inst_root or the -d option if started with —readonly. Instead, it will exit with an error message. Starting the server in a mode other than readonly will create the specified directory and any necessary subdirectories or files.
• pkg.depotd will no longer automatically rebuild the repository catalog if not started with —readonly. However, search indices will continue to be refreshed automatically if needed. If, for some reason, the package catalog does need to be rebuilt on pkg.depotd startup or the old behaviour is desired, pkg.depotd can be started with the —rebuild option. This change should result in significantly faster startup (and less disk I/O) for large repositories served by pkg.depotd in publishing mode.
• pkgsend and the importer now support file:// URLs for publishing. In the case of pkgsend:
  
  PKG_REPO=file:///path/to/repo pkgsend open foo@1.0
  pkgsend -s file:///path/to/repo open foo@1.0
  
  In the case of solaris.py (the importer), you will need to set the REPO variable in the makefile or override it another way. Be aware that if you specify a directory that is not a valid pkg(5) repository, publishing will fail with an error indicating that. You must create a repository first by starting the pkg.depotd server with the desired path and then exiting. A subcommand to create repositories will be added to pkgsend in the near future (bug 6574).
• It is important to note that only one user at a time should publish directly to a file:// repository URL; the result of more than one user publishing to a file:// repository URL at a time is completely untested and is likely to fail. Bug 2696 will help enforce this by ensuring that a repository lock is obtained during publishing mode. If you need the ability to allow multiple users to simultaneously publish to a repository, you must use pkg.depotd and publish to a http:// repository URL. The tradeoff is that file:// repository URL publishing can be significantly faster and will result in significantly reduced memory usage and disk I/O.
• Significantly more verification is now done during the publishing process. Expect things that may have appeared to work before to no longer be accepted by the server or by the pkgsend client. Additionally, pkgsend once more produces useful error information when operations fail.

I'd be interested in any feedback others might have about these changes or ways we can improve the process.  These changes should appear in build 108 of OpenSolaris 2009.  Enjoy!

Posted at 03:44PM Feb 13, 2009 by Shawn Walker in pkg  | 

A New Face on Things

As part of the drive for the OpenSolaris 2008.11 release, the pkg(5) team has been working hard on improving the user experience, and fixing as many outstanding issues as we possibly can in time for the release.  Among those recent improvements is a face-lift of the depot server's BUI (browser user interface).  The short summary of these changes is that users can now search package metadata and browse packages by release and branch.  You can see the result of these changes at the primary OpenSolaris packaging repository website.

This face-lift was more than just "skin-deep" though.  As part of the changes implemented, all of the depot web-facing pages are now Mako templates and there is a new python API specially designed for these pages that can be found in pkg.server.api  (see pydoc pkg.server.api for more information).

What does this mean?  Projects that want to integrate their own web content or customise the presentation of their depot server can now do so with almost no restrictions.  It also means that future ehancements to the depot pages will be far easier.  Please note that these changes are not yet included in an OpenSolaris release; if you'd like to run your own depot server and benefit from these changes, you'll need to build and install the latest version from the pkg gate.

The good news is that there is also a great number of improvements still planned.  The following enhancements are set to be implemented in the future (if you don't see what you need or want in the list below, please open an RFE):

• Advanced Depot Statistics Page
• Advanced Search Capabilities (ability to filter searches by release and branch, only show latest versions, etc.)
• Category-based RSS/Atom Feed (so users can easily get updates from a feed reader about changes to specific packages, or for security updates for any packages, etc.)
  
• Comprehensive Package Browsing Capability (so that you can easily navigate through the files contained within a package, it's dependencies, etc.)
  
• Depot Page Customisation Documentation
• Improved Package Information page (instead of the place-holder text-only display we currently have) that ties into the comprehensive browsing mentioned above.
• Package Count by Release and Branch
  
• Package Popularity / Ratings Information
• Recent Catalog Additions
  
• Useful Links section

 Finally, today also marks the official release of OpenSolaris 2008.11.  This release is the culmination of the hard work of many great people in the OpenSolaris community inside and outside of Sun (including independent community members; and folks from Intel, AMD, and others) and other communities as well.  My greatest hope is that if you're reading this, and you haven't tried OpenSolaris yet, you will now.

Congratulations and thanks to all who worked so hard to make this release a reality.

Posted at 12:54PM Dec 10, 2008 by Shawn Walker in pkg  | 

A New Beginning

Monday, July 7th, 2008 was my first day as a Sun Employee. I never dreamed I would be here. The road here has certainly be an interesting one.

I've spent many years involved in various open source projects and communities. However, the OpenSolaris community was the first one to really grab my heart and attention for any significant length of time.

That's why when I was finally given the opportunity to work for Sun, I could think of nothing else I'd rather do. I believe in what Sun is doing; they're not just changing their business model, opening source code, or doing their best to keep shareholders happy -- they're trying to bridge the digital divide.

The digital divide is what separates our society as a whole from truly communicating or achieving their goals. When people are of one heart and one mind, they can achieve great things.

It is my belief that Sun, through open standards, open technology, and open processes is the company that can empower others to bridge that divide and ultimately bring people together.

Lots of companies claim to be innovators, but ultimately are just lining their own pockets.

I believe that Sun has realized what few others have: True innovation is when you empower people in such a way that they can improve their own lives and that of others.

I can't think of anything greater that I'd want to be a part of, can you?

Update: I've had a few people ask what I will be doing here at Sun and in what area. In short, I'm part of Solaris Kernel Engineering. I'll be working with the great folks on the ips team to bring the new packaging system up to speed. Specifically, I will initially be doing a lot of work on the web interface / repository and assisting wherever else is needed. I'm currently working from Overland Park, KS.

Posted at 11:10PM Jul 10, 2008 by Shawn Walker in Sun  |  Comments[5]