Transitive

Monday Feb 08, 2010

Simplifying Publisher Configuration

Changeset 1736 in the pkg(5) gate made the following changes:

11522 pkg should require publisher prefix to match repository information
7156 client image api needs image creation interface
12744 update_publisher over-zealously testing publisher validity
14203 image-create usage doesn't mention mirror / origin options

What changed?

The pkg(1) client and the pkg.client.api will now attempt to retrieve publisher configuration information from package repositories. This information is used to auto-configure new publishers, validate configuration requests, and to update the configuration of existing publishers.

These changes were made with a focus on simplifying publisher addition and updates of existing publisher configuration.

Further improvements and additions to this functionality are planned for future releases.

Are there any pkg.depotd(1m) changes?

While the pkg.depotd(1m) program did not change, clients are now more reliant on its correct configuration.

As already mentioned in the past, please be certain that you set the publisher.prefix property found in the repository's cfg_cache file correctly. The pkg.depotd man page contains instructions on how to set this property.

Please also note that the repository.origins and/or repository.mirrors properties should be set as clients will use these to automatically configure new publishers and to update existing publishers. You might also want to consider providing a description so that users have an idea of what sort of packages the repository contains.

Part of an example cfg_cache might look like this:

[publisher]
prefix = example.com
alias = None
...
[repository]
origins = http://pkg-us.example.com/,http://pkg-ca.example.com/
mirrors = http://pkg-us-mirror1.example.com/,http://pkg-ca-mirror1.example.com/
...

How has pkg(1) image-create changed?

The client remains completely compatible syntax-wise, so no changes to existing scripts are required, and behaviour when using the older syntax (other than validating publisher configuration) remains unchanged.

However, image-create now also accepts -p <uri> where <uri> is the URI of a package repository and example usage is as follows:

pkg image-create -p <uri> /target

When using this syntax, image-create will retrieve all of the publisher's configuration information from the target package repository and add all of the publishers it finds to the image.

How has pkg(1) set-publisher changed?

The syntax of the set-publisher command remains backwards compatible, so no changes to existing scripts, etc. are required. However, set-publisher now accepts the -p option which accepts the URI of a package repository from which all publisher configuration information will be retrieved.

Any new publishers found in the retrieved configuration information will be added, while existing ones will be updated if the provided URI is already in the list of configured origins for the publisher being updated.

If a publisher name is provided, then -p will only use publisher configuration information that matches the provided name.

As an example, the old syntax to add the contrib repository might have been:

pkg set-publisher -g http://pkg.opensolaris.org/contrib \

  contrib.opensolaris.org

The new syntax is:

pkg set-publisher -p http://pkg.opensolaris.org/contrib

How has pkg.client.api changed?

Incompatible with clients using API versions 0-31.

The ImageInterface class has changed as follows:

The add_publisher and update_publisher methods now validate the image's publisher configuration against the origins of the publisher. If any of the origins are found to not match, an UnknownRepositoryPublishers exception will be raised. If one of the new repository origins does not provide publisher configuration information or it is incomplete, a RepoPubConfigUnavailable exception will be raised.

The pkg.client.api module has changed as follows:

A new method named image_create has been added. See 'pydoc pkg.client.api' for details.

The pkg.client.api_errors modules has changed as follows:

UnknownRepositoryPublishers, RepoPubConfigUnavailable, and UnknownErrors expections have been added for use by the pkg.client.api. API consumers are reminded that they should catch all ApiException class exceptions, although catching specific exception subclasses for case-by-case handling in addition to that is acceptable.

Feedback is welcomed on the pkg-discuss mailing list on opensolaris.org.

Posted at 05:46PM Feb 08, 2010 by Shawn Walker in Personal | Comments[0]

Thursday Jan 07, 2010

New pkg(5) license functionality

For those of you that require license acceptance or display for your packages, you should be aware that starting with OpenSolaris development build b131, new functionality will be delivered to support license acceptance and display.

The following fixes and enhancements were integrated:

5943 add 'must-accept' attribute to license actions
5586 licenseinfo api needs to expose license action attributes
13155 add 'must-display' attribute to license actions
13158 change pkg.client.api to understand and require license acceptance
13160 pkg(1) needs update for client api license acceptance changes

These changes were made to support packages that require display or acceptance of license related data during package install and update operations. As a result of these changes, the pkg(1) client and client API require explicit acceptance and/or indication of license display during install and update operations.

Please note that (currently) no packages exist that use this functionality, but they are expected in the near future.

Client Considerations

If any of the license actions contained in a package being installed or updated have must-accept=true, the following pkg(1) subcommands require that the new --accept option be provided before the operation will proceed:

install
fix
image-update
change-variant
change-facet

In addition, all of the above subcommands also now have a --licenses option to display the payload of all the licenses for packages part of the operation. For example:

pkg install -n --licenses foo

The above command would display all of the licenses for the packages that would be installed or updated if the package 'foo' were installed or updated. If the --accept option is not provided, and a license requires acceptance, the pkg(1) client will now exit with exit code 6, indicating license acceptance failure. If a license requires display, the pkg(1) client will display it during install/update operations; this cannot be suppressed.

Client API Changes

Version 29:
Incompatible with clients using versions 0-28:
The ImageInterface class has changed as follows:

set_plan_license_status() was added. This is used to indicate
          whether licenses for the packages being operated on have been
          accepted or displayed. Clients must do this if the related
          license requires acceptance or display.

The LicenseInfo class has changed as follows:

get_text() may now trigger a remote retrieval of the license
payload if needed to return the text.

The related package FMRI and license attributes are now
properties: fmri, license, must_accept, and must_display.

The PlanDescription class has changed as follows:

get_changes() is now a generator function.

get_licenses() was added to allow clients to retrieve the
          list of licenses related to the plan's operations as well
          as the current accepted and displayed status of each. Please
          note that this function returns _all_ licenses related to the
          operation; not just those that require acceptance or display.

Publication Considerations

To use this new license acceptance functionality, simply add must-accept=true or must-display=true (as appropriate) to license actions in your package manifest. An example pkgsend sequence might look like this:

open licensed@1.3,5.11-0
add depend type=require fmri=baz@1.0
add file /tmp/libc.so.1 mode=0555 owner=root group=bin path=/lib/libc.so.1
add license /tmp/libc.copyright license=libc.copyright must-display=True
add license /tmp/libc.license license=libc.license must-accept=True
close

Please note that this functionality is not supported before build 131, and that you should use this functionality sparingly. must-accept=true should not be placed on the majority of open source licenses (BSD, GPL, etc.) and must-display should only be set if absolutely necessary.

Comments or concerns should be sent to the pkg-discuss mailing list on opensolaris.org.

Posted at 01:40PM Jan 07, 2010 by Shawn Walker in pkg |

Wednesday Nov 18, 2009

Enabling pkg(5) Availability

With the putback for the following issues in revision 1504:

9969 client support for multiple origins desired
11715 ImageConfig does not handle None for publisher correctly
11793 image-create example partially disagrees with the usage

...the client now supports multiple origins for publisher repositories. The purpose of this change was to enable greater pkg (5) server availability and redundancy.

What is an origin?

An origin is simply a location of a package repository that contains package file, manifest, and catalog data; such as: http://pkg.opensolaris.org/dev. The default mode for the pkg.depotd process is to run as an 'origin'.

A mirror, in contrast, only contains package file data and is currently provided by a pkg.depotd server running in --mirror mode.

What does this change mean?

This change means that redundancy can now more easily be built into package server infrastructure. For example, you could have http://pkg1.example.com/, http://pkg2.example.com/, and http://pkg3.example.com/.

A client, once configured, would have pkg publisher output similar to the following:

PUBLISHER TYPE STATUS URI
example.com (preferred) origin online http://pkg1.example.com/
example.com (preferred) origin online http://pkg2.example.com/
example.com (preferred) origin online http://pkg3.example.com/

During package operations, the client will automatically attempt to select and use the best repository origin if any data needs to be retrieved for that publisher.

In addition, ipkg branded zone creation and attach has been updated to use this additional origin and mirror information when configuring zones.

How do I use this?

For consistency, and to support this new client functionality, the image-create and set-publisher subcommands for pkg(1) have changed as follows:

image-create
New Options:
[-g|--origin ...] [-m|--mirror ...]

set-publisher
New Options:
[-g origin_to_add | --add-origin=origin_to_add ...]
[-G origin_to_remove | --remove-origin=origin_to_remove ...]

Deprecated Options:
[-O uri]

So, as an example, to create a new, full image, with publisher example.com, that also has an additional mirror, two additional origins and that is stored at /aux0/example_root:

$ pkg image-create -F -p example.com=http://pkg.example.com:10000 \
-g http://alternate1.example.com:10000/ \
-g http://alternate2.example.com:10000/ \
-m http://mirror.example.com:10000/ \
/aux0/example_root

Alternatively, to add a new publisher or update an existing publisher with the above information:

$ pkg set-publisher \
-g http://pkg.example.com:10000 \
-g http://alternate1.example.com:10000/ \
-g http://alternate2.example.com:10000/ \
-m http://mirror.example.com:10000/ \
example.com

Are there are any client compatibility concerns?

No. Clients upgrading from older versions of the pkg client software to this version of the client software will not experience any issues. In addition, some care has been taken to ensure older clients that understand the current image format are compatible with this new configuration, although they will be limited to using the first origin for a repository.

Do I need to update my scripts or client usage?

While the '-O' option has been deprecated for the set-publisher command and is no longer documented, it will continue to work exactly as it has in the past so no change is strictly necessary. The command-line option compatibility for -O is currently planned to be maintained through the next major OpenSolaris release (2010.x), but will be dropped after that.

However, if you are using any scripts or programs that parse the output of the 'pkg publisher' command, and only expect a single origin to be listed for each publisher, they will need to be updated to account for this.

Comments or concerns should be sent to the pkg-discuss mailing list on opensolaris.org.

Posted at 02:26PM Nov 18, 2009 by Shawn Walker in pkg |

Friday Oct 23, 2009

Do you want to go faster?

Please read this if you publish packages or run a pkg.depotd server!

Overview

The short version? These changes make the client take a lot less time to perform common operations (info, list, etc.). Although, a greater benefit is only realized when the client is configured to get package data from a repository that is being served by the new pkg.depotd server in this putback.

The Nitty Gritty

Revision 1431:62b6033670e4 in the pkg(5) gate integrated the following fixes and enhancements:

10416 server catalog v1 support desired
243 need localized descriptions, other metadata at catalog level
2424 Need to use UTC consistently everywhere
3092 messaging api/framework needed for pkg clients (cli, gui, etc.)
7063 "pkg list -a -s" needs performance improvement
7163 manifests are modified by client before being written to disk
8217 package fmri should be added to manifest during publishing
9061 importer should not refresh indexes
9446 traceback for cfg_cache operations if read-only filesystem
10415 client catalog v1 support desired
11094 Client transport for catalog v1
11523 only permit FMRIs from same publisher for network repositories
11831 server api version incompatible templates can cause traceback
11832 depot needs ability to seed / alter repository configuration
11954 importer shows zero packages processed unless debug enabled
12006 merge utility should have a test suite

Client Software Considerations

The pkg(1) client now requires that the name (prefix) that you have configured your publisher with matches that reported by its repository (origin) when the package repository provides a version 1 catalog.

For example, if you have your client configured this way:

$ pkg publisher
PUBLISHER       TYPE     STATUS URI
dev (preferred) origin   online http://pkg.opensolaris.org/dev

Then the next time a pkg refresh is performed, you will see the following warning:

$ pkg refresh
Refreshing catalog 1/1 test
The catalog retrieved for publisher 'dev' only contains package data for these publisher(s): opensolaris.org. To resolve this issue, update this publisher to use the correct repository origin, or add one of the listed publishers using this publisher's repository origin.

To correct the repository origin, execute the following command as a privileged user:

pkg set-publisher -O <url> dev

To add a new publisher using this publisher's repository origin, execute the following command as a privileged user:

pkg set-publisher -O http://pkg.opensolaris.org/dev <publisher>

After the new publisher has been added, this one should be removed by executing the following command as a privileged user:

⁞pkg unset-publisher test

Please note that this means that available packages from that publisher (excluding already installed packages) will not be shown in pkg list, pkg info, etc.

This change is the first step in a set of changes to not require that a user specify the publisher's name (prefix) when using image-create, set-publisher, etc. and to further simplify configuration.

Disk space usage will also increase for the client when it is configured to use a publisher repository that contains catalog v1 data. As an example, when pkg.opensolaris.org/dev's depot software is upgraded, it now requires roughly 36MB of disk space (uncompressed) for local metadata storage (51K+ packages). This information does compress extremely well (around 7MB gzip'd).

Disk space usage will continue to slightly increase for each new build until historical and obsolete packages are implemented, at which point it will significantly decrease again.

Depot Software Considerations

As a result of these changes, if you use the new depot server with an existing repository without using the --readonly option it will be automatically upgraded to a new repository format.

If you use an older repository with the newer depot software, but want to use it with the newer depot software, you must specify --readonly and --writable-root. If you do that, a new catalog will be written to the --writable-root directory based on the contents of the repository.

In addition, there is now a hard requirement that you specify a default publisher prefix for the repository. Carefully consider what you specify here as it will be used during publication if the FMRI specified at publication time does not include a publisher prefix. The publisher prefix of any published packages will be placed into package manifests, the catalog, etc. and so changing it once set will not update the existing package data inside the repository as it is only a default.

Once a repository has been upgraded to the new format, you will be unable to use older pkg(5) depot server software, pksend, or pkgrecv with the new repository. If you attempt to use an older client with the upgraded repository, it will fail.

As a reminder, parallel publication is not supported at this time via file://, only with http://. Various import tests have been performed to verify that publication performance is roughly the same as it was before.

Finally, please note that this upgrade behaviour also applies if you are using 'file://' locations for destination repositories with pkgrecv or pkgsend.

Thanks

Enough thanks cannot be given to team members that helped complete this phase of the catalog v1 project over the past month. This was truly a team effort. Transport work provided by our resident guru: johansen.

Posted at 06:51PM Oct 23, 2009 by Shawn Walker in pkg |

Tuesday Mar 10, 2009

What's in a name? That which we call an Authority...

The pkg(5) team is always looking for ways to improve the user experience for those using the pkg(5) system and to promote a better understanding of it. One of the ways to do that is to ensure that clear, consistent terminology is used to describe ideas and concepts. That is why it was recently decided that a name change was in order for one of the core concepts in the pkg(5) system: authorities.

An authority is a forward or reverse domain name that identifies an organization (or group of individuals) responsible for providing one or more packages. However, the name that was chosen for that concept caused some confusion among our users. This was also further exacerbated by the interchangeable usage of the words "Authority" and "Repository" which are decidedly different in meaning. As a result, starting with revision 926 of the pkg gate code, an "Authority" is now called a "Publisher".

Before I move on to the details of the changes involved, I felt it was important to summarise what the current overall vision is for the structure of publishers and repositories and how the client is intended to use them:

Publisher - a forward or reverse domain name (e.g. "opensolaris.org" or "org.opensolaris") that can be used to identify an organization that is responsible for one or more packaging repositories. While a publisher may have (publish to) one or more repositories, pkg(5) clients are only intended to access one of them at a time (such as the "development" or "release" repositories for OpenSolaris). Please note that for compatibility reasons, the client is currently restricted to a single repository per publisher; this will change after the 2009.06 release of OpenSolaris.
Repository - a location where clients can publish and retrieve package content (files contained within the package such as programs, documents, etc.) and / or metadata (information about the package such as its name, description, etc.). It has the following characteristics:
- may have one or more origins (URIs) used for publication and retrieval of package metadata and content (the client currently only supports one of these)
- may have zero or more mirrors (URIs) for retrieval of package content
- can only contain packages from a single publisher

Now that you know what a publisher and a repository is, here is a high-level summary of the changes that were needed to make this happen:

All references to "authority" were replaced with "publisher".
A publisher management and client history logging interface was added to pkg.client.api so that all pkg(5) clients can consistently and easily manage publisher and repository information.
The output of the "pkg publisher" subcommand (aka "pkg authority") was changed to include mirrors in the default listing.
The cli and gui clients included with the pkg(5) system were updated to use the new publisher management interfaces.
Support for generating and parsing .p5i data (pkg(5) publisher and package information files) was added to the api and to the gui client. This functionality will eventually allow users to trigger installation of new software just by clicking a link in their web browser.
Fixed SSL Certificate handling to be more robust so that the client won't fail with a Python traceback when performing network operations (such as downloading package data from a repository) and so that the user will know why the failure occurred (expired certificate, etc.).
Fixed "pkg publisher" command to return a partial failure code of 3 if the information for one of the publishers or repositories couldn't be read (such as the SSL Certificate).

As a result of these changes the following commands have been renamed and altered as follows:

pkg authority -> pkg publisher

The output of this command also changed. Before, executing "pkg authority" would have given you a list of authorities and the origin URIs for each repository. Now it shows repository mirrors as well and indicates their status and type and ensures that the preferred publisher is always listed first and the rest in alphabetical order by name. Here is a before and after comparison from my own system:

$ pkg authority
AUTHORITY                           URL
opensolaris.org                     http://pkg.opensolaris.org/dev/
ipkg (preferred)                    http://ipkg.sfbay/dev/
extra                               https://pkg.sun.com/opensolaris/extra/

$ pkg publisher
PUBLISHER                             TYPE     STATUS   URI
ipkg                     (preferred)  origin   online   http://ipkg.sfbay/dev/
extra                                 origin   online   https://pkg.sun.com/opensolaris/extra/
opensolaris.org                       origin   online   http://pkg.opensolaris.org/dev/

pkg set-authority -> pkg set-publisher
pkg unset-authority -> pkg unset-publisher

All of the man pages and other technical documentation provided with the packaging system has been updated, and the IPS User Guide is in the process of being updated. Please note that the old commands are still recognised for the purposes of backwards compatibility, but they will be removed after the 2009.06 release of OpenSolaris.

For those that are interested, the technical details are listed below:

Revision 926 / 6ee411c9026a

5871 publisher apis desired
3682 pkg publisher should show mirrors in default listing
5071 provide direct way to change publisher to not use certificate
5373 pkg commands shouldn't traceback when client can't read a key/cert file
6843 problems with empty string for ssl_cert, ssl_key
6897 "authority" should be "publisher"
7046 client intent is not sent if same api object used for multiple operations / reasons

Posted at 06:37PM Mar 10, 2009 by Shawn Walker in pkg |

Friday Feb 13, 2009

"Direct" package publishing

Who says good things don't happen on Friday the 13th?

For several weeks now, I've been working on improvements to our publishing process. The primary one, among those, is the ability to publish using pkgsend or solaris.py directly to disk without running the pkg.depotd server. Here's a summary of the major changes:

Revision 873 / b95d76c53b5e

This revision integrated enhancements and fixes for the following issues:

2691 ability to publish packages to local disk repository
850 pkgsend traceback if invalid or unavailable repository url specified
894 pkgsend should do more action verification
1236 pkgsend open should not allow both -e and -n
2430 pkgsend add of non-existent file causes traceback
5088 server can still create a repo when in —readonly mode
5294 pkgsend doesn't provide server response when operations fail
5944 pkgsend traceback if invalid or no action type specified
6566 catalog is always rebuilt when server config is not readonly

As a result of these changes, the following changes in behaviour should be noted:

pkg.depotd will no longer automatically create or populate the inst_root specified using pkg/inst_root or the -d option if started with —readonly. Instead, it will exit with an error message. Starting the server in a mode other than readonly will create the specified directory and any necessary subdirectories or files.
pkg.depotd will no longer automatically rebuild the repository catalog if not started with —readonly. However, search indices will continue to be refreshed automatically if needed. If, for some reason, the package catalog does need to be rebuilt on pkg.depotd startup or the old behaviour is desired, pkg.depotd can be started with the —rebuild option. This change should result in significantly faster startup (and less disk I/O) for large repositories served by pkg.depotd in publishing mode.
pkgsend and the importer now support file:// URLs for publishing. In the case of pkgsend:

PKG_REPO=file:///path/to/repo pkgsend open foo@1.0
pkgsend -s file:///path/to/repo open foo@1.0

In the case of solaris.py (the importer), you will need to set the REPO variable in the makefile or override it another way. Be aware that if you specify a directory that is not a valid pkg(5) repository, publishing will fail with an error indicating that. You must create a repository first by starting the pkg.depotd server with the desired path and then exiting. A subcommand to create repositories will be added to pkgsend in the near future (bug 6574).
It is important to note that only one user at a time should publish directly to a file:// repository URL; the result of more than one user publishing to a file:// repository URL at a time is completely untested and is likely to fail. Bug 2696 will help enforce this by ensuring that a repository lock is obtained during publishing mode. If you need the ability to allow multiple users to simultaneously publish to a repository, you must use pkg.depotd and publish to a http:// repository URL. The tradeoff is that file:// repository URL publishing can be significantly faster and will result in significantly reduced memory usage and disk I/O.
Significantly more verification is now done during the publishing process. Expect things that may have appeared to work before to no longer be accepted by the server or by the pkgsend client. Additionally, pkgsend once more produces useful error information when operations fail.

I'd be interested in any feedback others might have about these changes or ways we can improve the process. These changes should appear in build 108 of OpenSolaris 2009. Enjoy!

Posted at 03:44PM Feb 13, 2009 by Shawn Walker in pkg |

Wednesday Dec 10, 2008

A New Face on Things

As part of the drive for the OpenSolaris 2008.11 release, the pkg(5) team has been working hard on improving the user experience, and fixing as many outstanding issues as we possibly can in time for the release. Among those recent improvements is a face-lift of the depot server's BUI (browser user interface). The short summary of these changes is that users can now search package metadata and browse packages by release and branch. You can see the result of these changes at the primary OpenSolaris packaging repository website.

This face-lift was more than just "skin-deep" though. As part of the changes implemented, all of the depot web-facing pages are now Mako templates and there is a new python API specially designed for these pages that can be found in pkg.server.api (see pydoc pkg.server.api for more information).

What does this mean? Projects that want to integrate their own web content or customise the presentation of their depot server can now do so with almost no restrictions. It also means that future ehancements to the depot pages will be far easier. Please note that these changes are not yet included in an OpenSolaris release; if you'd like to run your own depot server and benefit from these changes, you'll need to build and install the latest version from the pkg gate.

The good news is that there is also a great number of improvements still planned. The following enhancements are set to be implemented in the future (if you don't see what you need or want in the list below, please open an RFE):

Advanced Depot Statistics Page
Advanced Search Capabilities (ability to filter searches by release and branch, only show latest versions, etc.)
Category-based RSS/Atom Feed (so users can easily get updates from a feed reader about changes to specific packages, or for security updates for any packages, etc.)
Comprehensive Package Browsing Capability (so that you can easily navigate through the files contained within a package, it's dependencies, etc.)
Depot Page Customisation Documentation
Improved Package Information page (instead of the place-holder text-only display we currently have) that ties into the comprehensive browsing mentioned above.
Package Count by Release and Branch
Package Popularity / Ratings Information
Recent Catalog Additions
Useful Links section

Finally, today also marks the official release of OpenSolaris 2008.11. This release is the culmination of the hard work of many great people in the OpenSolaris community inside and outside of Sun (including independent community members; and folks from Intel, AMD, and others) and other communities as well. My greatest hope is that if you're reading this, and you haven't tried OpenSolaris yet, you will now.

Congratulations and thanks to all who worked so hard to make this release a reality.

Posted at 12:54PM Dec 10, 2008 by Shawn Walker in pkg |

Thursday Jul 10, 2008

A New Beginning

Monday, July 7th, 2008 was my first day as a Sun Employee. I never dreamed I would be here. The road here has certainly be an interesting one.

I've spent many years involved in various open source projects and communities. However, the OpenSolaris community was the first one to really grab my heart and attention for any significant length of time.

That's why when I was finally given the opportunity to work for Sun, I could think of nothing else I'd rather do. I believe in what Sun is doing; they're not just changing their business model, opening source code, or doing their best to keep shareholders happy -- they're trying to bridge the digital divide.

The digital divide is what separates our society as a whole from truly communicating or achieving their goals. When people are of one heart and one mind, they can achieve great things.

It is my belief that Sun, through open standards, open technology, and open processes is the company that can empower others to bridge that divide and ultimately bring people together.

Lots of companies claim to be innovators, but ultimately are just lining their own pockets.

I believe that Sun has realized what few others have: True innovation is when you empower people in such a way that they can improve their own lives and that of others.

I can't think of anything greater that I'd want to be a part of, can you?

Update: I've had a few people ask what I will be doing here at Sun and in what area. In short, I'm part of Solaris Kernel Engineering. I'll be working with the great folks on the ips team to bring the new packaging system up to speed. Specifically, I will initially be doing a lot of work on the web interface / repository and assisting wherever else is needed. I'm currently working from Overland Park, KS.

Posted at 11:10PM Jul 10, 2008 by Shawn Walker in Sun |