Local Repository Mirror
In my previous entry I wrote about creating a portable version of the repository using a USB stick. That option is great for taking the repository on the road and even sharing it with folks at conferences where the Internet connection usually sucks.
However, a more practical use of this repository image to many of you may be setting up a local mirror. In this situation, you get the best of both words - fast access to the repository with continued access to updates as they become available.
Setting Up the Local Repository
If you haven't already, download the ISO image of the repository (about 7 GB). After the download completes, mount the ISO:pfexec mount -F hsfs `pfexec lofiadm -a ~/Download/osol-repo-0906-full.iso` /mnt
In order for the mount to persist across reboots, we'll move it to a new file system, which we first need to create:
pfexec zfs create -o compression=on rpool/repo
Then copy the repository to the file system (this will take a little over an hour to complete):
bleonard@opensolaris:~$ pfexec rsync -aP /mnt/repo /rpool/repo
...
repo/updatelog/2009071315
67506 100% 150.17kB/s 0:00:00 (xfer#304859, to-check=1/609926)
repo/updatelog/2009071316
24956 100% 55.26kB/s 0:00:00 (xfer#304860, to-check=0/609926)
sent 6669164656 bytes received 8537336 bytes 1368521.77 bytes/sec
total size is 6630966532 speedup is 0.99
Configure the package server to use the local repository:
svccfg -s application/pkg/server setprop pkg/inst_root=/rpool/repo/repo svccfg -s application/pkg/server setprop pkg/readonly=true
You can also choose a port number different from the default of 80, which I did because I'm running Apache on port 80:
svccfg -s application/pkg/server setprop pkg/port=81
Edit the repo/cfg_cache file, changing the origins property from http://pkg.opensolaris.org/release to http://<domainname>:<port>, where <domainname> is network accessible and <port> is needed if you used something other than the default of 80. For example:
origins = http://opensolaris:81
Then refresh the package server service to pick up the configuration changes and start it:
svcadm refresh application/pkg/server svcadm enable application/pkg/server
Accessing the Local Repository
Now all those wishing to use the local package server simply need to make the following change:
pfexec pkg set-publisher -m http://opensolaris:81 opensolaris.org
Which will result in the following:
bleonard@opensolaris:~$ pkg publisher PUBLISHER TYPE STATUS URI opensolaris.org (preferred) origin online http://pkg.opensolaris.org/release/ opensolaris.org (preferred) mirror online http://opensolaris:81/
If you only want to use the local repository, first remove the mirror:
pfexec pkg set-publisher -M http://opensolaris:81 opensolaris.org
Then:
For more information see the README.pfexec pkg set-publisher -O http://opensolaris:81 opensolaris.org
Hello Brian,
thank you for this very useful post, it helped me set up my repository. However I think there is a mistake when configuring the location of the repository:
svccfg -s application/pkg/server pkg/inst_rool=/rpool/repo/repo
Should read
svccfg -s application/pkg/server setprop pkg/inst_root=/rpool/repo/repo
It should use "setprop" and the property is pkg/inst_root not inst_rool !
Regards
Steve
Posted by Stephen Myles on August 13, 2009 at 12:28 PM GMT #
Hi Stephen,
Thanks for the correction. I have a general rule to copy and paste directly from the terminal - which I obviously broke this time. Should be all set now.
/Brian
Posted by Brian Leonard on August 13, 2009 at 03:57 PM GMT #
Hi Brian,
Looks much better.
Regards
Steve
Posted by Stephen Myles on August 17, 2009 at 10:50 AM GMT #
Hi,
It took me only 39 minutes to copy the entire FS on a MacBook Pro.
Cheers
Posted by KBerrah on September 08, 2009 at 03:47 PM GMT #
Hi,
Good job, but how do you update your local repository from version 111 to latest (currently 122) ?
Thanks
Posted by Christophe Pauliat on September 10, 2009 at 12:49 PM GMT #
Hi Christophe,
I believe only the 2009.06 release repository has been made available in this way. Regardless, I don't think it would be possible to update the local repository. I think would you have to download and copy it again - which would be a time consuming endeavor every 2 weeks.
I would try the pkg-discuss e-mail alias for a more thorough answer.
Regards,
Brian
Posted by Brian Leonard on September 10, 2009 at 01:00 PM GMT #
Hi Brian,
It worked fine ! excellent to speed up download when installing big packages (amp-dev, sunstudio, openoffice).
Two things:
- the default port is 10000, not 80
- the service refused to start until I added the hostname in /etc/hosts
With this post, the Switzerland OpenSolaris User group could setup his own IPS mirror. So, thanks again for you blog !
Karim
Posted by KBerrah on September 11, 2009 at 01:29 PM GMT #
Hi Karim,
Great news. With regards to your notes, if I run the following on a fresh installation:
bleonard@opensolaris:~$ svccfg -s application/pkg/server listprop pkg/port
pkg/port count 80
The default port is 80, not 1000. Also, my host name (opensolaris) was already in my /etc/hosts file:
bleonard@opensolaris:~$ cat /etc/hosts
...
::1 localhost
127.0.0.1 localhost loghost opensolaris
However, I am curious why it was different in your case.
Posted by Brian Leonard on September 11, 2009 at 03:13 PM GMT #
hello,
a question about mirrors, it's not clear for me.
Is it possible to keep all the release's repositories (local and remote) in case of local's one failure ?
But it seems that pkg always use the official remote depot instead of the local one?
# pkg publisher
PUBLISHER TYPE STATUS URI
opensolaris.org (preferred) origin online http://ultra20:81/
opensolaris.org (preferred) mirror online http://pkg.opensolaris.org/
With this config, "pkg install openffice-fr" is slow as if there was no local mirror.
Posted by gerard on November 12, 2009 at 12:42 PM GMT #
Hi Gerard,
IPS always attempts to use the configured mirror first. You're mirror is configured to use the remote depot. Try switching them and you should notice the performance improvement.
Regards,
Brian
Posted by Brian Leonard on November 12, 2009 at 08:06 PM GMT #
Hi Brian,
Thanks for this information. It has been very helpful.
I am currently trying to set up an environment which has local repositories for both release and dev. Can this be done on the same system? I have copies of both repositories but not sure if/how to configure pkg/server to see them both.
Thanks,
Tyson
Posted by Tyson on November 13, 2009 at 05:07 PM GMT #
Hi Tyson,
it's not a clear answer, but an idea that we tested here at CHOSUG and is working fine, with Brian's procedure
- installing an IPS server as a mirror for /release in a zone, here called IPS
- installing an IPS server from the latest build full ISO image in a separate zone, here called IPS-DEV
We rsynced the mounted ISO directly from the global zone the zone IPS-DEV directory
Now, for each new full ISO repository for a new build, we redo the RSYNC, then restart the pkg server in the IPS-DEV
We then have two independant network services running at low workload on a single system :)
You effectively have now one IP address per IPS server but, our next step is to add a reverse proxy zone to map /dev to zone IPS-DEV, and /release to zone IPS.
Each zone is very light, and I think that it takes less time to do it this way than trying the find how to do it in another way ...
HTH
Karim
Posted by Karim on November 13, 2009 at 05:34 PM GMT #
Hi Karim,
You propose an interesting solution. I was going to suggest running another instance of the pkg depot server. I haven't tried that myself yet though.
/Brian
Posted by Brian Leonard on November 13, 2009 at 05:51 PM GMT #
Thanks Karim for your response. I was wondering if I would have to use zones or some other virtual system to add the second repository. After seeing Brian's post I decided to look more into running another instance of the pkg/server.
This is how I got it to work:
(http://www.sun.com/bigadmin/features/articles/smf_example.jsp help me with this.)
# cp /var/svc/manifest/application/pkg-server.xml pkg-server-dev.xml
Made the following changes to /var/svc/manifest/application/pkg-server-dev.xml (not sure if all these changes are necessary):
Changed
<service_bundle type='manifest' name=':pkg-server'>
To
<service_bundle type='manifest' name=':pkg-server-dev'>
Changed
<service
name='application/pkg/server'
type='service'
version='1'>
To
<service
name='application/pkg/server-dev'
type='service'
version='1'>
Changed
<propval name='port' type='count' value='80' />
To
<propval name='port' type='count' value='81' />
Changed
image packaging repository
To
image packaging development repository
# svccfg
svc:> validate /var/svc/manifest/application/pkg/server-dev.xml
svc:> import /var/svc/manifest/application/pkg/server-dev.xml
svc:> quit
# svcadm enable -t svc:/application/pkg/server-dev
# svcs -a | grep pkg/server
online 13:02:48 svc:/application/pkg/server:default
online 13:02:48 svc:/application/pkg/server-dev:default
Now it is just a matter of following Brian's procedure but using the pkg/server-dev for the development repostiory instead of pkg/server.
Posted by Tyson on November 17, 2009 at 09:25 PM GMT #