VirtualGuru

The Home of Virtualization Workshops

Give a man a fish; you have fed him for today.        
Teach a man to fish; you have fed him for a lifetime.

Get & Inst: Net based install of IPS packages into OpenSolaris 2008.05

Čtvrtek V 22, 2008

In fact this an another example how composite image can be created, this time I will use IPS net based install from default OpenSolaris repository.

This entry is part of 'OpenSolaris on Amazon EC2' workshop

New IPS packaging system

OpenSolaris use new packaging system called IPS based on network based repositories, and ZFS for rollback capabilities, see more about IPS in:

OpenSolaris Project: Image Packaging System
Image Packaging System Documentation
Package Management Task Tables

If you are familiar with rpm +yum or deb + apt-get you will love our new IPS based system.

Interface for new IPS format is managed by new pkg command: 

pkg help

Usage:
        pkg [options] command [cmd_options] [operands]

Basic subcommands:
        pkg install [-nvq] package...
        pkg uninstall [-nrvq] package...
        pkg list [-aHsuv] [package...]
        pkg image-update [-nvq]
        pkg refresh [--full]
        pkg version
        pkg help

Advanced subcommands:
        pkg info [-lr] [--license] [pkg_fmri_pattern ...]
        pkg search [-lr] [-s server] token
        pkg verify [-fHqv] [pkg_fmri_pattern ...]
        pkg contents [-Hmr] [-o attribute ...] [-s sort_key] [-t action_type ... ]
            pkg_fmri_pattern [...]
        pkg image-create [-FPUz] [--full|--partial|--user] [--zone]
            [-k ssl_key] [-c ssl_cert] -a <prefix>=<url> dir

        pkg set-authority [-P] [-k ssl_key] [-c ssl_cert]
            [-O origin_url] authority
        pkg unset-authority authority ...
        pkg authority [-H] [authname]

Options:
        -R dir

Environment:
        PKG_IMAGE

See mote in man pkg

Prepare Boot Environment

1. Delete shapshots

Boot Environment is feature based on ZFS root and its utilize ZFS snapshots

Before we will install we need to be sure then there is no any shapshots in system, if they present they will log FS changes and make our AMI image grow. 

zfs list
NAME                         USED  AVAIL  REFER  MOUNTPOINT
rpool                        838M  7.80G    55K  /rpool
rpool/ROOT                   837M  7.80G    18K  /rpool/ROOT
rpool/ROOT/opensolaris       837M  7.80G   810M  legacy
rpool/ROOT/opensolaris/opt  27.0M  7.80G  27.0M  /opt
rpool/export                  37K  7.80G    19K  /export
rpool/export/home             18K  7.80G    18K  /export/home

ZFS snapshot of boot environment look like this 

rpool/ROOT/opensolaris@install          0      -  1.50G  -

You can delete snapshots with beadm command or manually 

zfs destroy rpool/ROOT/opensolaris@install

Don't worry we will recreate snapshots after installation before we will (re)bundle AMI

Notes:

*Download sizes bellow are for OpenSolaris JeOS, if you will use full OpenSolaris 2008.05 image download sizes can be smaller because some componets like picture formats LIBs will be in system allready.

  • Installation of Glassfish and WebStack (amp-dev) are in os2008.05 Getting started guide
  • Repositories outputs are form opensolaris.org and sunfreeware IPS repositories on (01/May/08)

Check repository setup

1. Check networking setup and main IPS repository status 

nslookup pkg.sunfreeware.org
Server:         172.16.0.23
Address:        172.16.0.23#53

Non-authoritative answer:
Name:   pkg.sunfreeware.org
Address: 209.62.20.192

Check then main IPS repsorty is accessible 

curl -s  http://pkg.opensolaris.org/status | egrep "server |Number of"
   <h1><img src="/logo" /> <code>pkg</code> server ok</h1>
Number of packages: 4464
Number of in-flight transactions: 0
Number of catalogs served: 5132
Number of manifests served: 922398
Number of files served: 23616
Number of flists requested: 726913
Number of files served by flist: 35353905
Number of packages renamed: 0

2. Check net based IPS repository setup and refresh repository cache 

pkg authority -H
opensolaris.org (preferred)         http://pkg.opensolaris.org:80/
sunfreeware                         http://pkg.sunfreeware.com:9000/

If you don't see sunfreeware repository you can add it with:

pkg set-authority -O http://pkg.sunfreeware.com:9000 sunfreeware

pkg refresh --full

Install Glassfish and MySQL as packages selection

1. Check available MySQL packages

Check with pkg list remote search -a and names only -s 

pkg list -a -s | grep -i mysql | uniq

SUNWmysql                      mysql - MySQL Database Management System (usr component)
SUNWmysql-base                 MySQL Database Management System (Base)
SUNWmysql-python               A MySQL database adapter for the Python programming language
SUNWmysql5                     MySQL 5.0.45 - MySQL Database Management System (usr component)
SUNWmysql5test                 MySQL 5.0.45 - MySQL Database Management System (test component)
SUNWmysqlt                     mysql - MySQL Database Management System (test component)
SUNWphp524-mysql               PHP5 module for MySQL
IPSFWmphp (sunfreeware)        php - Scripting Lang. Tool for web pages (MySQL API)
IPSFWmyjdb (sunfreeware)       myjdb - MySQL Connector/J

Check with pkg list remote search -a with default versions output 

pkg list -a | grep -i mysql
SUNWmysql                                     4.0.24-0.75     known      ----
SUNWmysql                                     4.0.24-0.79     known      ----
SUNWmysql                                     4.0.24-0.86     known      ----
SUNWmysql-base                                0.5.11-0.86     known      ----
SUNWmysql-python                              0.5.11-0.86     known      ----
SUNWmysql5                                    5.0.45-0.79     known      ----
SUNWmysql5                                    5.0.45-0.86     known      ----
SUNWmysql5test                                5.0.45-0.79     known      ----
SUNWmysql5test                                5.0.45-0.86     known      ----
SUNWmysqlt                                    4.0.24-0.75     known      ----
SUNWmysqlt                                    4.0.24-0.79     known      ----
SUNWmysqlt                                    4.0.24-0.86     known      ----
SUNWphp524-mysql                              5.2.4-0.79      known      ----
SUNWphp524-mysql                              5.2.4-0.86      known      ----

Do similar for Glassfish 

pkg list -a -s | grep -i glassfish | uniq
glassfishv2                    GlassFish Java EE 5 Application Server

2. Install Glassfish and MySQL

First do dry-run to see what packages will be automatically selected by dependencies 

pkg install -n -v glassfishv2 IPSFWmyjdb SUNWmysql5
Before evaluation:
UNEVALUATED:
+pkg:/glassfishv2@0.5.11,5.11-0.86:20080428T163323Z
+pkg://sunfreeware/IPSFWmyjdb@0.5.11,5.11-5.7:20080507T040806Z
+pkg:/SUNWmysql5@5.0.45,5.11-0.86:20080426T173344Z

After evaluation:
None -> pkg:/glassfishv2@0.5.11,5.11-0.86:20080428T163323Z
None -> pkg://sunfreeware/IPSFWmyjdb@0.5.11,5.11-5.7:20080507T040806Z
None -> pkg:/SUNWmysql5@5.0.45,5.11-0.86:20080426T173344Z
None -> pkg:/SUNWj6dmo@0.5.11,5.11-0.86:20080426T175002Z
None -> pkg:/SUNWj6cfg@0.5.11,5.11-0.86:20080426T174949Z
None -> pkg:/mq41@4.1,5.11-0.86:20080428T163407Z
None -> pkg:/SUNWj6man@0.5.11,5.11-0.86:20080426T175029Z
None -> pkg:/SUNWant@1.6.5,5.11-0.86:20080426T172852Z
None -> pkg:/SUNWj6dvx@0.5.11,5.11-0.86:20080426T174932Z
None -> pkg:/SUNWj6rtx@0.5.11,5.11-0.86:20080426T182209Z
None -> pkg:/SUNWj6dmx@0.5.11,5.11-0.86:20080426T175002Z
None -> pkg:/SUNWj6jmp@0.5.11,5.11-0.86:20080426T181701Z
None -> pkg:/SUNWj6dev@0.5.11,5.11-0.86:20080426T174949Z
None -> pkg:/SUNWjavadb@0.5.11,5.11-0.86:20080426T175030Z
None -> pkg:/SUNWmysql-base@0.5.11,5.11-0.86:20080426T173338Z
None

OK install will also select JDK6, MQ and Java DB packages

Process install of Glassfish and MySQL 

pkg install glassfishv2 IPSFWmyjdb SUNWmysql5
DOWNLOAD                                    PKGS       FILES     XFER (MB)
Completed                                  15/15   6848/6848 366.54/366.54

PHASE                                        ACTIONS
Update Phase                                     2/2
Install Phase                              7922/7922

3. After installation we can check whats get installed

Check versions 

pkg list  | grep -i mysql
SUNWmysql-base                                0.5.11-0.86     installed  ----
SUNWmysql5                                    5.0.45-0.86     installed  ----

pkg list -a | grep -i mysql | uniq
SUNWmysql                                     4.0.24-0.75     known      ----
SUNWmysql                                     4.0.24-0.79     known      ----
SUNWmysql                                     4.0.24-0.86     known      ----
SUNWmysql-base                                0.5.11-0.86     known      ----
SUNWmysql-base                                0.5.11-0.86     installed  ----
SUNWmysql5                                    5.0.45-0.79     known      ----
SUNWmysql5                                    5.0.45-0.86     known      ----
SUNWmysql5                                    5.0.45-0.86     installed  ----
SUNWmysql5test                                5.0.45-0.79     known      ----
SUNWmysql5test                                5.0.45-0.86     known      ----

We can also use pkg contents, command similar to rpm --querystring to list installed packages database

Example to find file location 

pkg contents  -t file -o "pkg.name,path,pkg.size" IPSFWmyjdb | grep mysql-connector
IPSFWmyjdb opt/sfw/mysql/share/jdbc/mysql-connector-java-3.0.8-stable-bin.jar                                  213820

Example to count sizes of installed packages 

pkg contents  -t file -o "pkg.name,path,pkg.size" SUNWmysql5 SUNWmysql-base IPSFWmyjdb \
 | awk '{ SUM += $3} END { print SUM/1024/1024 "MB"}'
123.987MB

pkg contents  -t file -o "pkg.name,path,pkg.size" glassfishv2 SUNWjavadd mq41 \
 | awk '{ SUM += $3} END { print SUM/1024/1024 "MB"}'
89.6158MB

pkg contents  -t file -o "pkg.name,path,pkg.size" SUNWj6cfg SUNWj6dev SUNWj6dmo \
 SUNWj6dmx SUNWj6dvx SUNWj6jmp SUNWj6man SUNWj6rt SUNWj6rtx \
 | awk '{ SUM += $3} END { print SUM/1024/1024 "MB"}'
180.54MB

Install WebStack AMP (Apache, MySQL, PHP) as cluster bundle

WebStack AMP (Apache, MySQL, PHP) is prepared by amp and amp-dev clusters 

pkg list -a -s| grep -i amp
amp                            Cluster containing an AMP (Apache, MySQL, PHP) deployment kit
amp-dev                        AMP Development cluster


pkg list -a | grep -i amp
amp                                           0.5.11-0.86     known      ----
amp-dev                                       0.5.11-0.86     known      ----

amp-dev is amp (212MB )with netbeans development GUI (602MB), on server you will probally like just amp

1. List what will be installed with dry-run

List context of amp and amp-dev clusters 

pkg install -n -v amp
Before evaluation:
UNEVALUATED:
+pkg:/amp@0.5.11,5.11-0.86:20080504T074642Z

After evaluation:
None -> pkg:/amp@0.5.11,5.11-0.86:20080504T074642Z
None -> pkg:/SUNWapch22m-security@2.1.5,5.11-0.86:20080426T172939Z
None -> pkg:/SUNWmysql5@5.0.45,5.11-0.86:20080426T173344Z
None -> pkg:/SUNWmysql-base@0.5.11,5.11-0.86:20080426T173338Z
None -> pkg:/SUNWapch22m-dtrace@0.3.1,5.11-0.86:20080426T172926Z
None -> pkg:/SUNWapch22@2.2.8,5.11-0.86:20080426T172911Z
None -> pkg:/SUNWapch22m-php5@2.2.6,5.11-0.86:20080426T172938Z
None -> pkg:/SUNWphp524man@5.2.4,5.11-0.86:20080426T175919Z
None -> pkg:/SUNWphp524@5.2.4,5.11-0.86:20080426T175747Z
None -> pkg:/SUNWphp524core@5.2.4,5.11-0.86:20080426T175914Z
None -> pkg:/SUNWpng@0.5.11,5.11-0.86:20080426T175945Z
None -> pkg:/SUNWjpg@0.5.11,5.11-0.86:20080426T175051Z
None -> pkg:/SUNWfreetype2@2.3.5,5.11-0.86:20080426T173619Z
None -> pkg:/SUNWtidy@1.0.0,5.11-0.86:20080426T180509Z
None -> pkg:/SUNWapch22m-fcgid@2.2,5.11-0.86:20080426T172937Z
None -> pkg:/SUNWphp524-mysql@5.2.4,5.11-0.86:20080426T175753Z
None


pkg install -n -v amp-dev
Before evaluation:
UNEVALUATED:
+pkg:/amp-dev@0.5.11,5.11-0.86:20080504T074641Z

After evaluation:
None -> pkg:/amp-dev@0.5.11,5.11-0.86:20080504T074641Z
None -> pkg:/SUNWsvn@1.4.3,5.11-0.86:20080426T175941Z
None -> pkg:/SUNWapch22@2.2.8,5.11-0.86:20080426T172911Z
None -> pkg:/SUNWneon@0.25.5,5.11-0.86:20080426T175632Z
None -> pkg:/SUNWmysql5@5.0.45,5.11-0.86:20080426T173344Z
None -> pkg:/SUNWmysql-base@0.5.11,5.11-0.86:20080426T173338Z
None -> pkg:/netbeans@6.0.1,5.11-0.86:20080430T212951Z
None -> pkg:/SUNWlibnb-visualweb1@6.0.1,5.11-0.86:20080430T213136Z
None -> pkg:/SUNWlibnb-ide8@6.0.1,5.11-0.86:20080430T213033Z
None -> pkg:/SUNWnetbeans-ide@6.0.1,5.11-0.86:20080430T212952Z
None -> pkg:/SUNWnetbeans-desktop@6.0.1,5.11-0.86:20080430T212954Z
None -> pkg:/SUNWlibnb-platform7@6.0.1,5.11-0.86:20080430T213048Z
None -> pkg:/SUNWlibnb-java1@6.0.1,5.11-0.86:20080430T213019Z
None -> pkg:/SUNWlibnb-enterprise4@6.0.1,5.11-0.86:20080430T213001Z
None -> pkg:/SUNWlibnb-cnd1@6.0.1,5.11-0.86:20080430T212957Z
None -> pkg:/SUNWnetbeans-javase@6.0.1,5.11-0.86:20080430T212955Z
None -> pkg:/SUNWlibnb-profiler2@6.0.1,5.11-0.86:20080430T213055Z
None -> pkg:/SUNWlibnb-apisupport1@6.0.1,5.11-0.86:20080430T212955Z
None -> pkg:/SUNWnetbeans-javaee@6.0.1,5.11-0.86:20080430T212954Z
None -> pkg:/SUNWlibnb-xml1@6.0.1,5.11-0.86:20080430T213155Z
None -> pkg:/SUNWnetbeans-cpp@6.0.1,5.11-0.86:20080430T212955Z
None -> pkg:/SUNWlibnb-ruby1@6.0.1,5.11-0.86:20080430T213059Z
None -> pkg:/SUNWnetbeans-ruby@6.0.1,5.11-0.86:20080430T212955Z
None -> pkg:/SUNWmercurial@0.9.5,5.11-0.86:20080426T173239Z
None -> pkg:/SUNWphp524doc@5.2.4,5.11-0.86:20080426T175754Z
None -> pkg:/SUNWapch22m-jk@1.2.25,5.11-0.86:20080426T172937Z
None -> pkg:/SUNWapch22m-security@2.1.5,5.11-0.86:20080426T172939Z
None -> pkg:/SUNWapch22d@2.2.8,5.11-0.86:20080426T172927Z
None -> pkg:/SUNWcvs@1.12.13,5.11-0.86:20080426T173148Z
None -> pkg:/SUNWphp524man@5.2.4,5.11-0.86:20080426T175919Z
None -> pkg:/SUNWtcat@5.5.26,5.11-0.86:20080426T180346Z
None -> pkg:/SUNWsquid@2.6.17,5.11-0.86:20080426T180329Z
None -> pkg:/SUNWpython-twisted@0.5.11,5.11-0.86:20080426T180103Z
None -> pkg:/SUNWapch22m-php5@2.2.6,5.11-0.86:20080426T172938Z
None -> pkg:/SUNWphp524@5.2.4,5.11-0.86:20080426T175747Z
None -> pkg:/SUNWphp524core@5.2.4,5.11-0.86:20080426T175914Z
None -> pkg:/SUNWpng@0.5.11,5.11-0.86:20080426T175945Z
None -> pkg:/SUNWjpg@0.5.11,5.11-0.86:20080426T175051Z
None -> pkg:/SUNWfreetype2@2.3.5,5.11-0.86:20080426T173619Z
None -> pkg:/SUNWtidy@1.0.0,5.11-0.86:20080426T180509Z
None -> pkg:/SUNWapch22m-fcgid@2.2,5.11-0.86:20080426T172937Z
None -> pkg:/SUNWmemcached@1.2.2,5.11-0.86:20080426T173238Z
None -> pkg:/SUNWlibevent@1.3.5,5.11-0.86:20080426T175122Z
None -> pkg:/SUNWapch22m-dtrace@0.3.1,5.11-0.86:20080426T172926Z
None -> pkg:/webstackui@0.5.11,5.11-0.86:20080428T164650Z
None -> pkg:/SUNWphp524-pgsql@5.2.4,5.11-0.86:20080426T175754Z
None -> pkg:/SUNWpostgr-82-libs@8.2.6,5.11-0.86:20080426T180011Z
None -> pkg:/SUNWphp524-mysql@5.2.4,5.11-0.86:20080426T175753Z
None

2. Lets install one of them 

 pkg install amp
DOWNLOAD                                    PKGS       FILES     XFER (MB)
Completed                                  16/16   1729/1729 212.86/212.86

PHASE                                        ACTIONS
Update Phase                                     2/2
Install Phase                              2225/2225

pkg install amp-dev
DOWNLOAD                                    PKGS       FILES     XFER (MB)
Completed                                  48/48 20429/20429 602.10/602.10

PHASE                                        ACTIONS
Update Phase                                     2/2
Install Phase                            22696/22696

Finalize Boot Environment

1. Clean unused disk space fro better compression 

dd if=/dev/zero of=/zero.zer bs=10000000; rm -f /zero.zer

2. Recreate an install snapshots 

zfs snapshot rpool@install
zfs snapshot rpool/ROOT@install
zfs snapshot rpool/ROOT/opensolaris@install
zfs snapshot rpool/ROOT/opensolaris/opt@install
zfs snapshot rpool/export@install
zfs snapshot rpool/export/home@install

zfs list
NAME                                 USED  AVAIL  REFER  MOUNTPOINT
rpool                               1.52G  7.09G    55K  /rpool
rpool@install                           0      -    55K  -
rpool/ROOT                          1.52G  7.09G    18K  /rpool/ROOT
rpool/ROOT@install                      0      -    18K  -
rpool/ROOT/opensolaris              1.52G  7.09G  1.50G  legacy
rpool/ROOT/opensolaris@install          0      -  1.50G  -
rpool/ROOT/opensolaris/opt          26.7M  7.09G  26.7M  /opt
rpool/ROOT/opensolaris/opt@install      0      -  26.7M  -
rpool/export                          37K  7.09G    19K  /export
rpool/export@install                    0      -    19K  -
rpool/export/home                     18K  7.09G    18K  /export/home
rpool/export/home@install               0      -    18K  -

Now, you are ready for (re)bundling

[2] Comments
Like this post? del.icio.us | furl | slashdot | technorati | digg
Posted at 02:28odp. V 22, 2008 by natiku in Virtualization
Tags:
Comments:

For the "find mysql-connector" case,

$ pkg search -r mysql-connector

will be faster. Omit -r to search on the local system.

-- Stephen

Posted by Stephen on květen 23, 2008 at 05:47 dop. CEST #

Great article.
I've run into one problem with the package installer.
One you run

pkg uninstall foo

It doesn't like you trying to run

pkg install foo

I.e.once you remove something for whatever reason, it is impossible to add it back again.

Do you know how to do this?

Thanks

ET

Posted by EvilTed on únor 25, 2009 at 01:39 dop. CET #

Post a Comment:
  • HTML Syntax: NOT allowed