Updated script for fully local IPS mirror for faster DC and AI experimenting

Středa VII 15, 2009

Updated script for fully local IPS mirror of selected packages for faster OpenSolaris 200906 DC or AI install Prototyping.

This is updated IPS packages subset mirror script with support of OpenSoalris 200906 names with "/" like "system/ldoms/ldomsmanager" and also strip out most foreign arch data (For example SPARC on x86). I use this script to make a fast repeatable installations of already installed packages set of reduced OpenSolaris 200906. I create a fully local IPS repository on local virtual disk and I am installing from localhost, this fully eliminate any dependence on network reliability. I strip our opposite arch to save disk space, you can comment out this code, if you have enough disk space and you want to get 1:1 repo clone of selected packages. Provided script bellow is sample code and is valid only in context of current 200906 repository.

Good news is then in context of DEV (OpenSolaris 200906) we will not need this script any more, I will show it later.

See also OpenSolaris 200811 post: Home No-Network needed OpenSolaris Live-CD IPS REPO full mirror, about the origin of script and more info.

This script is sample of fully local (local disk, localhost net) one arch only repo clone.

cat repo-self.sh
#!/bin/sh

AUTHORITY=http://pkg.opensolaris.org:80/release

TARGET=/localrepos/ips/self

ISA_INFO=`/sbin/uname -p`

echo "INFO: We are on platform: $ISA_INFO"

# This is how I start the ips repository daemon after updating the mirror:
#
# Ideally create a separe zpool for repo
#
# 1. Create a dedicated zfs based system
# echo -y | format
# Searching for disks...done
#   0. c3t0d0 system
#   1. c3t1d0 instance data
#   2. c3t2d0 
#
# zpool create localrepos c3t2d0
# zfs create localrepos/ips
# zfs create localrepos/ips/self
#
# 2. Get repo clone from pre-prepared list
#
# rm -f /tmp/repo-self.log
# repo-self.sh ./OSOL0906-JeOS-pkgsfmris.lst 2>&1 | tee /tmp/repo-self.log
#
# 3. After clone is done, run this pkg.depmod comand to start repo:
#
# screen -L -d -m /usr/lib/pkg.depotd --rebuild -d /localrepos/ips/self -p 10000
#
# ifconfig -a to find global IP so you can check if repository FREEZE of selected packages is created correctly. 
#

# fail() { rm -rf ${TDIR}; exit 1; }

if [ ! -d ${TARGET} ]; then
    mkdir -p ${TARGET} 2>/dev/null
fi

TDIR=`mktemp -d`

if [ ! -d $1 ]; then
    echo "Passed file wilh pkg list of full FMRIs" $1
    cat $1 | grep -v "^#" |sed s%\pkg:/%%|sort -u >${TDIR}/packages.lst
else
    echo "Geting installed package list from local pkg database ..."
    pkg list -v | awk '{print $1}'|sed s%\pkg:/%%|sort -u | sed -e 's/\//%2f/g' >${TDIR}/packages.lst
fi

DIRS="catalog trans updatelog"
for d in $DIRS; do [ -d $d ] && rm -rf $d; done

FILES="search.dir search.pag"
for f in $FILES; do [ -f $f ] && rm -f $f; done

# I add there escaping for / as %2F for names like 'system/ldoms/ldomanager' packages

cat <${TDIR}/packages-download.sh
x() {
PKG_NAME=\`echo \$1|sed s/@.*//\| sed s/\//%2F/g`
PKG_VERS=\`echo \$1|sed s/.*@//|sed s/,/%2C/|sed s/:/%3A/\`
[ -d ${TARGET}/pkg/\$PKG_NAME ] || mkdir -p ${TARGET}/pkg/\$PKG_NAME
echo Downloading \$1
wget ${AUTHORITY}/manifest/0/\$1 -O ${TDIR}/package && mv ${TDIR}/package ${TARGET}/pkg/\$PKG_NAME/\$PKG_VERS
}

EOF
cat ${TDIR}/packages.lst |sed s/\^/x\ / >>${TDIR}/packages-download.sh
sh ${TDIR}/packages-download.sh

echo Generating list of files referenced by the packages...
# For now we exclude binaly files for oposite achitecture i386 and SPARC to save space
# Other stuff like Scripts or License have neglible size and they don't need to be ecluded
#
#

if [ $ISA_INFO = "sparc" ] ; then
 for i in `ls ${TARGET}/pkg`; do
    cat ${TARGET}/pkg/$i/*
 done | grep -v "elfarch=i386"| egrep "^(file|license)\ "|awk '{ print $2 }'|sort -u >${TDIR}/files-to-download.lst
else
 for i in `ls ${TARGET}/pkg`; do
    cat ${TARGET}/pkg/$i/*
 done | grep -v "elfarch=sparc"| egrep "^(file|license)\ "|awk '{ print $2 }'|sort -u >${TDIR}/files-to-download.lst
fi

mkdir ${TDIR}/files.d
split -l 100 -a 8 ${TDIR}/files-to-download.lst ${TDIR}/files.d/files-

# I'm using bash because of the "echo -n" bit

cat <${TDIR}/files-download.sh

for list in \`ls ${TDIR}/files.d\`; do
    POST_TMP=\`cat ${TDIR}/files.d/\${list}|nl -s\=|sed s/\^\ */File-Name-/\`
    POST=\`for i in \$POST_TMP; do echo -n \&\$i; done\`
    echo Downloading...
    wget  --post-data="\$POST" ${AUTHORITY}/filelist/0 -O ${TDIR}/files.d/\${list}.tar || fail
    echo Extracting...
    mkdir ${TDIR}/files.d/\${list}.d
    (cd ${TDIR}/files.d/\${list}.d; tar xf ${TDIR}/files.d/\${list}.tar)
    for file in \`ls ${TDIR}/files.d/\${list}.d\`; do
        FILE_DIRL1=\`echo \$file|cut -c -2\`
        FILE_DIRL2=\`echo \$file|cut -c 3-|cut -c -6\`
        mkdir -p ${TARGET}/file/\${FILE_DIRL1}/\${FILE_DIRL2} 2>/dev/null && mv \
            ${TDIR}/files.d/\${list}.d/\$file ${TARGET}/file/\${FILE_DIRL1}/\${FILE_DIRL2} || exit 1
    done
    rmdir ${TDIR}/files.d/\${list}.d
    rm ${TDIR}/files.d/\${list}.tar
    rm ${TDIR}/files.d/\${list}
done
EOF

bash ${TDIR}/files-download.sh

#rm -rf ${TDIR}
exit 0

Like this post? | | | |

Posted at 01:50odp. VII 15, 2009 by natiku in Virtualization
Tags: cloud jeos opensolaris virtualbox virtualization vmware