ALERT: Amazon just informed that the Pinotage endpoint server will be down for scheduled maintenance on Wednesday starting at 2AM PDT. The maintenance is scheduled to last for 2 - 3hours. Sorry for any inconvenience this may cause.
ALERT: Amazon just informed that the Pinotage endpoint server will be down for scheduled maintenance on Wednesday starting at 2AM PDT. The maintenance is scheduled to last for 2 - 3hours. Sorry for any inconvenience this may cause.
| -d user_data |
Data to make available to the
instances. This data is read from the command line of the USER_DATA
argument. If you want the data to be read from a file, see the -f
option. Example: -d "my user data" |
| -f user_data_file |
Data to make available to these
instances. The data is read from the file specified by FILE_NAME. To
specify user data on the command line, use the -d option. Example: -f data.zip |
| bash# ec2-run-instances -k
<my-key-pair> -t m1.small ami-eb7a9f82 bash# ec2-describe-instances | grep INSTANCE | cut -f2,3,4,6 i-196fb970 ami-eb7a9f82 ec2-75-101-225-153.compute-1.amazonaws-DOT-com running bash# ssh -i <my-key-pair-file> root-AT-ec2-75-101-225-153.compute-1.amazonaws.com Last login: Mon Apr 28 02:24:21 2008 Sun Microsystems Inc. SunOS 5.11 snv_79 January 2008 # bash -o vi |
| #!/bin/sh ec2autorundir="/var/ec2" ec2autorunfile="ec2autorun.input" ec2autorunscript="ec2autorun.script" ec2userdataurl="http://169.254.169.254/latest/user-data" # we password protect the user data file since any user can retrieve. # we need to remember the password as we will need to use when # creating the user data zip file. This password can be anything # and is not the same as your AWS secret key. ec2password="put your password here" # we usually pass the user data as a ZIP archive file, but we # can also simply pass a plain text script file. zipfiletype="ZIP archive" case "$1" in 'start') # create directory for autorun scripts and data if [ ! -d ${ec2autorundir} ] then /usr/bin/mkdir -p ${ec2autorundir} else # if the directory already exists we do not run the script again echo "`date`: ec2autorun script already ran, exiting." \ >> ${ec2autorundir}/${ec2autorunfile}.stdout exit 0 fi cd ${ec2autorundir} # get the user data file passed to the instance when created /usr/sfw/bin/wget --quiet \ --output-document=${ec2autorundir}/${ec2autorunfile} \ ${ec2userdataurl} # test the file size, if zero then no user data was provided # when instance was started if [ ! -s ${ec2autorundir}/${ec2autorunfile} ] then echo "User data was not passed to instance." \ >> ${ec2autorunfile}.stdout exit 0 fi # if the file is a zip file, unzip it and then run the # script extracted from the zip file, else run the file # assuming it is a script filetype=`file ${ec2autorundir}/${ec2autorunfile} | /usr/bin/cut -f2` if [ "${filetype}" = "${zipfiletype}" ] then unzip -P ${ec2password} ${ec2autorunfile} \ >> ${ec2autorunfile}.stdout 2>>${ec2autorunfile}.stderr bash ./${ec2autorunscript} \ >> ${ec2autorunscript}.stdout 2>>${ec2autorunscript}.stderr else bash ./${ec2autorunfile} \ >> ${ec2autorunfile}.stdout 2>>${ec2autorunfile}.stderr fi # set the autorun directory to be read only by root chmod 700 ${ec2autorundir} ;; *) echo "Usage: $0 { start }" ;; esac exit 0 |
| bash# chmod 700
/etc/init.d/ec2autorun bash# ln /etc/init.d/ec2autorun /etc/rc3.d/S10ec2autorun |
| bash# mkdir -p
/usr/local/aws bash# cd /usr/local/aws bash# /usr/sfw/bin/wget \ http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz bash# gzcat s3sync.tar.gz | tar xf - bash# cd /usr/local/aws/s3sync bash# ln -s s3cmd.rb s3cmd bash# export PATH=$PATH:/usr/local/aws/s3sync bash# rm /usr/local/aws/s3sync.tar.gz |
| #!/usr/bin/bash zpool="s3pool" zpooldisk="c0d1p0" zfssnapshots="home share" snapshotsdir="/var/tmp/snapshots" snapshotsbucket="<my-bucket-name-for-zfs-snapshots>" s3toolspath="/usr/local/aws/s3sync" ntpdate 0.north-america.pool.ntp.org # create the ZFS pool /usr/sbin/zpool create ${zpool} ${zpooldisk} # download the ZFS snapshots and restore the file systems if ! [[ -d ${snapshotsdir} ]] then /usr/bin/mkdir -p ${snapshotsdir} fi # setup the environment for S3 export PATH=$PATH:${s3toolspath} export AWS_ACCESS_KEY_ID=<my-aws-access-key> export AWS_SECRET_ACCESS_KEY=<my-aws-secret-access-key> # download the saved ZFS file systems and restore builtin cd ${snapshotsdir} for i in ${zfssnapshots} do s3cmd get ${snapshotsbucket}:s3-${i}.bz2 s3-${i}.bz2 done # restore the snapshots and share for NFS for i in ${zfssnapshots} do bzcat s3-${i}.bz2 | zfs recv -F -d ${zpool} zfs set sharenfs=on ${zpool}/${i} done # start the NFS services for this instance svcadm enable -r svc:/network/nfs/server:default |
| bash# zip -P "put your
password here" ec2autorun.zip ec2autorun.script |
| bash# ec2-run-instances -k <my-key-pair> -f
ec2autorun.zip ami-93b652fa RESERVATION r-5609c43f 578281684738 default INSTANCE i-3f994856 ami-93b652fa pending <my-key-pair> 0 m1.small 2008-07-01T02:55:46+0000 us-east-1b aki-b57b9edc ari-b47b9edd |
| bash# ec2-describe-instances | grep i-3f994856 |
cut -f2,5 i-3f994856 ip-10-251-203-4.ec2.internal |
| #!/usr/bin/bash # NFS server private host name. This name is also referred to in # EC2 as the Private DNS Name. nfsserver="ip-10-251-203-4.ec2.internal" # start the services for this instance svcadm enable -r svc:/network/nfs/client:default # mount the NFS file systems from the NFS server mkdir -p /s3pool/home mkdir -p /s3pool/share mount -F nfs ${nfsserver}:/s3pool/home /s3pool/home mount -F nfs ${nfsserver}:/s3pool/share /s3pool/share # add to vfstab file echo "${nfsserver}:/s3pool/home - /s3pool/home nfs - yes rw,xattr" >> /etc/vfstab echo "${nfsserver}:/s3pool/share - /s3pool/share nfs - yes rw,xattr" >> /etc/vfstab |
| bash# zip -P "put your
password here" ec2autorun.zip ec2autorun.script bash# ec2-run-instances -k <my-key-pair> -f ec2autorun.zip ami-93b652fa |
| bash# cd /var/ec2 bash# ls # the zip file that was passed on the ec2-run-instance # command line ec2autorun.input # stderr from the /etc/init.d/ec2autorun script ec2autorun.input.stderr # stdout from the /etc/init.d/ec2autorun script ec2autorun.input.stdout # the script that was bundled in the zip file, # called by /etc/init.ec2atorun ec2autorun.script # stderr from ec2autorun.script ec2autorun.script.stderr # stdout from ec2autorun.script ec2autorun.script.stdout |
Saving and Restoring ZFS Snapshots to and from Amazon S3
by Sean O'Dell
| bash# ec2-run-instances -k
<my-key-pair> -t m1.small ami-eb7a9f82 bash# ec2-describe-instances | grep INSTANCE | cut -f2,3,4,6 i-196fb970 ami-eb7a9f82 ec2-75-101-225-153.compute-1.amazonaws-DOT-com running bash# ssh -i <my-key-pair-file> root-AT-ec2-75-101-225-153.compute-1.amazonaws.com Last login: Mon Apr 28 02:24:21 2008 Sun Microsystems Inc. SunOS 5.11 snv_79 January 2008 # bash -o vi |
| bash# mkdir -p
/usr/local/aws bash# cd /usr/local/aws bash# /usr/sfw/bin/wget \ http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz bash# gzcat s3sync.tar.gz | tar xf - bash# cd /usr/local/aws/s3sync bash# ln -s s3cmd.rb s3cmd bash# export PATH=$PATH:/usr/local/aws/s3sync |
| bash# which ruby no ruby in /usr/sbin /usr/bin /usr/local/aws/s3sync bash# pkg install SUNWruby18 bash# which ruby /usr/bin/ruby |
| bash# ntpdate
0.north-america.pool.ntp.org bash# export AWS_ACCESS_KEY_ID=<my-aws-access-key> bash# export AWS_SECRET_ACCESS_KEY=<my-aws-secret-access-key> bash# s3cmd createbucket <my-bucket-name-for-zfs-snapshots> |
| bash# zpool create s3pool
c0d1p0 bash# zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT s3pool 149G 111K 149G 0% ONLINE - bash# for i in home mysqldb share www do zfs create s3pool/$i done bash# zfs list -r s3pool NAME USED AVAIL REFER MOUNTPOINT s3pool 210K 147G 23K /s3pool s3pool/home 18K 147G 18K /s3pool/home s3pool/mysqldb 18K 147G 18K /s3pool/mysqldb s3pool/share 18K 147G 18K /s3pool/share s3pool/www 18K 147G 18K /s3pool/www bash# ls /s3pool home mysqldb share www |
| bash# zfs list -r s3pool NAME USED AVAIL REFER MOUNTPOINT s3pool 2.11G 13.5G 23K /s3pool s3pool/home 117M 13.5G 117M /s3pool/home s3pool/mysqldb 20.8M 13.5G 20.8M /s3pool/mysqldb s3pool/share 7.90M 13.5G 7.90M /s3pool/share s3pool/www 609K 13.5G 609K /s3pool/www |
Create a snapshot for each of the
filesystems.
bash# export
snapshotdate=`date '+%Y%m%d-%H%M%S'`
bash# export poolname=s3pool
bash# for i in home mysqldb share www
do
zfs snapshot
-r ${poolname}/${i}@s3-${i}-$snapshotdate
done
bash# zfs list -t snapshot -r s3pool
NAME
USED AVAIL REFER MOUNTPOINT
s3pool/home@s3-home-20080629-223417
0 - 117M -
s3pool/mysqldb@s3-mysqldb-20080629-223417
0 - 20.8M -
s3pool/share@s3-share-20080629-223417
0 - 7.90M -
s3pool/www@s3-www-20080629-223417
0 - 609K -
For each ZFS snapshot, save to a stream file.
bash# mkdir
/var/tmp/snapshots
bash# for i in home mysqldb share www
do
zfs send -R
${poolname}/${i}@s3-${i}-$snapshotdate | bzip2 \
> /var/tmp/snapshots/s3-${i}.bz2
done
bash# cd /var/tmp/snapshots
bash# ls
s3-home.bz2 s3-mysqldb.bz2
s3-share.bz2 s3-www.bz2
Upload the ZFS snapshot streams to S3.
bash# cd /var/tmp/snapshots
bash# export
s3bucketname=<my-bucket-name-for-zfs-snapshots>
bash# for i in `ls`
do
s3cmd put
${s3bucketname}:$i $i
done
bash# s3cmd list ${s3bucketname}
--------------------
s3-home.bz2
s3-mysqldb.bz2
s3-share.bz2
s3-www.bz2
bash# rm /var/tmp/snapshots/*.bz2
Restoring ZFS Snapshots
from S3
On our destination EC2 instance we
can restore the ZFS Snapshots previously saved.
Create the new ZFS pool.
bash# zpool create s3pool
c0d1p0
bash# zpool list
NAME SIZE USED
AVAIL CAP HEALTH ALTROOT
s3pool 149G 94K
149G 0% ONLINE -
Download
the ZFS Snapshots from S3.
bash# mkdir
/var/tmp/snapshots
bash# cd /var/tmp/snapshots
bash# export
s3bucketname=<my-bucket-name-for-zfs-snapshots>
bash# for i in `s3cmd list
${s3bucketname}`
do
if ! [[ $i ==
"--------------------" ]]
then
s3cmd get ${s3bucketname}:$i $i
fi
done
bash# ls
s3-home.bz2 s3-mysqldb.bz2
s3-share.bz2 s3-www.bz2
Restore
the ZFS Snapshots.
bash# for i in `ls`
do
bzcat $i |
zfs recv -F -d s3pool
done
bash# zfs list -r s3pool
NAME
USED AVAIL REFER MOUNTPOINT
s3pool
146M 147G 23K /s3pool
s3pool/home
117M 147G 117M /s3pool/home
s3pool/home@s3-home-20080629-223417
0 - 117M -
s3pool/mysqldb
20.8M 147G 20.8M /s3pool/mysqldb
s3pool/mysqldb@s3-mysqldb-20080629-223417
0 - 20.8M -
s3pool/share
7.90M 147G 7.90M /s3pool/share
s3pool/share@s3-share-20080629-223417
0 - 7.90M -
s3pool/www
609K 147G 609K /s3pool/www
s3pool/www@s3-www-20080629-223417
0 - 609K -
References
Solaris ZFS
Administration Guide
S3 Sync - free and open
source interfaces to the Amazon S3 system
In some of the Solaris Build 79 based images it may be possible that it doesn't have AMI/API tool bundled.
If the Solaris Build 79 image you are using does not have the Sun
AMI/API tools installed. You can follow the steps below to load/install the
tools on your instance.
The archive of the current tools can be retrieved with
the following command:
# curl -O
http://s3.amazonaws.com/sun-osol-tools/sun-amiapi-tools-latest.cpio
Once the archive is loaded, the checksum can be easily checked to make sure that you have the right file:
# md5sum sun-amiapi-tools-latest.cpio
1de19f491d32a2fe23b9a0515ec025dd sun-amiapi-tools-latest.cpio
Then extract the archive as shown below. Full paths are used in the archive
so the tools are always extracted in the same location.
# cpio -icvdum < sun-amiapi-tools-latest.cpio
Once the tools are installed, rebundling can be done as documented in the Sun Amazon
EC2 Getting Started Guide.
http://www.sun.com/third-party/global/amazon/Sun_AmazonEC2_GettingStarted050208.pdf
Note that the guide references mounting disks at /mnt but that is
completed automatically in certain AMI's.
The following command can be run to see if there is a preconfigured
filesystem at /mnt.
# df -kl /mnt
For OpenSolaris 2008.05 (
ami-0c41a465):
Q: What additional software is available after the installation of
Solaris 2008.05
A: The default installation will point toward the opensolaris.org
package authority. There are a large number of packages available
for install in that repository. The "pkg" command will assist in
retrieving software.
Refer to http://dlc.sun.com/osol/docs/content/IPS/ggcph.html for
information on the new IPS (Image Packaging System) in OpenSolaris
2008.05
In addition to standard software package a package group has been made available so the user wanting to have the complete software suite can do so with the invocation of a single pkg command.
Q: Is there a package which will install a set of web tools (a SAMP
stack) on OpenSolaris 2008.05?
A: Yes, there is a web stack package available which includes Apache,
MySQL, PHP/Perl/Python. Installation is done with a single command:
# pkg install amp-dev
A list of packages installed with this group is listed at:
http://dlc.sun.com/osol/docs/content/IPS/webstacktbl.html
Q: What package groups are available for OpenSolaris 2008.05?
A: There are many developer packages available.
amp-dev: SAMP, a web stack with Apache, MySQL, PHP/Perl/Python.
ss-dev: Sun Studio Express 5/08
gcc-dev: GNU compilers, make, gdb and other GNU programs.
java-dev: Glassfish, ant, Sun Studio, Netbeans
Installation can be done with:
# pkg install <package name or package group name>
More details can be found at "Installing Developer Packages".
Q: The time on the instance is set back in 1970.
A: This is due to a bug in OpenSolaris 2008.05 (CR 6574993). There
are no
current workarounds but setting up and running ntp helps. ntp can be
configured to use an open ntp server. It is also possible to run ntpdate
after boot to set the time properly. A list of public ntp servers can
be found at: http://support.ntp.org/bin/view/Servers/NTPPoolServers
-bash-3.2# ntpdate 0.pool.ntp.org
Please note that setting the time after boot still leaves a number
of system files and directories with the errant time.
Build 79 :
Q: The time is not set correctly, is there a way to fix it?
A: After boot, run the command "rtc -z GMT" as root. This will update
the kernel with information on the timzone of the hardware clock. To
change the timezone, edit /etc/TIMEZONE with the proper TZ entry. See
the timezone(4) manpage for pointers to valid TZ entries and TIMEZONE(4)
manpage for more information on the /etc/TIMEZONE file.
Generic
:
Q: Is there additional storage on the instances?
A: Yes, depending on the model you launch there are one to three
additional storage devices. A quick way to access the storage is to
use ZFS. The following commands will create a ZFS filesystem on
the instance.
OpenSolaris 2008.05 32 Bit small: zpool create storage c4d1
SXCE Build 79 - 32 Bit small: zpool create storage c0d1
SXCE Build 79 - 64 Bit large: zpool create storage c0d1 c0d2
SXCE Build 79 - 64 Bit xlarge: zpool create storage c0d1 c0d2 c0d3
c0d4
The above commands will allocate all of the storage from the instances
disks into a large zfs pool accessible at /storage. See the zpool(1m)
and
zfs(1M) man pages for further information.
We thought you may want to know about the team who is working on
making OpenSolaris integrate and work on Amazon EC2. This team is your
first point of contact for any information on EC2/OpenSolaris and for
providing any technical assistance you may need while you develop and
create your stacks on OpenSolaris for Amazon EC2. You can reach the
team at ec2-solaris [at] sun [dot] com.
Rajesh RamchandaniI am a Senior Market Development manager and manage relationships with startups, developers and strategic partners and am part of Startups and Emerging Markets team at Sun. As part of Sun Startup Essentials team, I am currently working on ways to help early stage startups to provide them Sun technologies, products and services which help them take-off the ground and grow their businesses with very little or no investments and shorter time to market. I am managing business relationship with Amazon Web services and working with the team to bring best Operating System on Planet aka OpenSolaris to the developers, startups and other EC2 users.
Sujeet Vasudevan
Sujeet Vasudevan is a Senior Engineering Manager in Global Market Development and engineering. He has
over 17 years of industry experience working with various ISV,
customers, SIs and startups. He works primarily with Oracle as a team
leader, and also leads team of engineers who work with many others ISVs
across the spectrum. Sujeet is lead engineering manager on integration of OpenSolaris with EC2.
Dileep Kumar
Dileep Kumar is a Staff Engineer in the ISV Engineering Group at Sun
Microsystems, Inc. He has over ten years of experience in the computer
industry and now works on OpenSolaris and IBM WebSphere products on
Solaris. His area of expertise includes Solaris, Java and J2EE based
system design and development, performance enhancement, and
implementation. He holds a M.S. degree in Engineering Management from
Santa Clara University, Santa Clara. Read Dileep's professional blog at
http://blogs.sun.com/dkumar
Alan Yoshida
Alan is one of the lead developers on OpenSolaris on EC2 engineering team. Alan has been working on porting AMI and API tools to OpenSolaris and most often you will hear from Alan in case you run into any OpenSolaris issues.
Sharlene Wong
Sharlene is program manager for OpenSolaris on EC2 beta program. If you have been approved for access to OpenSolaris AMIs, you've seen emails from Sharlene. Sharlene will continue to be you point of communication during the Beta program.
Divyen Patel
Divyen Patel is an engineer in the ISV Engineering group at Sun Microsystems. He is graduated from San Jose State University with Major in Software Engineering. His area of interest includes web 2.0 and its related technologies. Read Divyen's weblog at http://blogs.sun.com/divyen
Prateek Parekh
Prateek Parekh is a developer in the ISV Engineering team at Sun Microsystems, and has worked on J2EE, J2ME, Java CAPS, Unix and Web 2.0 related technologies. He holds an MS degree in Software Engineering from San Jose State University. He maintains a blog at http://blogs.sun.com/prateek
Announcing availability of new AMIs based on SXCE AMI and GlassFish. For those of you who have been provided access to OpenSolaris AMIs, you have access to these AMIs as well.
1. OpenSolaris (SXCE) + GlassFish ( This is smaller footprint of SXCE or "Just Enough OS" )
a. 32-bit AMI: ami-8142a7e8 aki-b57b9edc ari-b47b9edd / sun-osol/JeOS-79_32_1.0.img.manifest.xml
b. 64-bit AMI: ami-314da858 aki-8e7a9fe7 ari-817a9fe8 / sun-osol/JeOS-79_64_1.0.img.manifest.xml
2. OpenSolaris (SXCE) + GlassFish + MySQL
a. 32-bit AMI: ami-3742a75e aki-b57b9edc ari-b47b9edd / sun-osol/GFAS-MySQL-79_32_1.0.img.manifest.xml
3. OpenSolaris (SXCE) + GlassFish + Liferay
a. 32-bit AMI: ami-cb40a5a2 aki-b57b9edc ari-b47b9edd / sun-osol/GF-LF-79_32_1.0.img.manifest.xml
b. 64-bit AMI: ami-dd40a5b4 aki-8e7a9fe7 ari-817a9fe8 / sun-osol/GF-LF-79_64_1.0.img.manifest.xml
For more information and details on these AMIs including user names and passwords, please refer to Rudolf's detailed blog. As always, please feel free to contact us at ec2-solaris [at] sun [dot] com.
We are almost reaching the capacity limit on number of instances we have available on Amazon EC2 for running OpenSolaris 2008.05 AMI. As we continue to integrate and test OpenSolaris 2008.05 with EC2, we have limitation on the number of instances allocated currently. If you have been granted access to OpenSolaris 2008.05 AMI, please note that EC2_URL must be set to :
EC2_URL=https://ec2-pinotage.amazonaws.com. Once Amazon team has completed testing, the URL will change and communicated.
Please note that the image re-bundling tools are being under development / testing at this time as well.
Solaris Express.79 AMIs are available for deployment in full capacity. AMI / API tools have been completed ported and tested and are available for use.
Thank you for your patience as Sun and Amazon continue to work together to make EC2 and OpenSolaris a platform to offer best development experience.
If you have any suggestions or comments or would like to help out with OpenSolaris 2008.05 re-bundling efforts, please feel free to contact us at ec2-solaris [AT] SUN [dot] COM.
So, what did this collaboration between Amazon Web services and Sun do? It offers choice, choice to developers and startups in two ways-
1) OpenSolaris provides ZFS filesystem, first 128-bit filesystem which offer 64-bit checksum capability to maintain data integrity and Rollback functionality to be able to quickly rollback changes to the last known stable state in case of data corruption of instability in the fileysytsem. Secondly, OpenSolaris is the only OS which offers Dtrace - a observability and tracing tool which helps developers and administrators to debug and trace issues and bottlenecks right from the hardware level (OS level in case of EC2) to their application code and middleware in between. Most importantly, developers and EC2 users can now have choice of multiple OSes other than various variants of Linux.
2) Freedom to run MySQL on Cloud. MySQl is most critical component of any web applications and MySQL's commitment to support Cloud such as EC2 as supported platform, opens the doors for startups to leverage the cloud infrastructure to further reduce the cost and capex investment towards the infrastructure build-out at early stages of the company.
We are excited to be working with Amazon Web services and providing choice to developers, startups and students who wish to use EC2 cloud for their development and deployment use.
Here you will find the latest information on the program and any late breaking information. OpenSolaris on EC2 team welcomes you and we look forward to providing you any technical assistance and information you need for a best possible OpenSolaris experience. Please feel free to leave comments here or contact us at ec2-solaris-support [AT] SUN [dot] COM.
This blog copyright 2009 by rrajesh