Thursday Sep 04, 2008

Thanks to one of our users who informed us of a problem which is noticeable only after rebooting an instance of the Ruby On Rails Stack with DTrace AMI - this was caused by a rebundling error. We are already working to correct this issue and the current AMI is being retired, effective immediately. If you are currently using this AMI, please terminate your instance and we will notify you about a new AMI to be available shortly.

We apologize for any inconvenience this may have caused. Thanks in advance for your understanding and patience.
OpenSolaris on Amazon EC2 Team

Friday Aug 29, 2008

This 32 bit AMI is based OpenSolaris 2008.05 (with packages updated to build 91) and Joomla 1.5.6 (in pre-configured state). Following components are included within this AMI
  • Apache 2.2 , PHP 5.2.6, MySQL 5.0.x and Memcached 1.2.5 .

  • Joomla 1.5.6 (pre-configured state)

  • phpMyAdmin

  • Apache and MySQL services are pre-configured to start on boot.

  • Dtrace probes are also available within Apache and PHP runtime. Sample Dtrace scripts are also available under /opt/DTT/ .


Users can launch and configure Joomla by accessing http://<DNS name associated with the instance>/joomla in their browser.


AMI ID:

ami-2bf01442

AMI Manifest:

sun-osol/opensol2008.05-b91-AMPStack-Joomla.img.manifest.xml

License:

Public

Description:

OpenSolaris 2008.05 is a distribution based on OpenSolaris operating system and includes popular features like

  • ZFS (as its root file system)

  • new packaging system known as Image packaging system (IPS) where users can download any additional software, if needed, from a repository.

  • Dtrace for performance monitoring and tuning purposes.

  • GNU binaries under default PATH and many more...


For more information, check out OpenSolaris 2008.05 Getting Starting Guide from here


Amazon EC2 customers can get more information on OpenSolaris by visiting our OpenSolaris on Amazon EC2 Getting Started Guide from http://www.sun.com/thirdparty/global/amazon/Sun_AmazonEC2_GettingStartedAug08Update.pdf


Joomla - an open source content management platform - supports developing wide variety of websites ranging from personal blogs, corporate brochures and large community-driven websites. For more information on Joomla , refer to its Getting Started Guide within - http://www.joomla.org/about-joomla/getting-started.html

AMP Stack File Layout:


Apache

PHP

MySQL

Binary Runtime Files

/usr/apache2/2.2/bin

/usr/php/5.2/bin

/usr/mysql/5.0/bin

Configuration Files

/etc/apache2/2.2

/etc/php/5.2

/etc/mysql/5.0

Web Documents / Data Files

/var/apache2/2.2

/var/php/5.2

/var/mysql/5.0

Administering AMP Stack:

Commands

Apache

MySQL

Start Service

svcadm enable http:apache22

svcadm enable mysql:version_50

Stop Service

svcadm disable http:apache22

svcadm disable mysql:version_50

Restart Service (only if the service is already running)

svcadm restart http:apache22

svcadm restart mysql:version_50


Enabling phpMyAdmin for MySQL Administration:

phpMyAdmin – popular tool written in PHP to handle MySQL administation over the web – is bundled within this AMI. However, for security purposes, 'phpMyAdmin' application is 'disabled' by default within this AMI. Users are strongly urged to protect their MySQL 'root' user with a proper password before enabling this application.

  • Secure your MySQL database by doing some thing like

    • /usr/mysql/5.0/bin/mysql -u root -p password <give a new password>

  • Un-comment the following line in /etc/apache2/2.2/conf.d/phpmyadmin.conf

Alias /phpmyadmin /usr/phpmyadmin

  • Restart Apache Service by doing some thing like

  • Now, access phpMyAdmin application over the web by visiting

    • http://<dns name associated with this AMI>/phpmyadmin . Will need to login with a valid MySQL database's user name and password.


Configuring Joomla:

  • Joomla (bundled within this AMI in pre-configured state) is available under location /var/joomla

  • Joomla recommends having a database protected with a valid user name and password to be created on the system before configuring Joomla. Hence, users are advised to either use 'ssh' to login to your AMI or phpMyAdmin to create such database before proceeding to configuring Joomla. For more information, refer to MySQL documentation by visiting here

  • Joomla specific configuration for Apache Web Server is available within /etc/apache2/2.2/conf.d/joomla.conf .

  • Users can launch and configure Joomla by accessing http://<DNS name associated with the instance>/joomla in their browser.


Support :

  • For technical support during Beta period, please send emails to ec2-solaris-support[AT]SUN[DOT]COM.

  • AMP Stack within OpenSolaris are delivered as part of WebStack project. For any questions related to these components, please write to webstack-discuss[AT]opensolaris[DOT]org


Register for OpenSolaris AMIs by visiting here.

This 32 bit AMI is based on OpenSolaris 2008.05 (with packages updated to build 91) and Drupal 6.4 (in pre-configured state). Following components are included within this AMI

  • Apache 2.2 , PHP 5.2.6, MySQL 5.0.x and Memcached 1.2.5 .

  • Drupal 6.4 (pre-configured state)

  • phpMyAdmin

  • Apache and MySQL services are pre-configured to start on boot.

  • Dtrace probes are also available within Apache and PHP runtime. Sample Dtrace scripts are also available under /opt/DTT/ .

Users can launch and configure Drupal by accessing http://<DNS name associated with the instance>/drupal in their browser.

AMI ID:

ami-4ff01426

AMI Manifest:

sun-osol/opensol2008.05-b91-AMPStack-Drupal.img.manifest.xml

License:

Public

Description:

OpenSolaris 2008.05 is a distribution based on OpenSolaris operating system and includes popular features like

  • ZFS (as its root file system)

  • new packaging system known as Image packaging system (IPS) where users can download any additional software, if needed, from a repository.

  • Dtrace for performance monitoring and tuning purposes.

  • GNU binaries under default PATH and many more...


For more information, check out OpenSolaris 2008.05 Getting Starting Guide from here


Amazon EC2 customers can get more information on OpenSolaris by visiting our OpenSolaris on Amazon EC2 Getting Started Guide from http://www.sun.com/thirdparty/global/amazon/Sun_AmazonEC2_GettingStartedAug08Update.pdf

Drupal - an open source content management platform - supports a variety of websites ranging from personal blogs, corporate brochures and large community-driven websites. For more information on Drupal , refer to its Getting Started Guide within - http://drupal.org/getting-started

AMP Stack File Layout:


Apache

PHP

MySQL

Binary Runtime Files

/usr/apache2/2.2/bin

/usr/php/5.2/bin

/usr/mysql/5.0/bin

Configuration Files

/etc/apache2/2.2

/etc/php/5.2

/etc/mysql/5.0

Web Documents / Data Files

/var/apache2/2.2

/var/php/5.2

/var/mysql/5.0

Administering AMP Stack:

Commands

Apache

MySQL

Start Service

svcadm enable http:apache22

svcadm enable mysql:version_50

Stop Service

svcadm disable http:apache22

svcadm disable mysql:version_50

Restart Service (only if the service is already running)

svcadm restart http:apache22

svcadm restart mysql:version_50

Enabling phpMyAdmin for MySQL Administration:

phpMyAdmin – popular tool written in PHP to handle MySQL administation over the web – is bundled within this AMI. However, for security purposes, 'phpMyAdmin' application is 'disabled' by default within this AMI. Users are strongly urged to protect their MySQL 'root' user with a proper password before enabling this application.


  • Secure your MySQL database by doing some thing like

    • /usr/mysql/5.0/bin/mysql -u root -p password <give a new password>

  • Un-comment the following line in /etc/apache2/2.2/conf.d/phpmyadmin.conf

Alias /phpmyadmin /usr/phpmyadmin

  • Restart Apache Service by doing some thing like

  • Now, access phpMyAdmin application over the web by visiting

    • http://<dns name associated with this AMI>/phpmyadmin . Will need to login with a valid MySQL database's user name and password.

Configuring Drupal:

  • Drupal (bundled within this AMI in pre-configured state) is available under location /var/drupal.

  • Drupal recommends having a database protected with a valid user name and password to be created on the system before configuring Drupal. Hence, users are advised to either use 'ssh' to login to your AMI or phpMyAdmin to create such database before proceeding to configuring Drupal. For more information, refer to MySQL documentation by visiting here

  • Drupal specific configuration for Apache Web Server is available within /etc/apache2/2.2/conf.d/drupal.conf .

  • Users can launch and configure Drupal by accessing http://<DNS name associated with the instance>/drupal in their browser.


Support :

  • For technical support during Beta period, please send emails to ec2-solaris-support[AT]SUN[DOT]COM.

  • AMP Stack within OpenSolaris are delivered as part of WebStack project. For any questions related to these components, please write to webstack-discuss[AT]opensolaris[DOT]org


Register for OpenSolaris AMIs by visiting here.

This 32 bit AMI is based on OpenSolaris 2008.05 (with packages updated to build 91) and MediaWiki 1.12 (in pre-configured state). Following components are included within this AMI
  • Apache 2.2 , PHP 5.2.6, MySQL 5.0.x and Memcached 1.2.5 .

  • MediaWiki 1.12

  • phpMyAdmin

  • Apache and MySQL services are pre-configured to start on boot.

  • Dtrace probes are also available within Apache and PHP runtime. Sample Dtrace scripts are also available under /opt/DTT/ .


Users can launch and configure MediaWiki by accessing http://<DNS name associated with the instance>/mediawiki in their browser.

AMI ID:

ami-2ef11547

AMI Manifest:

sun-osol/opensol2008.05-b91-AMPStack-MediaWiki.img.manifest.xml

License:

Public

Description:

OpenSolaris 2008.05 is a distribution based on OpenSolaris operating system and includes popular features like

  • ZFS (as its root file system)

  • new packaging system known as Image packaging system (IPS) where users can download any additional software, if needed, from a repository.

  • Dtrace for performance monitoring and tuning purposes.

  • GNU binaries under default PATH and many more...


For more information, check out OpenSolaris 2008.05 Getting Starting Guide from here


Amazon EC2 customers can get more information on OpenSolaris by visiting our OpenSolaris on Amazon EC2 Getting Started Guide from http://www.sun.com/thirdparty/global/amazon/Sun_AmazonEC2_GettingStartedAug08Update.pdf


MediaWiki - an open source Wiki software that powers popular site like Wikipedia – free encyclopedia. For more information on MediaWiki , refer to its Getting Started Guide within - http://www.mediawiki.org/wiki/Manual:Contents

AMP Stack File Layout:


Apache

PHP

MySQL

Binary Runtime Files

/usr/apache2/2.2/bin

/usr/php/5.2/bin

/usr/mysql/5.0/bin

Configuration Files

/etc/apache2/2.2

/etc/php/5.2

/etc/mysql/5.0

Web Documents / Data Files

/var/apache2/2.2

/var/php/5.2

/var/mysql/5.0

Administering AMP Stack:

Commands

Apache

MySQL

Start Service

svcadm enable http:apache22

svcadm enable mysql:version_50

Stop Service

svcadm disable http:apache22

svcadm disable mysql:version_50

Restart Service (only if the service is already running)

svcadm restart http:apache22

svcadm restart mysql:version_50

Enabling phpMyAdmin for MySQL Administration:

phpMyAdmin – popular tool written in PHP to handle MySQL administation over the web – is bundled within this AMI. However, for security purposes, 'phpMyAdmin' application is 'disabled' by default within this AMI. Users are strongly urged to protect their MySQL 'root' user with a proper password before enabling this application.

  • Secure your MySQL database by doing some thing like

    • /usr/mysql/5.0/bin/mysql -u root -p password <give a new password>

  • Un-comment the following line in /etc/apache2/2.2/conf.d/phpmyadmin.conf

Alias /phpmyadmin /usr/phpmyadmin

  • Restart Apache Service by doing some thing like

  • Now, access phpMyAdmin application over the web by visiting

    • http://<dns name associated with this AMI>/phpmyadmin . Will need to login with a valid MySQL database's user name and password.

Configuring MediaWiki:

  • MediaWiki specific configuration for Apache Web Server is available within /etc/apache2/2.2/conf.d/mediawiki.conf

  • Users can launch and configure MediaWiki by accessing http://<DNS name associated with the instance>/mediawiki in their browser.

Support :

  • For technical support during Beta period, please send emails to ec2-solaris-support[AT]SUN[DOT]COM.

  • AMP Stack within OpenSolaris are delivered as part of WebStack project. For any questions related to these components, please write to webstack-discuss[AT]opensolaris[DOT]org


Register for OpenSolaris AMIs by visiting here.

This 32 bit AMI is based on OpenSolaris 2008.05 (with packages updated to build 91) . Following components are included within this AMI
  • Apache 2.2 , PHP 5.2.6, MySQL 5.0.x and Memcached 1.2.5 .

  • phpMyAdmin

  • Apache and MySQL services are pre-configured to start on boot.

  • Dtrace probes are also available within Apache and PHP runtime. Sample Dtrace scripts are also available under /opt/DTT/ .

AMI ID:

ami-45f1152c

AMI Manifest:

sun-osol/opensol2008.05-b91-AMPStack.img.manifest.xml

License:

Public

Description:

Open Solaris 2008.05 is a distribution based on OpenSolaris operating system and includes popular features like

  • ZFS (as its root file system)

  • new packaging system known as Image packaging system (IPS) where users can download any additional software, if needed, from a repository.

  • Dtrace for performance monitoring and tuning purposes.

  • GNU binaries under default PATH and many more...


For more information, check out OpenSolaris 2008.05 Getting Starting Guide from here


Amazon EC2 customers can get more information on OpenSolaris by visiting our OpenSolaris on Amazon EC2 Getting Started Guide from http://www.sun.com/thirdparty/global/amazon/Sun_AmazonEC2_GettingStartedAug08Update.pdf

AMP Stack File Layout:


Apache

PHP

MySQL

Binary Runtime Files

/usr/apache2/2.2/bin

/usr/php/5.2/bin

/usr/mysql/5.0/bin

Configuration Files

/etc/apache2/2.2

/etc/php/5.2

/etc/mysql/5.0

Web Documents / Data Files

/var/apache2/2.2

/var/php/5.2

/var/mysql/5.0

Administering AMP Stack:

Commands

Apache

MySQL

Start Service

svcadm enable http:apache22

svcadm enable mysql:version_50

Stop Service

svcadm disable http:apache22

svcadm disable mysql:version_50

Restart Service (only if the service is already running)

svcadm restart http:apache22

svcadm restart mysql:version_50


Enabling phpMyAdmin for MySQL Administration:

phpMyAdmin – popular tool written in PHP to handle MySQL administation over the web – is bundled within this AMI. However, for security purposes, 'phpMyAdmin' application is 'disabled' by default within this AMI. Users are strongly urged to protect their MySQL 'root' user with a proper password before enabling this application.

  • Secure your MySQL database by doing some thing like

    • /usr/mysql/5.0/bin/mysql -u root -p password <give a new password>

  • Un-comment the following line in /etc/apache2/2.2/conf.d/phpmyadmin.conf

Alias /phpmyadmin /usr/phpmyadmin

  • Restart Apache Service by doing some thing like

  • Now, access phpMyAdmin application over the web by visiting

    • http://<dns name associated with this AMI>/phpmyadmin . Will need to login with a valid MySQL database's user name and password.

Support :

  • For technical support during Beta period, please send emails to ec2-solaris-support[AT]SUN[DOT]COM.

  • AMP Stack within OpenSolaris are delivered as part of WebStack project. For any questions related to these components, please write to webstack-discuss[AT]opensolaris[DOT]org


Register for OpenSolaris AMIs by visiting here.

Following the launch of the Ruby on Rails 2 AMI, we are pleased to announce the launch of our "MySQL 5 with Dtrace probes" AMI.

This 32-bit AMI is based on the OpenSolaris 2008.05 AMI with updated software revisions to build 91. It contains MySQL 5.1.25 binaries, and SysBench. SysBench is a modular, cross-platform and multi-threaded benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load. There are also sample Dtrace scripts which utilize the probes in MySQL to collect information about SQL calls. Users should check /export/home/mysql README files for directions.The MySQL and SysBench binaries are located in the /usr/local/mysql and /sysbench directories. An existing user 'mysql' is authorized to start the MySQL server and run tests.

The OpenSolaris distribution is based on the OpenSolaris project, and features functionality such as ZFS, DTrace, and a new packaging system, the Image Packaging System(IPS). IPS is a network based package management system that allows for users to install applications from online repositories using the "pkg" command. Developers can install additional software on this AMI from the repository using the pkg command. The root file system is ZFS in this AMI.

Also, just like in the RoR AMI, the "pkg image-update" command updates the kernel and ramdisk which is not allowed in EC2. Therefore, in order to prevent your instances from becoming non-compatible with the Amazon EC2 environment, we have disabled this command. More details including re-bundling instructions can be found in the Getting Started Guide.

More details can be found at:

http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1653&categoryID=101

For more information on Dtrace and SysBench, please visit
http://www.sun.com/software/solaris/howtoguides/dtracehowto.jsp
http://sourceforge.net/projects/sysbench


Wednesday Aug 27, 2008

Going forward we are planning to have more and more OpenSolaris based AMIs on Amazon EC2 so we thought providing a table would help users quickly locate and access the AMIs. We will keep updating the table as and when a new AMI gets added to EC2 environment.

AMI ID
AKI/ARI Description
ami-41e70328
aki-b78367de/ari-b68367df

OpenSolaris 2008.05 (build 91) 32-bit AMI for developers.

For more information click here.

ami-89ef0be0 aki-b78367de/ari-b68367df

The MySQL  with Dtrace probes AMI is based on ami-41e70328 and contains MySQL 5.1.25 binaries, and sysbench (used to create and edit MySQL databases). There are also sample Dtrace scripts which utilize the probes in MySQL to collect information about SQL calls.

For more information go here.

ami-8fe206e6 aki-b78367de/ari-b68367df

The JBOSS + Apache + MySQL AMI is based on ami-41e70328 and contains 32-bit OpenSolaris 2008.05 with packages updated to build 91, Jboss 4.2.3 App Server, JDK 1.5/1.6, Mysql 5.0 and Apache 2.2.

For more information go here.

ami-45f1152c

aki-b78367de/ari-b68367df

The AMP stack AMI is based on ami-41e70328 and contains:

  • Apache 2.2 , PHP 5.2.6, MySQL 5.0.x and Memcached 1.2.5 .
  • phpMyAdmin
  • Apache and MySQL services are pre-configured to start on boot.
  • Dtrace probes are also available within Apache and PHP runtime. Sample Dtrace scripts are also available under /opt/DTT/.
For more information go here.
ami-4ff01426 aki-b78367de/ari-b68367df

The Drupal using AMP stack AMI is based on ami-45f1152c and contains Drupal 6.4 (in pre-configured state). Following components are included:

  • Drupal 6.4 (pre-configured state)
  • phpMyAdmin
  • Apache and MySQL services are pre-configured to start on boot.
  • Dtrace probes are also available within Apache and PHP runtime. Sample Dtrace scripts are also available under /opt/DTT/ .
For more information go here.
aki-b78367de/ari-b68367df

The Joomla using AMP stack AMI is based on ami-45f1152c and contains Joomla 1.5.6 (in pre-configured state). Following components are included within this AMI

  • Apache 2.2 , PHP 5.2.6, MySQL 5.0.x and Memcached 1.2.5 .
  • Joomla 1.5.6 (pre-configured state)
  • phpMyAdmin
  • Apache and MySQL services are pre-configured to start on boot.
  • Dtrace probes are also available within Apache and PHP runtime. Sample Dtrace scripts are also available under /opt/DTT/.
For more information go here.

ami-2ef11547

aki-b78367de/ari-b68367df

This MediaWiki using AMP stack AMI is based on ami-45f1152c and contains MediaWiki 1.12 (in pre-configured state). Following components are included within this AMI Apache 2.2 , PHP 5.2.6, MySQL 5.0.x and Memcached 1.2.5 .

  • MediaWiki 1.12
  • phpMyAdmin
  • Apache and MySQL services are pre-configured to start on boot.
  • Dtrace probes are also available within Apache and PHP runtime. Sample Dtrace scripts are also available under /opt/DTT/ .

For more information go here.

ami-8142a7e8 (32-bit) and ami-314da858 (64-bit)

aki-b57b9edc/ari-b47b9edd(32-bit)

aki-8e7a9fe7/ari-817a9fe8(64-bit)

This AMI is for developers who wish to deploy web applications on GlassFish application server on OpenSolaris(SXCE). Both 32-bit and 64-bit versions of the AMI are available.

For more information go here.

ami-9040a5f9 aki-b57b9edc/ari-b47b9edd Solaris Express Community Edition is Sun's binary release for OpenSolaris developers (code named "Nevada"). It is built from the latest OpenSolaris source and additional technology that has not been published in the OpenSolaris source base.

Thursday Aug 21, 2008

How to use EBS with OpenSolaris AMI  

MySQL

Today Amazon Web Services announced  availability of most awaited feature of Persistent Storage (officially called as  Elastic Block Storage) on EC2 cloud. And we are excited to announce that OpenSolaris does support creating ZFS file system on EBS. ZFS is the default filesystem in OpenSolaris and provides data integrity by 128-bit checksumming and has Rollback features to roll back any changes in the filessystem to previous consistent state.

 As a EC2 user, you probably know, running database servers on EC2 instances required you to put together mechanisms to backup the databases since the local storage wasn't persistent and would go away once the instances were shutdown or crashed. EBS solves this problem and provides local persistent storage which can be attached the running instances for database servers such as MySQL. Since MySQL is the most popular open source database server on EC2, it stands to benefit the most from EBS. Here's a link to Amazon Web services article on benefits of using EBS for MySQL. One I would like to highlight is:

Improved performance - Early reports from studies on EBS disk IO performance indicate that EBS IO rates can be multiple times faster than ephemeral storage and even local disk IO. This has obvious benefits for databases which are often IO bound.

Since, EBS provides capability to create file systems on the volumes, users can now take advantage of ZFS on EBS and you may get improved performance for certain workloads and database sizes. Here's an interesting article on MySQL and ZFS and performance comparison of ZFS Vs UFS file system (I think its OK to assume, ext3 will be comparable to UFS and hence these numbers may be good comparison between Ext3 fs and ZFS

I am sure you'll find EBS feature useful and even better take advantages of ZFS on EBS. If there's any feedback or comments, as always, please send them to us via ec2-solaris [AT] SUN [DOT] COM.

Wednesday Aug 13, 2008

Following the success and interest in the May 5th launch of OpenSolaris 2008.05, we together with Amazon are happy to release the OpenSolaris 2008.05-based 32-bit AMI in a public grid. We also took this opportunity to include few changes/enhancement and bug fixes. 

This means that capacity has been added for the OpenSolaris-based AMI so everyone can access it and launch as many instances as needed. The "Getting Started Guide" has been updated with instructions on how to rebundle a running OpenSolaris 2008.05 -based 32-bit AMI on Amazon EC2 which was not available previously due to technical issues. The rebundling process has been accelerated so it takes less time to rebundle. If you haven't been able to do so, try the rebundling process and let us know if you find any issues using the Guide. As we already know, OpenSolaris boots from a ZFS- based root file system, (project name "zfs boot") which is the reason why rebundling uses ZFS-based commands.

There is a lot of work in progress to deliver pre-packaged stacks, which includes a set of software stack, targeted towards a specific need. This will save time from installing and configuring the software and the user can be ready to start working on what they need to instead of configuring the individual software. Keep checking the "Open Solaris on EC2" web-page for availability of these images.

Following is the AMI/AKI/ARI ID's for this new 32-bit machine image:

 AMI ID
ami-41e70328
 AKI ID
aki-b78367de
 ARI ID
ari-b68367df

Here are the steps  for launching this new image:

ec2-run-instances --url https://ec2.amazonaws.com ami-41e70328 -k <KEY PAIR NAME>

or

export EC2_URL=https://ec2.amazonaws.com
ec2-run-instances ami-41e70328 -k <KEY PAIR NAME>

Once the list along with the AMI id's is available, check for what you need and if there is something missing which you'd like to be added, do let us know. AMI contribution  and ideas from users are welcome and we can work together to get these ready for others to use. As OpenSolaris is a community based effort, we welcome the community contributions to make it easier for others.

For more details and new technical information keep checking:


Tuesday Jul 15, 2008

ALERT: Amazon just informed that the Pinotage endpoint server will be down for scheduled maintenance on Wednesday starting at 2AM PDT. The maintenance is scheduled to last for 2 - 3hours. Sorry for any inconvenience this may cause.

Tuesday Jul 08, 2008

EC2 Solaris Parameterized Launches
Solaris Parameterized Launches Example by Sean O'Dell

Building on the excellent writeup, Using Parameterized Launches to Customize Your AMIs, in this blog entry we provide an example tailored for Solaris. In the very near future Sun will be adding the capability to re-bundle OpenSolaris 2008.05 AMIs. Once this functionality is available, this example will work for both OpenSolaris 2008.05 and SXCE.

Introduction

It is possible to make "user data" available to an EC2 instance, specified when the instance is launched from the command line. From the Amazon Elastic Compute Cloud Developer Guide, we find the following options available for the ec2-run-instance command:

-d user_data
Data to make available to the instances. This data is read from the command line of the USER_DATA argument. If you want the data to be read from a file, see the -f option.
Example: -d "my user data"

-f user_data_file
Data to make available to these instances. The data is read from the file specified by FILE_NAME. To specify user data on the command line, use the -d option.
Example: -f data.zip

Passing data to the instance at launch time is very useful as a mechanism for customizing instances without having to create a new AMI image for each type of instance.

For example, using this technique we can use the same AMI to run a NFS file server and a NFS client. The only difference is that for the NFS file server we share filesystems and start NFS services, and for the NFS client we mount the NFS filesystems from the server. We will use this NFS scenario for our example.


Getting Starting

The first step is to use one of the Sun provided Solaris SXCE Build 79 AMIs (or the OpenSolaris 2008.05 AMI once re-bundling is available) to create a custom AMI with the addition of a script that runs at startup. We also add the S3 Sync tools which we use to interface with S3. You can use one of the Sun provided AMIs as a starting point, or if you have already bundled a new AMI, you can add to one of your own.

The following is a high level summary of what we will cover.
  1. Start a new instance, either from a Sun provided AMI or one of your own.
  2. Add a script that runs when the instance starts for the first time. Install the S3 Sync tools.
  3. Create a new AMI which includes the script that runs at startup and S3 Sync tools from step 2.
  4. Demonstrate how to launch a NFS server instance and a NFS client instance, using the single AMI created in step 3.
We assume that the reader has a basic understanding of EC2 and Solaris. We also build on the previous blog entry which described how to save and restore ZFS snapshots to and from S3.

Launch an instance of the Solaris AMI and login to the new instance.

bash# ec2-run-instances -k <my-key-pair> -t m1.small ami-eb7a9f82

bash# ec2-describe-instances | grep INSTANCE | cut -f2,3,4,6
i-196fb970  ami-eb7a9f82  ec2-75-101-225-153.compute-1.amazonaws-DOT-com  running

bash# ssh -i <my-key-pair-file> root-AT-ec2-75-101-225-153.compute-1.amazonaws.com
Last login: Mon Apr 28 02:24:21 2008
Sun Microsystems Inc.   SunOS 5.11      snv_79  January 2008

# bash -o vi

Create the script to run at boot time with the following contents. We name this file:
/etc/init.d/ec2autorun

#!/bin/sh

ec2autorundir="/var/ec2"
ec2autorunfile="ec2autorun.input"
ec2autorunscript="ec2autorun.script"
ec2userdataurl="http://169.254.169.254/latest/user-data"

# we password protect the user data file since any user can retrieve.
# we need to remember the password as we will need to use when
# creating the user data zip file. This password can be anything
# and is not the same as your AWS secret key.

ec2password="put your password here"

# we usually pass the user data as a ZIP archive file, but we
# can also simply pass a plain text script file.

zipfiletype="ZIP archive"

case "$1" in
'start')
        # create directory for autorun scripts and data
        if [ ! -d ${ec2autorundir} ]
        then
            /usr/bin/mkdir -p ${ec2autorundir}
        else
            # if the directory already exists we do not run the script again
            echo "`date`: ec2autorun script already ran, exiting." \
              >> ${ec2autorundir}/${ec2autorunfile}.stdout
            exit 0
        fi
       
        cd ${ec2autorundir}

        # get the user data file passed to the instance when created
        /usr/sfw/bin/wget --quiet \
          --output-document=${ec2autorundir}/${ec2autorunfile} \
            ${ec2userdataurl}

        # test the file size, if zero then no user data was provided
        # when instance was started
        if [ ! -s ${ec2autorundir}/${ec2autorunfile} ]
        then
            echo "User data was not passed to instance." \
              >> ${ec2autorunfile}.stdout
            exit 0
        fi

        # if the file is a zip file, unzip it and then run the
        # script extracted from the zip file, else run the file
        # assuming it is a script
        filetype=`file ${ec2autorundir}/${ec2autorunfile} | /usr/bin/cut -f2`
        if [ "${filetype}" = "${zipfiletype}" ]
        then
            unzip -P ${ec2password} ${ec2autorunfile} \
              >> ${ec2autorunfile}.stdout 2>>${ec2autorunfile}.stderr

            bash ./${ec2autorunscript} \
              >> ${ec2autorunscript}.stdout 2>>${ec2autorunscript}.stderr
        else
            bash ./${ec2autorunfile} \
              >> ${ec2autorunfile}.stdout 2>>${ec2autorunfile}.stderr
        fi

        # set the autorun directory to be read only by root
        chmod 700 ${ec2autorundir}

        ;;
*)
        echo "Usage: $0 { start }"
        ;;
esac
exit 0

Change the permissions on the startup script to be readable only by the root user and create a link in /etc/rc3.d so that the script runs at boot time.

bash# chmod 700 /etc/init.d/ec2autorun
bash# ln /etc/init.d/ec2autorun /etc/rc3.d/S10ec2autorun

Download and install the S3 Sync tools. Install the tools in your preferred location. In our example, we use /usr/local/aws.

bash# mkdir -p /usr/local/aws
bash# cd /usr/local/aws

bash# /usr/sfw/bin/wget \
  http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz

bash# gzcat s3sync.tar.gz | tar xf -
bash# cd /usr/local/aws/s3sync

bash# ln -s s3cmd.rb s3cmd
bash# export PATH=$PATH:/usr/local/aws/s3sync

bash# rm /usr/local/aws/s3sync.tar.gz

Further customize the instance as you desire and then build a new AMI using the instructions found in Getting Started Guide for OpenSolaris on Amazon EC2. See the section "Rebundling and uploading OpenSolaris images on Amazon EC2".

Launching a new NFS Sever

Using the newly created AMI from the Getting Started section above, we are now ready to use the new AMI to launch an instance to run as the NFS file server. In our example, we assume the following:
  1. The AMI id of the image we created above is: ami-93b652fa
  2. We have previously created and saved two ZFS filesystems that the NFS server will share.
  3. You have a system with the EC2 API tools installed and can run the command: ec2-run-instances
First, we create a script that will be passed to a new EC2 instance and then processed using the ec2autorun script that we created and installed within our new AMI as described in the Getting Started section above.

Create the script with the following contents. We name this file: ec2autorun.script

#!/usr/bin/bash

zpool="s3pool"
zpooldisk="c0d1p0"
zfssnapshots="home share"
snapshotsdir="/var/tmp/snapshots"
snapshotsbucket="<my-bucket-name-for-zfs-snapshots>"
s3toolspath="/usr/local/aws/s3sync"

ntpdate 0.north-america.pool.ntp.org

# create the ZFS pool
/usr/sbin/zpool create ${zpool} ${zpooldisk}

# download the ZFS snapshots and restore the file systems
if ! [[ -d ${snapshotsdir} ]]
then
    /usr/bin/mkdir -p ${snapshotsdir}
fi

# setup the environment for S3
export PATH=$PATH:${s3toolspath}
export AWS_ACCESS_KEY_ID=<my-aws-access-key>
export AWS_SECRET_ACCESS_KEY=<my-aws-secret-access-key>

# download the saved ZFS file systems and restore
builtin cd ${snapshotsdir}
for i in ${zfssnapshots}
do
    s3cmd get ${snapshotsbucket}:s3-${i}.bz2 s3-${i}.bz2
done

# restore the snapshots and share for NFS
for i in ${zfssnapshots}
do
    bzcat s3-${i}.bz2 | zfs recv -F -d ${zpool}
    zfs set sharenfs=on ${zpool}/${i}
done

# start the NFS services for this instance
svcadm enable -r svc:/network/nfs/server:default


Create a zip file which includes the script created above, and any other files that you want to pass to the new instance. Since all users have access to the user data passed to an instance, we encrypt the zip file with a password.

Note: make sure you use the exact same password that you specified in the
/etc/init.d/ec2autorun file created in the Getting Started section.

bash# zip -P "put your password here" ec2autorun.zip ec2autorun.script

Now we are ready to launch the a new instance, passing the zip file as user data on the command line.

bash# ec2-run-instances -k <my-key-pair> -f ec2autorun.zip ami-93b652fa

RESERVATION     r-5609c43f      578281684738    default
INSTANCE        i-3f994856      ami-93b652fa    pending
                <my-key-pair>   0               m1.small
                2008-07-01T02:55:46+0000        us-east-1b
                aki-b57b9edc    ari-b47b9edd

The last thing we need to do is take note of the EC2 private host name for the new NFS server instance since we will use it to launch our NFS clients. Once the NFS server startup has completed, we can get the private host name as follows. Note the instance id displayed when the instance was launched from above.

bash# ec2-describe-instances | grep i-3f994856 | cut -f2,5
i-3f994856      ip-10-251-203-4.ec2.internal

Launching a new NFS Client

Once the NFS server is up and running we can launch any number of NFS clients. The script to launch the NFS client is much simpler since all we need to do is launch the instance and then mount the file systems from the NFS server. We use exactly the same steps that we used for the NFS server, just a different script to launch at boot time.

Create a script with the following contents. We name this file: ec2autorun.script
Note that in this script we include the EC2 private host name of the NFS server.

#!/usr/bin/bash

# NFS server private host name. This name is also referred to in
# EC2 as the Private DNS Name.
nfsserver="ip-10-251-203-4.ec2.internal"

# start the services for this instance
svcadm enable -r svc:/network/nfs/client:default

# mount the NFS file systems from the NFS server
mkdir -p /s3pool/home
mkdir -p /s3pool/share

mount -F nfs ${nfsserver}:/s3pool/home  /s3pool/home
mount -F nfs ${nfsserver}:/s3pool/share /s3pool/share

# add to vfstab file
echo "${nfsserver}:/s3pool/home - /s3pool/home nfs - yes rw,xattr"   >> /etc/vfstab
echo "${nfsserver}:/s3pool/share - /s3pool/share nfs - yes rw,xattr" >> /etc/vfstab

We zip this file and launch a new instance, same as above, except this time we launch with the NFS client version of the script.

bash# zip -P "put your password here" ec2autorun.zip ec2autorun.script
bash# ec2-run-instances -k <my-key-pair> -f ec2autorun.zip ami-93b652fa

Summary

As you can see from this simple example, Parameterized Launches in EC2 is a very flexible and powerful technique for launching custom instances from a single AMI. In combination with S3 there are many other uses, for example: patching systems, adding users and groups, and running specific services at launch time.

All output from the scripts is captured and can be reviewed for troubleshooting purposes. Log in to one of your new instances and review the following.

bash# cd /var/ec2
bash# ls

# the zip file that was passed on the ec2-run-instance
# command line
ec2autorun.input

# stderr from the /etc/init.d/ec2autorun script
ec2autorun.input.stderr

# stdout from the /etc/init.d/ec2autorun script
ec2autorun.input.stdout

# the script that was bundled in the zip file,
# called by /etc/init.ec2atorun
ec2autorun.script

# stderr from ec2autorun.script
ec2autorun.script.stderr

# stdout from ec2autorun.script
ec2autorun.script.stdout



Tuesday Jul 01, 2008

Saving and Restoring ZFS Snapshots to and from Amazon S3

by Sean O'Dell

We can use ZFS snapshots to save and restore filesystems from one Solaris EC2 instance to another. This functionality is very useful, for example, for saving user home directories, web server documents, MySQL databases, etc., terminating a EC2 instance, and then restoring these filesystems on a new EC2 instance created at a later date.

Amazon Simple Storage Service (S3) provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites.

Getting Starting

In our example, we use the second EC2 ephemeral storage device, c0d1p0 (c4d1p0 for OpenSolaris 2008.05), to create a ZFS pool. We create filesystems within this pool for data that we want to restore for each new EC2 instance, and then save and restore the data using S3.

We use the S3 Sync command line tools listed in the references section to interface with S3. We assume the that reader has an AWS account and has a basic understanding of EC2, S3 and Solaris ZFS.

In this section we describe the steps necessary to get started using ZFS within the EC2 environment. For our example, we use the SXCE Build 79 32 bit image AMI:
ami-eb7a9f82. This example has been tested and works with any of the Sun provided Solaris AMIs, including OpenSolaris 2008.05.

Launch an instance of the Solaris AMI and login to the new instance.

bash# ec2-run-instances -k <my-key-pair> -t m1.small ami-eb7a9f82

bash# ec2-describe-instances | grep INSTANCE | cut -f2,3,4,6
i-196fb970  ami-eb7a9f82  ec2-75-101-225-153.compute-1.amazonaws-DOT-com  running

bash# ssh -i <my-key-pair-file> root-AT-ec2-75-101-225-153.compute-1.amazonaws.com
Last login: Mon Apr 28 02:24:21 2008
Sun Microsystems Inc.   SunOS 5.11      snv_79  January 2008

# bash -o vi

Download and install the S3 Sync tools. Install the tools in your preferred location. In our example, we use /usr/local/aws.

bash# mkdir -p /usr/local/aws
bash# cd /usr/local/aws

bash# /usr/sfw/bin/wget \
  http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz
bash# gzcat s3sync.tar.gz | tar xf -
bash# cd /usr/local/aws/s3sync
bash# ln -s s3cmd.rb s3cmd
bash# export PATH=$PATH:/usr/local/aws/s3sync

Please note that there is an additional step required if you are using the OpenSolaris 2008.05 AMI. This AMI does not have ruby installed. Since the S3 Sync tools require ruby, we need to install ruby if using the OpenSolaris 22008.05 AMI.

bash# which ruby
no ruby in /usr/sbin /usr/bin /usr/local/aws/s3sync

bash# pkg install SUNWruby18
bash# which ruby
/usr/bin/ruby

Set the correct time, set your AWS environment variables, and create a S3 bucket to store your saved ZFS snapshots.

bash# ntpdate 0.north-america.pool.ntp.org
bash# export AWS_ACCESS_KEY_ID=<my-aws-access-key>
bash# export AWS_SECRET_ACCESS_KEY=<my-aws-secret-access-key>
bash# s3cmd createbucket <my-bucket-name-for-zfs-snapshots>

We are now ready to create our ZFS pool and a few file systems. Logged in to the instance created and setup with the steps above, do the following.

Note: the extra disk for the OpenSolaris 2008.05 image is: c4d1p0. From this point forward, we use the SXCE Build 79 32 bit image in our examples. If using OpenSolaris 2008.05, substitute c4d1p0 for c0d1p0.

bash# zpool create s3pool c0d1p0
bash# zpool list
NAME     SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
s3pool   149G   111K   149G     0%  ONLINE  -

bash# for i in home mysqldb share www
do
    zfs create s3pool/$i