Running OpenSolaris and Zones in the Amazon Cloud - Part 1
Introduction
Now that OpenSolaris 2009.06 is available on Amazon EC2, I have been interested in setting up zones within a OpenSolaris EC2 instances utilizing the virtual networking features provided by Crossbow.
In this tutorial I will provide a step-by-step guide describing how to get this environment up and running. We use Crossbow together with NAT to build a complete virtual network connecting multiple Zones within a Solaris EC2 host.
Much of the networking information used in this tutorial is taken directly from the excellent article Private virtual networks for Solaris xVM and Zones with Crossbow by Nicolas Droux.
This is Part 1 of this tutorial series. In Part 2 we will explain how to use ZFS and AWS snapshots to backup the zones. In Part 3 we will explain how to save a fully configured environment using a AMI and EBS snapshots, which can then be cloned and up and running in minutes.
Prerequisites
-
Basic understanding of AWS EC2 including: managing AMIs, launching instances, managing EBS volumes and snapshots, and firewall management.
-
Basic understanding of OpenSolaris including system setup, networking, and zone management.
Building the EC2 environment
For this tutorial, I used the OpenSolaris 2009.06 AMI
ami-e56e8f8c. I also created three EBS volumes, one for shared software, one for zones storage, and another one for zones backup. In Part 2 of this tutorial, I will explain the use of ZFS snapshots and EBS snapshots for the purposes of backing up the zones. The EC2 environment is displayed below.
A summary of the steps are as follows:
-
Launch a new EC2 instance from AMI ami-e56e8f8c.
-
Create the EBS volumes and attach them to the instance.
-
Create ZFS pools, one for the shared software, one for zones, and one for zones backup.
When finished, I have a OpenSolaris 2009.06 EC2 instance running with three ZFS filesystems on top of three EBS volumes as shown below.
|
root:~# zfs list -r sharedsw root:~# zfs list -r zones root:~# zfs list -r zones-backup |
Building the private network
The next task is to create the virtual network as shown in the diagram below.
We create an etherstub and three VNICs for our virtual network.
|
root:~# dladm show-phys |
Assign a static IP address to vnic0 in the global zone:
|
root:~# ifconfig vnic0 plumb |
Note that the usual configuration variables (e.g. /etc/hostname.
|
root:~# routeadm -u -e ipv4-forwarding |
Next, we enable NAT on the xnf0 interface. We also want to be able to connect to the zones from the public internet, so we enable port forwarding. In EC2 make sure you open these ports within the EC2 firewall, following best practices.
| root:~# cat /etc/ipf/ipnat.conf map xnf0 192.168.0.0/24 -> 0/32 portmap tcp/udp auto map xnf0 192.168.0.0/24 -> 0/32 rdr xnf0 0.0.0.0/0 port 22101 -> 192.168.0.101 port 22 rdr xnf0 0.0.0.0/0 port 22102 -> 192.168.0.102 port 22 rdr xnf0 0.0.0.0/0 port 8081 -> 192.168.0.101 port 80 rdr xnf0 0.0.0.0/0 port 8082 -> 192.168.0.102 port 80 rdr xnf0 0.0.0.0/0 port 40443 -> 192.168.0.102 port 443 root:~# |
Creating the zones
Create and install zone1
|
root:~# zonecfg -z zone1 Next Steps: Boot the zone, then log into the zone console root:~#
|
Create and install zone2
|
root:~# root:~# zonecfg -z zone2 zone2: No such zone configured Use 'create' to begin configuring a new zone. zonecfg:zone2> create zonecfg:zone2> set zonepath=/zones/zone2 zonecfg:zone2> set ip-type=exclusive zonecfg:zone2> add net zonecfg:zone2:net> set physical=vnic2 zonecfg:zone2:net> end zonecfg:zone2> add fs zonecfg:zone2:fs> set dir=/opt zonecfg:zone2:fs> set special=/opt zonecfg:zone2:fs> set type=lofs zonecfg:zone1:fs> add options ro zonecfg:zone2:fs> end zonecfg:zone2> verify zonecfg:zone2> commit zonecfg:zone2> exit root:~# root:~# zoneadm -z zone2 install A ZFS file system has been created for this zone. Publisher: Using opensolaris.org (http://pkg.opensolaris.org/release/). Image: Preparing at /zones/zone2/root. Cache: Using /var/pkg/download. Sanity Check: Looking for 'entire' incorporation. Installing: Core System (output follows) Postinstall: Copying SMF seed repository ... done. Postinstall: Applying workarounds. Done: Installation completed in 125.975 seconds. Next Steps: Boot the zone, then log into the zone console (zlogin -C) to complete the configuration process root:~# |
Zone configuration
Now that the zones are installed, we are ready to boot each zone and perform system configuration. First we boot each zone.
|
root:~# zoneadm -z zone1 boot |
The next step is to connect to the console for each zone and perform system configuration. The configuration params that I used are listed below. Connect to the console with the command "zlogin -C zone_name", for example: zlogin -C zone1
|
zone1 zone2 |
Test connection to the zones
Once the zones are running and have been configured, we should be able to connect to the zones from the "outside". This connection test is dependent on having the EC2 firewall correctly setup. In the example below, we connect to zone1 via port 22101.
|
-bash-3.2$ ssh ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com -p 22101 -bash-3.2$ hostname |
Hi Sean ODell,
did you test the performance on the virtual
windows-guest inside the solaris host ?
regards
Danny
Posted by Daniel on September 11, 2009 at 12:53 AM PDT #
Hi Danny,
No, I did not run a Windows guest inside the Solaris host. We are running OpenSolaris zones.
Regards,
Sean
Posted by Sean O'Dell on September 11, 2009 at 11:58 PM PDT #
I noticed your source IP range is wide open. If you wanted to lock that down, would you add a line to ipf.conf to push the packets from xnf0 to vnic0, and then allow all from vnic0 to your various ssh?
ie
pass in quick on xnf0 fastroute vnic0 from w.x.y.z to any
Having the extra layer between the originating outside source IP seems to be confusing me.
public -> EC2 internal IP -> xnf0 -> vnic0
How do you appropriately handle the NAT to limit the public source? Or is that best left at the EC2 instance security policy to handle?
Posted by Adam Levine on September 13, 2009 at 01:58 AM PDT #
I did some playing around, with some headaches, and I found a few things.
1) something in the OS ami doesn't like fastroute. must use 'to'. Once I changed that, my rules seem to work. (It's quite possible I had a munged line before s/fastroute/to, but I do recall that being the only change I made and I no longer got syntax errors.
Putting things in the cloud has caused me to overthink things. So, to answer my own question:
ipf.conf:
pass in all
block in on xnf0 from any to any port=XXX
pass in on xnf0 from X.Y.Z.Z to any port=XXX
pass out all
ipnat.conf
rdr xnf0 0.0.0.0/0 port XXX -> 192.168.1.4 port XXX tcp
Posted by Adam Levine on September 13, 2009 at 03:29 PM PDT #
Hi Adam,
Thank you for your comments and follow up. In my example I did not use ipfilter for the firewall and instead relied on the EC2 firewall. In your example, I believe that you will still need to open the EC2 firewall for the ports that you will use to connect from the Internet.
Regards,
Sean
Posted by Sean O'Dell on September 14, 2009 at 07:48 AM PDT #
When is the 2nd installment coming? soon please!
Posted by jesse sanford on September 18, 2009 at 11:16 PM PDT #
Please see the following for Part 2:
http://blogs.sun.com/seano/entry/running_opensolaris_and_zones_in
Posted by Sean O'Dell on September 22, 2009 at 07:44 AM PDT #
Cool work. Very similar in fact to what I have been doing with a security focus on the Immutable Service Container project (http://kenai.com/projects/isc/pages/OpenSolaris). Just posted ISCs on EC2 at: http://blogs.sun.com/gbrunett/entry/immutable_service_containers_on_amazon. From the look of things, it seems like you could leverage this work to improve your out of the box security configuration.
Posted by Glenn Brunette on September 24, 2009 at 12:49 PM PDT #
Hi Glenn,
Thanks for the tip. I'll take a look at the ISC project.
Regards,
Sean
====
Posted by Sean O'Dell on September 27, 2009 at 08:53 PM PDT #
in the ipnat step - how do you know the interface name is xnf0? i usually get the interface name from ifconfig and in this case, it would show: vnic0. what is the correct way to get the interface name?
Posted by david on November 02, 2009 at 05:25 PM PST #
David, xnf0 is the default (standard) name for the physical NIC on OpenSolaris when deployed on EC2.
Posted by Glenn Brunette on November 02, 2009 at 06:04 PM PST #
sorry - xnf0 was underneath the staple holding the pages together. thanks for helping me.
Posted by david on November 02, 2009 at 07:01 PM PST #
I had some problems with this set-up: although some incoming traffic worked, large TCP packets split into multiple IP segments were not getting throught the vnic bridge. This was particularly noticeable whenever doing large HTTP transfers from a client running in the bridged zone. I saw this with wget, curl, and mercurial as HTTP clients. Snoop running on the ec2 global zone showed traffic abruptly halting after the first segment.
The solution seems to be to match MTU between the xnf0 and vnic interfaces (setting both to 1500).
My EBS volume which hosts the zones also has a "boot.sh" script to start the etherstub bridge and vnics, copy /etc/zones and /etc/ipf to the newly instantiated opensolaris global zone, etc. This script now includes "mtu 1500" in the ifconfig line for vnic0. Within the non-global zone(s) it's handled by /etc/hostname.vnic1, etc.
Thanks to Darren Reed for tracking the solution: http://marc.info/?l=ipfilter&m=125800661125402&w=2
Posted by David Champion on November 12, 2009 at 02:02 PM PST #
Hi David,
Thanks for sharing the solution. Works for me as well. Used you mercurial hg clone example as a test.
Regards,
Sean
====
Posted by Sean O'Dell on November 13, 2009 at 04:56 AM PST #