Tim Foster's Weblog

I'm honestly not sure the world needs another "here's how I solved my problem with Crossbow" blog post - it's been covered pretty well already by Nicholas, Chris, Ben, Joerg and likely many others, but still - adding to the collective knowledge of your search engine of choice, here's my version:

Original image by ToastyKen

Introduction

Support for virt-install to initiate PV and HVM installs of OpenSolaris guests using OpenSolaris AI (Automated Installer) is on it's way once our 3.3 wad integrates, and for the most part, I did that work on a standalone network of two machines. This meant that I had to go into the office every time I wanted to try something out. Of course, with Crossbow, I needn't have gone to that much trouble - so, in case anyone else finds themselves in the same situation, here's how to do it.

Setting up an AI server for testing on a network where you don't own the dhcp server can be a pain. However, given a .vdi that already houses the AI server, all you need is the right network configuration in your host to make testing AI + xVM easy - thanks Crossbow! I now just need to ship around a domU .vdi with my AI server on it, and the corresponding virsh xml definition and I can have it running on any machine in a jiffy.

The things you want are:

1. a dedicated etherstub in dom0 that you can create vnics from
2. vnics on that etherstub, each with a local network address, one of which is plumbed in your host
3. ipnat rules in dom0 to allow that local network out through your primary interface, and ssh into the guest on defined ports
4. ipfilter rules in dom0 to block stuff you don't want reaching the outside
5. ( and visa versa, perhaps providing port redirection into your guest )

Implementation

Here's what you need to run:

# dladm create-etherstub timswitch0
# dladm set-linkprop -p mtu=1500 timswitch0
# dladm create-vnic -l timswitch0 timnic0
# ifconfig timnic0 plumb 192.168.1.1 netmask 255.255.255.0 up
# (if necess.) ifconfig timnic1 plumb
   otherwise, pass virt-install timswitch0, and the vif-vnic script
   will create a vnic on top of it to plug guests into that etherstub

# routeadm -u -e ipv4-forwarding

then having edited /etc/ipf/ipf.conf and /etc/ipf/ipnat.conf (see below)

# svcadm enable ipfilter
# ipnat -f /etc/ipf/ipnat.conf

This gives us the following configuration:

# dladm show-link
LINK        CLASS    MTU    STATE    OVER
ath0        phys     1500   up       --
atge0       phys     1500   unknown  --
timswitch0  etherstub 1500  unknown  --
timnic1     vnic     1500   up       timswitch0
timnic0     vnic     1500   up       timswitch0


# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
	inet 127.0.0.1 netmask ff000000 
ath0: flags=1104843 mtu 1500 index 15
	inet 10.0.0.10 netmask ffffff00 broadcast 10.0.0.255
	ether 0:15:af:6a:64:ec 
atge0: flags=201100843 mtu 1500 index 16
	inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
	ether 0:1e:8c:bb:8e:7a 
timnic0: flags=1100843 mtu 1500 index 18
	inet 192.168.1.1 netmask ffffff00 broadcast 192.168.1.255
	ether 2:8:20:7a:17:4f 
lo0: flags=2002000849 mtu 8252 index 1
	inet6 ::1/128 
ath0: flags=2004841 mtu 1500 index 15
	inet6 fe80::215:afff:fe6a:64ec/10 
	ether 0:15:af:6a:64:ec 

Here's what you need in /etc/ipf. Replace 'ath0' with your primary interface. In ipnat.conf we redirect port 9022 to the ssh port in our guest on 192.168.1.2 (the guest containing the AI server) so to ssh in to the guest, do:

 # ssh -p 9022 <IP address of dom0>

----------- /etc/ipf/ipnat.conf --------------
map ath0 192.168.1.0/24 -> 0/32 portmap tcp/udp auto
map ath0 192.168.1.0/24 -> 0/32
rdr ath0 0.0.0.0/0 port 9022 -> 192.168.1.2 port 22
--

Various IP filter rules applied in dom0. At the moment, we're logging packets that aren't allowed through timnic0. Run 'ipmon' in dom0 to watch this log - useful when debugging (and choosing what other rules to add)

----------- /etc/ipf/ipf.conf -------------------
#
# ipf.conf
#
# IP Filter rules to be loaded during startup
#
# See ipf(4) manpage for more information on
# IP Filter rules syntax.

# allow ssh into our guest from to anywhere
pass out quick on timnic0 from any to any port = 22 keep state

# allow dns, http, ssh in from timnic0, remember we're using NAT here,
# and we're a router, so this is actually traffic from the guest network
pass in quick on timnic0 proto udp from any to any port = 53 keep state
pass in quick on timnic0 proto tcp from any to any port = 22 keep state
pass in quick on timnic0 proto tcp from any to any port = 80 keep state

# allow anything at all inside our guest network
pass in quick on timnic0 from 192.168.1.0/24 to 192.168.1.0/24
pass out quick on timnic0 from 192.168.1.0/24 to 192.168.1.0/24

# allow nothing else through timnic0
block in log quick on timnic0
block out log quick on timnic0
--

In a guest, we see:

root@opensolaris:/tmp# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
	inet 127.0.0.1 netmask ff000000 
xnf0: flags=1004843 mtu 1500 index 2
	inet 192.168.1.45 netmask ffffff00 broadcast 192.168.1.255
	ether 0:16:36:5f:7f:c5 
lo0: flags=2002000849 mtu 8252 index 1
	inet6 ::1/128 

root@opensolaris:/tmp# netstat -rn

Routing Table: IPv4
  Destination           Gateway           Flags  Ref     Use     Interface 
-------------------- -------------------- ----- ----- ---------- --------- 
default              192.168.1.1          UG        1         76 xnf0      
192.168.1.0          192.168.1.45         U         1         31 xnf0      
127.0.0.1            127.0.0.1            UH        1         28 lo0       

Routing Table: IPv6
  Destination/Mask            Gateway                   Flags Ref   Use    If   
--------------------------- --------------------------- ----- --- ------- ----- 
::1                         ::1                         UH      1       0 lo0 

Update: had a typo in one of the commands above, thanks for spotting it seanmcg