Connecting KVM or LXC to Open vSwitch

In this article I'll demonstrate how to install Open vSwitch and directly connect an LXC virtual interface to a switch port on it, instead to the standard bridge that comes build-in with most kernels.

As stated in the OVS documentation, hypervisors need the ability to bridge traffic between VMs and the outside world. On Linux-based hypervisors, this used to mean using the built-in L2 switch  - the Linux bridge.

Open vSwitch is targeted at multi-server virtualization deployments where VM mobility and network dynamics are important.

Open vSwitch supports a number of features that allow a network control system to respond and adapt as the environment changes. This includes simple accounting and visibility support such as NetFlow and sFlow. But perhaps more useful, Open vSwitch supports a network state database (OVSDB) that supports remote triggers. Therefore, a piece of orchestration software can "watch" various aspects of the network and respond if/when they change. This is used heavily today, for example, to respond to and track VM migrations.

Open vSwitch also supports OpenFlow as a method of exporting remote access to control traffic. There are a number of uses for this including global network discovery through inspection of discovery or link-state traffic (e.g. LLDP, CDP, OSPF, etc.).

The goal with Open vSwitch is to keep the in-kernel code as small as possible (as is necessary for performance) and to re-use existing subsystems when applicable (for example Open vSwitch uses the existing QoS stack).

For more information on OVS refer to [1].

Most Linux distributions now come with the OVS user-space tools and the kernel module, but I prefer to get the latest code and compile it manually (the source code also comes with spec files for building rpm or deb packages).

First lets download and compile the code:

	[root@hype1:src]# cd /usr/src
	[root@hype1:src]# wget http://openvswitch.org/releases/openvswitch-1.9.0.tar.gz
	[root@hype1:src]# tar zxfv openvswitch-1.9.0.tar.gz; cd openvswitch-1.9.0
	[root@hype1:openvswitch-1.9.0]# ./congiure;make;make install;cd

view raw gistfile1.sh hosted with ❤ by GitHub

Alternatively you can build the kernel modules (openvswitch.ko and brcompat.ko) and the user-space tools as packages and install them:

	[root@hype1:~]# cd
	[root@hype1:~]# mkdir -p rpmbuild/SOURCES
	[root@hype1:~]# cd rpmbuild/SOURCES
	[root@hype1:SOURCES]# wget http://openvswitch.org/releases/openvswitch-1.9.0.tar.gz
	[root@hype1:SOURCES]# tar xvfz openvswitch-1.9.0.tar.gz
	[root@hype1:SOURCES]# cd openvswitch-1.9.0
	[root@hype1:SOURCES]# ./configure
	[root@hype1:SOURCES]# make dist
	[root@hype1:SOURCES]# cp openvswitch-1.9.0.tar.gz ../
	[root@hype1:SOURCES]# rpmbuild -bb rhel/openvswitch-kmod-rhel6.spec
	[root@hype1:SOURCES]# rpmbuild -bb rhel/openvswitch.spec
	[root@hype1:SOURCES]# yum localinstall ~/rpmbuild/RPMS/x86_64/kmod-openvswitch-1.9.0-1.el6.x86_64.rpm
	[root@hype1:SOURCES]# yum localinstall ~/rpmbuild/RPMS/x86_64/openvswitch-1.10.90-1.x86_64.rpm

view raw gistfile1.sh hosted with ❤ by GitHub

Remove the bridge module if loaded before loading the openvswitch one:

[root@hype1:~]# rmmod bridge

view raw gistfile1.sh hosted with ❤ by GitHub

Load the openvswitch module:

[root@hype1:~]# modprobe openvswitch

view raw gistfile1.sh hosted with ❤ by GitHub

As of this writing I was only able to start my LXC container by first loading the brcompat compatibility module:

[root@hype1:~]# modprobe brcompat

view raw gistfile1.sh hosted with ❤ by GitHub

Without this module loaded I was getting the following error:

	[root@hype1:~]# virsh -c lxc:// start haproxy1
	error: Failed to start domain haproxy1
	error: Unable to add bridge br0 port veth0: Operation not supported

view raw gistfile1.sh hosted with ❤ by GitHub

Load the OVS modules at boot and blacklist the bridge module (on RHEL):

	[root@hype1:~]# cat /etc/sysconfig/modules/openvswitch.modules
	#!/bin/sh
	exec /sbin/modprobe openvswitch
	exec /sbin/modprobe brcompat
	[root@hype1:~]# chmod u+x /etc/sysconfig/modules/openvswitch.modules
	[root@hype1:~]# echo "blacklist bridge" >> /etc/modprobe.d/blacklist.conf

view raw gistfile1.sh hosted with ❤ by GitHub

Initialize the configuration database using ovsdb-tool:

	[root@hype1:~]# mkdir -p /usr/local/etc/openvswitch
	[root@hype1:~]# ovsdb-tool create /usr/local/etc/openvswitch/conf.db vswitchd/vswitch.ovsschema

view raw gistfile1.sh hosted with ❤ by GitHub

Before starting ovs-vswitchd itself, you need to start its configuration database, ovsdb-server. Each machine on which Open vSwitch is installed should run its own copy of ovsdb-server.

	[root@hype1:~]# ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \
	--remote=db:Open_vSwitch,manager_options \
	--private-key=db:SSL,private_key \
	--certificate=db:SSL,certificate \
	--bootstrap-ca-cert=db:SSL,ca_cert \
	--pidfile --detach

view raw gistfile1.sh hosted with ❤ by GitHub

If you built Open vSwitch without SSL support, then omit --private-key, --certificate, and --bootstrap-ca-cert.

Then initialize the database using ovs-vsctl. This is only necessary the first time after you create the database with ovsdb-tool (but running it at any time is harmless):

[root@hype1:~]# ovs-vsctl --no-wait init

view raw gistfile1.sh hosted with ❤ by GitHub

Then start the main Open vSwitch daemon, telling it to connect to the same Unix domain socket:

[root@hype1:~]# ovs-vswitchd --pidfile --detach

view raw gistfile1.sh hosted with ❤ by GitHub

If you build the openvswitch user-space tools from the packages as I showed above, alternatively you can start OVS from the init script:

[root@hype1:~]# service openvswitch start

view raw gistfile1.sh hosted with ❤ by GitHub

Time to create the bridge and add ports to it:

	[root@hype1:~]# ovs-vsctl add-br br0
	[root@hype1:~]# ovs-vsctl add-port br0 eth2
	[root@hype1:~]# ovs-vsctl add-port br0 veth0
	[root@hype1:~]# ifconfig eth2 0
	[root@hype1:~]# ifconfig br0 10.24.6.213 netmask 255.255.240.0
	[root@hype1:~]# route add default gw 10.24.0.1 br0

view raw gistfile1.sh hosted with ❤ by GitHub

This will add eth2 which is my public interface and veth0 which is a virtual interface of one of my LXC containers to the OVS bridge.

To directly connect your VM to the bridge and not use NAT make sure your VM network definition looks similar to this:

	[root@hype1:~]# cat lxc.xml
	--snip--
	<interface type='bridge'>
	<mac address='52:54:00:cb:ee:4b'/>
	<source bridge='br0'/>
	</interface>
	--snip--

view raw gistfile1.sh hosted with ❤ by GitHub

Now your VM is connected to the OVS and can talk to whatever network is connected to eth2, as long as it's configured in the same subnet.
To see the ports that are connected to the bridge run:

	[root@hype1:~]# ovs-vsctl show
	95098a68-1281-4dab-90d7-46374a5d5e23
	Bridge "br0"
	Port "veth0"
	Interface "veth0"
	Port "eth2"
	Interface "eth2"
	Port "br0"
	Interface "br0"
	type: internal

view raw gistfile1.sh hosted with ❤ by GitHub

The network configuration should look like the following (on RHEL):

	[root@hype1:~]# cat /etc/sysconfig/network-scripts/ifcfg-br0
	DEVICE=br0
	BOOTPROTO=static
	IPADDR=10.24.6.213
	NETMASK=255.255.240.0
	GATEWAY=10.24.0.1
	ONBOOT=yes
	TYPE="OVSBridge"
	DEVICETYPE="ovs"
	[root@hype1:~]# cat /etc/sysconfig/network-scripts/ifcfg-eth2
	DEVICE="eth2"
	HWADDR="00:8C:FA:00:C6:00"
	NM_CONTROLLED="no"
	ONBOOT="yes"
	DEVICETYPE="ovs"
	OVS_BRIDGE=br0

view raw gistfile1.sh hosted with ❤ by GitHub

Note the last two lines, that specify what kernel module to use.

Resources:
[1] http://openvswitch.org/