Configuring LXC using libvirt

In a previous post I showed how to configure LXC and cgroups by using user-space tools, a rather laborious process. In this post I'll demonstrate a much easier way of say running Apache in a container by using libvirt.
libvirt is a toolkit to interact with the virtualization capabilities of recent versions of Linux [1] and a set of API bindings for common languages like Python. It is widely used for managing KVM and XEN instances, but it can be used with the same success by LXC, though still not quite secure as it should be, but something that can be worked around by using SELinux.
For this example I'll be using CentOS 6.3, but I've tested this on Debian derivatives and it should works just the same.
First let's install and run the libvirt package and daemon, if not done so already:

	[root@host1 ~]# yum install -y libvirt
	[root@host1 ~]# service libvirtd start

view raw gistfile1.sh hosted with ❤ by GitHub

Next lets create the file-system for the container. In this example I'll use a custom /etc and /var/www, the / and all other directories will be visible and used from the host OS - not the most secure way, but it's the simplest:

	[root@host1 ~]# mkdir -p /containers/http1/etc
	[root@host1 ~]# mkdir -p /containers/http1/var/www/html
	[root@host1 ~]# mkdir -p /containers/http1/etc/httpd/conf
	[root@host1 ~]# mkdir -p /containers/http1/etc/sysconfig

view raw gistfile1.sh hosted with ❤ by GitHub

Install Apache on the host system, but don't run it, then copy the config file to the /etc in the container:

	[root@host1 ~]# yum install -y httpd
	[root@host1 ~]# cp -r /etc/httpd/conf /containers/http1/etc

view raw gistfile1.sh hosted with ❤ by GitHub

To be able to run Apache in the container, or any other services that requite networking, we need to copy few more files:

	[root@host1 ~]# cp -r /etc/passwd /containers/http1/etc/
	[root@host1 ~]# cp -r /etc/shadow /containers/http1/etc/
	[root@host1 ~]# cp -r /etc/group /containers/http1/etc/
	[root@host1 ~]# cp -r /etc/mime.types /containers/http1/etc/
	[root@host1 ~]# cp -r /etc/sysconfig/network /containers/http1/etc/sysconfig/
	[root@host1 ~]# cp -r /etc/sysconfig/network-scripts /containers/http1/etc/sysconfig/
	[root@host1 ~]# cp -r /etc/init.d/ /containers/http1/etc/
	[root@host1 ~]# cp -r /etc/resolv.conf /containers/http1/etc/
	[root@host1 ~]# cp -r /etc/rc.d/ /containers/http1/etc/
	[root@host1 ~]# cp -r /etc/httpd/* /containers/http1/etc/httpd/

view raw gistfile1.sh hosted with ❤ by GitHub

Configure your networking:

	[root@host1 ~]# cat /containers/http1/etc/sysconfig/network-scripts/ifcfg-eth0
	DEVICE=eth0
	BOOTPROTO=static
	IPADDR=192.168.122.50
	NETMASK=255.255.255.0
	DEFROUTE=yes
	GATEWAY=192.168.122.1
	DNS1=192.168.1
	ONBOOT=yes
	NM_CONTROLLED=no
	[root@host1 ~]# echo "nameserver 192.168.122.1" > /containers/http1/etc/resolv.conf

view raw gistfile1.sh hosted with ❤ by GitHub

Create an index page for the web server, if desired:

[root@host1 ~]# echo "Apache in LXC" >> /containers/http1/var/www/html/index.html

view raw gistfile1.sh hosted with ❤ by GitHub

Now that we have the file system laid out for the container, let's create the XML file for libvirt to use:

	[root@host1 ~]# cat http1.xml
	<domain type='lxc'>
	<name>http1</name>
	<memory>102400</memory>
	<os>
	<type>exe</type>
	<init>/bin/startup.sh</init>
	</os>
	<vcpu>1</vcpu>
	<on_poweroff>destroy</on_poweroff>
	<on_reboot>restart</on_reboot>
	<on_crash>destroy</on_crash>
	<devices>
	<emulator>/usr/libexec/libvirt_lxc</emulator>
	<filesystem type='mount'>
	<source dir='/containers/http1/etc/httpd/conf/'/>
	<target dir='/etc/httpd/conf'/>
	</filesystem>
	<filesystem type='mount'>
	<source dir='/containers/http1/var/www/html/'/>
	<target dir='/var/www/html'/>
	</filesystem>
	<filesystem type='mount'>
	<source dir='/containers/http1/etc/'/>
	<target dir='/etc'/>
	</filesystem>
	<interface type='network'>
	<source network='default'/>
	</interface>
	<console type='pty'/>
	</devices>
	</domain>

view raw gistfile1.xml hosted with ❤ by GitHub

This is a pretty standard libvirt configuration file if you are already familiar with XEN, or KVM in libvirt. We define the name, how many CPU cores and memory to be allocated to the container (managed by cgroups), along with what file-system resources to mount inside the container. In the <emulator> element at line 14, be sure you specify the correct path to libvirt_lxc, if it does not live in /usr/libexec on your system.
One important difference is the <init> tag. This tells the container to execute this script when LXC starts. The file contains the following:

	[root@host1 ~]# cat /bin/startup.sh
	#!/bin/bash
	export PATH=$PATH:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
	export PS1="[\u@\h \W]\\$"
	echo "Starting Networking" >> /var/log/messages
	/etc/init.d/network start
	echo "Starting httpd" >> /var/log/messages
	/etc/init.d/httpd start
	/bin/bash

view raw gistfile1.sh hosted with ❤ by GitHub

Make sure it's executable:

[root@host1 ~]# chmod u+x /bin/startup.sh

view raw gistfile1.sh hosted with ❤ by GitHub

It basically starts the networking, Apache and gives you back a bash shell to work with. Save the file and define the container:

[root@host1 ~]# virsh --connect lxc:/// define http1.xml

view raw gistfile1.sh hosted with ❤ by GitHub

The libvirt LXC driver requires that certain cgroups controllers are mounted on the host OS. The minimum required controllers are 'cpuacct', 'memory' and 'devices', while recommended extra controllers are 'cpu', 'freezer' and 'blkio'. The /etc/cgconfig.conf & cgconfig init service used to mount cgroups at host boot time. Make sure the cgconfig daemon is runnig and the /cgroup is mounted. In some cases you have to reboot the server to make cgroups work:

	[root@host1 ~]# /etc/init.d/cgconfig start
	[root@host1 ~]# /etc/init.d/cgconfig status
	Running
	[root@host1 ~]# ls -la /cgroup/
	total 12
	drwxr-xr-x 10 root root 4096 Feb 4 18:59 .
	drwxr-xr-x. 24 root root 4096 Feb 4 18:59 ..
	drwxr-xr-x 3 root root 0 Feb 4 18:59 blkio
	drwxr-xr-x 3 root root 0 Feb 4 18:59 cpu
	drwxr-xr-x 3 root root 0 Feb 4 18:59 cpuacct
	drwxr-xr-x 3 root root 0 Feb 4 18:59 cpuset
	drwxr-xr-x 3 root root 0 Feb 4 18:59 devices
	drwxr-xr-x 3 root root 0 Feb 4 18:59 freezer
	drwxr-xr-x 3 root root 0 Feb 4 18:59 memory
	drwxr-xr-x 2 root root 0 Feb 4 18:59 net_cls

view raw gistfile1.sh hosted with ❤ by GitHub

To start the container execute:

[root@host1 ~]# virsh --connect lxc:/// start http1

view raw gistfile1.sh hosted with ❤ by GitHub

To list all available containers run:

	[root@host1 ~]# virsh --connect lxc:/// list --all
	Id Name State
	----------------------------------------------------
	1799 http1 running

view raw gistfile1.sh hosted with ❤ by GitHub

To login to the container using the console run:

	[root@host1 ~]# virsh --connect lxc:/// console http1
	[root@host1 ~]# ps axf
	PID TTY STAT TIME COMMAND
	1 pts/3 Ss 0:00 /bin/bash /bin/startup.sh
	201 ? Ss 0:00 /usr/sbin/httpd
	204 ? S 0:00 \_ /usr/sbin/httpd
	205 ? S 0:00 \_ /usr/sbin/httpd
	206 ? S 0:00 \_ /usr/sbin/httpd
	207 ? S 0:00 \_ /usr/sbin/httpd
	208 ? S 0:00 \_ /usr/sbin/httpd
	209 ? S 0:00 \_ /usr/sbin/httpd
	210 ? S 0:00 \_ /usr/sbin/httpd
	211 ? S 0:00 \_ /usr/sbin/httpd
	203 pts/3 S 0:00 /bin/bash
	218 pts/3 R+ 0:00 \_ ps axf

view raw gistfile1.sh hosted with ❤ by GitHub

Logs go to:

	[root@host1 ~]# tail -f /var/log/libvirt/lxc/http1.log
	PATH=/sbin:/usr/sbin:/bin:/usr/bin LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/libexec/libvirt_lxc --name http1 --console 19 --handshake 23 --background --veth veth2
	PATH=/bin:/sbin TERM=linux container=lxc-libvirt LIBVIRT_LXC_UUID=333cce9a-047f-5e27-d2f0-3cd4e06dd182 LIBVIRT_LXC_NAME=http1 /bin/startup.sh

view raw gistfile1.sh hosted with ❤ by GitHub

One thing to note is that after you exit the console the container will terminate. You can work around this by either starting SSH in the container in a similar way Apache is started, or just disconnect from the session.

You can see the running container process that was started by libvrit_lxc:

	[root@host1 ~]# ps axfw
	[truncated]
	1799 ? Ss 0:00 /usr/libexec/libvirt_lxc --name http1 --console 19 --handshake 23 --background --veth veth2
	1808 pts/6 Ss 0:00 \_ /bin/bash /bin/startup.sh
	2122 ? Ss 0:00 \_ /usr/sbin/sshd
	2131 pts/6 S+ 0:00 \_ /bin/bash
	2878 ? Ss 3:30 \_ /usr/sbin/httpd
	24010 ? S 0:02 \_ /usr/sbin/httpd
	26204 ? S 0:02 \_ /usr/sbin/httpd
	1466 ? S 0:02 \_ /usr/sbin/httpd
	6785 ? S 0:01 \_ /usr/sbin/httpd
	24968 ? S 0:01 \_ /usr/sbin/httpd
	31661 ? S 0:01 \_ /usr/sbin/httpd
	[truncated]

view raw gistfile1.sh hosted with ❤ by GitHub

To see the memory limit for the container we just created execute:

	[root@host1 ~]# cat /cgroup/memory/libvirt/lxc/http1/memory.limit_in_bytes
	104857600

view raw gistfile1.sh hosted with ❤ by GitHub

You can change this on the fly as well. To read more about cgroups you can read my previous post on that topic.

Apache should be accessible now in the container:

	[root@host1 ~]# curl 192.168.122.50
	Apache in LXC

view raw gistfile1.sh hosted with ❤ by GitHub

Here's a python example creating a container with HAProxy in it:

	#!/usr/bin/python
	'''
	1. Install libvirt
	2. Install libvirt-python (RedHat) python-libvirt(Debian)
	3. Install haproxy
	4. Create container file system that includes network configs, haproxy configs,
	haproxy service
	5. Change BASE_CONTAINER path to lcoation of the template file system
	6. Change CONTAINER_PATH to the location where you want the lxc filesystem
	to be.
	'''
	import libvirt
	import os
	import argparse
	import random
	class Paths(object):
	CONTAINER_PATH = '/containers/{0}'
	CONTAINER_ETC = CONTAINER_PATH + '/etc'
	CONTAINER_INIT_D = CONTAINER_ETC + '/init.d'
	CONTAINER_NETWORK = CONTAINER_ETC + '/network'
	CONTAINER_HAPROXY = CONTAINER_ETC + '/haproxy'
	BASE_CONTAINER = '~/containers_base'
	def create_domain(name):
	container_path = Paths.CONTAINER_PATH.format(name)
	container_etc = Paths.CONTAINER_ETC.format(name)
	os.system('mkdir -p {0}'.format(container_path))
	os.system('cp -r {0}/* {1}'.format(Paths.BASE_CONTAINER, container_etc))
	domain_xml = '''
	<domain type="lxc">
	<name>{0}</name>
	<memory>102400</memory>
	<currentmemory>102400</currentmemory>
	<vcpu>1</vcpu>
	<os>
	<type arch="x86_64">exe</type>
	<init>/bin/startup.sh</init>
	</os>
	<clock offset="utc">
	<on_poweroff>destroy</on_poweroff>
	<on_reboot>restart</on_reboot>
	<on_crash>destroy</on_crash>
	<devices>
	<emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
	<filesystem accessmode="passthrough" type="mount">
	<source dir="{1}/etc/network"></source>
	<target dir="/etc/network">
	</target></filesystem>
	<filesystem accessmode="passthrough" type="mount">
	<source dir="{1}/etc/init.d"></source>
	<target dir="/etc/init.d">
	</target></filesystem>
	<filesystem accessmode="passthrough" type="mount">
	<source dir="{1}/etc/haproxy"></source>
	<target dir="/etc/haproxy">
	</target></filesystem>
	<interface type="network">
	<source network="default"></source>
	</interface>
	<console type="pty">
	<target port="0" type="lxc">
	</target></console>
	</devices>
	</clock></domain>
	'''
	domain_xml = domain_xml.format(name, container_path)
	lxc_conn = libvirt.open('lxc:///')
	domain = lxc_conn.defineXML(domain_xml)
	if domain.create() == 0:
	print '{0} domain created successfully.'.format(name)
	return domain
	else:
	print '{0} did NOT create successfully.'.format(name)
	def delete_domain(domain):
	try:
	domain.destroy()
	domain.undefine()
	except Exception as destroy_exception:
	try:
	domain.undefine()
	except Exception as undefine_exception:
	print undefine_exception
	print 'Could not clean up domain.'
	def delete_domain_by_name(name):
	lxc_conn = libvirt.open('lxc:///')
	try:
	domain = lxc_conn.lookupByName(name)
	except:
	print 'Failed to find the domain {0}'.format(name)
	return
	delete_domain(domain)
	if __name__ == '__main__':
	parser = argparse.ArgumentParser()
	parser.add_argument('container_name', metavar='container_name', type=str,
	help='Name of container to create.')
	parser.add_argument('-d', '--delete', action='store_true',
	help='Delete container name specified.')
	args = parser.parse_args()
	if args.delete:
	delete_domain_by_name(args.container_name)
	else:
	create_domain(args.container_name)

view raw gistfile1.py hosted with ❤ by GitHub

Resources:

[1] http://libvirt.org/