Virtualization (LXC Containers)

As of Fall 2009, we use Linux containers to maintain virtual machines, most notably caffeine, which is hosted on glomag. The various commands to manipulate Linux containers are prefixed with "lxc-"; see their individual manpages for usage.

Management Quick Guide

To manage containers, use the lxc-* tools, which require root privilege. Some examples (replace caffeine with the appropriate container name):

# check if caffeine is running
lxc-info -n caffeine

# start caffeine in the background
lxc-start -d -n caffeine

# stop caffeine gracefully
lxc-halt -n caffeine

# stop caffeine forcefully
lxc-stop -n caffeine

# launch a TTY console for the container
lxc-console -n caffeine

To install Linux container support on a recent Debian (squeeze or newer) system:

• Install the lxc and bridge-utils packages.
• Create a bridged network interface (this can be configured in /etc/network/interfaces as though it were a normal Ethernet device, with the additional bridge_ports parameter. This is usually called br0 (can be created manually with brctl). LXC will create a virtual Ethernet device and add it to the bridge when each container starts.

To start caffeine, run the following command as root on glomag:

lxc-start -d -n caffeine

Containers are stored on the host filesystem in /var/lib/lxc (root filesystems are symlinked to the appropriate directory on /vm).

ehashman's Guide to LXC on Debian

Configuring the host machine

First, install all required packages:

# apt-get install lxc bridge-utils

Setting up ethernet bridging

Next, create an ethernet bridge for the container. Edit /etc/network/interfaces:

# The primary network interface
#auto eth0
#iface eth0 inet static
#       address 129.97.134.200
#       netmask 255.255.255.0
#       gateway 129.97.134.1

# Bridge ethernet for containers
auto br0
iface br0 inet static
    bridge_ports eth0
    address 129.97.134.200
    netmask 255.255.255.0
    gateway 129.97.134.1
    dns-nameservers 129.97.2.1 129.97.2.2
    dns-search wics.uwaterloo.ca uwaterloo.ca

Cross your fingers and restart networking for your configuration to take effect!

# ifdown br0 && ifup br0
// bash enter to see if you lost connectivity and have to make a machine room trip

Note: !!! Do not use !!!

# service networking restart

The init scripts are broken and this likely will result in a machine room trip (or IPMI power cycle).

Setting up storage

Last, allocate some space in your volume group to put the container root on:

// Find the correct volume group to put the container on
# vgdisplay

// Create the volume in the appropriate volume group
# lvcreate -L 20G -n container vg0

// Find it in the dev mapper
# ls /dev/mapper/

// Create a filesystem on it
# mkfs.ext4 /dev/mapper/vg0-container

// Add a mount point
# mkdir /vm/container 

Last, add it to /etc/fstab:

/dev/mapper/vg0-container /vm/container        ext4    defaults        0       2

Test the entry with mount:

# mount /vm/container

Now you're done!

Creating a new container

Create a new container using lxc-create:

// Create new container "container" with root fs located at /vm/container
# lxc-create --dir=/vm/container -n container --template download

This will prompt you for distribution, release, and architecture. (Architecture must match host machine.)

Take this time to review its config in /var/lib/lxc/container/config, and tell it to auto-start if you like:

# Auto-start the container on boot
lxc.start.auto = 1

You'll also want to set up networking (if applicable):

# Networking
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.name = eth0
lxc.network.hwaddr = de:ad:be:ef:ba:be  # or something sensible

Now,

// List containers, -f for fancy
# lxc-ls -f

to ensure that your container has been successfully created; it should be listed. You can also list its root directory if you like. To start it in the background and obtain a root shell, do

// Start and attach a root shell
# lxc-start -d -n container
# lxc-attach -n container

Migrating a container between hosts

Start by shutting the container down:

root@container:~# halt

Then make a tarball of the container's filesystem:

# tar --numeric-owner -czvf container.tar.gz /vm/container

Copy it to its target destination, along with the configs:

$ scp container.tar.gz new-host:
$ scp -r /var/lib/lxc/container/ new-host:/var/lib/lxc/

Now carefully extract it. If you haven't already, provision storage and ethernet per the container creation section.

Yes, we really do want to stick it directly into /:

# tar --numeric-owner -xzvf container.tar.gz -C /

Verify the container's existence:

# lxc-ls -f
NAME       STATE    IPV4  IPV6  AUTOSTART  
-----------------------------------------
container  STOPPED  -     -     YES   

Now just start it on up:

# lxc-start -d -n container

And test by trying an ssh in!