Virtualization (LXC Containers)
As of Fall 2009, we use Linux containers to maintain virtual machines, most notably caffeine, which is hosted on glomag. The various commands to manipulate Linux containers are prefixed with "lxc-"; see their individual manpages for usage.
Management Quick Guide
To manage containers, use the lxc-* tools, which require root privilege. Some examples (replace caffeine with the appropriate container name):
# check if caffeine is running lxc-info -n caffeine # start caffeine in the background lxc-start -d -n caffeine # stop caffeine gracefully lxc-halt -n caffeine # stop caffeine forcefully lxc-stop -n caffeine # launch a TTY console for the container lxc-console -n caffeine
To install Linux container support on a recent Debian (squeeze or newer) system:
- Install the lxc and bridge-utils packages.
- Create a bridged network interface (this can be configured in /etc/network/interfaces as though it were a normal Ethernet device, with the additional bridge_ports parameter. This is usually called br0 (can be created manually with brctl). LXC will create a virtual Ethernet device and add it to the bridge when each container starts.
To start caffeine, run the following command as root on glomag:
lxc-start -d -n caffeine
Containers are stored on the host filesystem in /var/lib/lxc (root filesystems are symlinked to the appropriate directory on /vm).
ehashman's Guide to LXC on Debian
Configuring the host machine
First, install all required packages:
# apt-get install lxc bridge-utils
Setting up ethernet bridging
Next, create an ethernet bridge for the container. Edit
# The primary network interface #auto eth0 #iface eth0 inet static # address 18.104.22.168 # netmask 255.255.255.0 # gateway 22.214.171.124 # Bridge ethernet for containers auto br0 iface br0 inet static bridge_ports eth0 address 126.96.36.199 netmask 255.255.255.0 gateway 188.8.131.52 dns-nameservers 184.108.40.206 220.127.116.11 dns-search wics.uwaterloo.ca uwaterloo.ca
Cross your fingers and restart networking for your configuration to take effect!
# ifdown br0 && ifup br0 // bash enter to see if you lost connectivity and have to make a machine room trip
Note: !!! Do not use !!!
# service networking restart
The init scripts are broken and this likely will result in a machine room trip (or IPMI power cycle).
Setting up storage
Last, allocate some space in your volume group to put the container root on:
// Find the correct volume group to put the container on # vgdisplay // Create the volume in the appropriate volume group # lvcreate -L 20G -n container vg0 // Find it in the dev mapper # ls /dev/mapper/ // Create a filesystem on it # mkfs.ext4 /dev/mapper/vg0-container // Add a mount point # mkdir /vm/container
Last, add it to
/dev/mapper/vg0-container /vm/container ext4 defaults 0 2
Test the entry with
# mount /vm/container
Now you're done!
Creating a new container
Create a new container using
// Create new container "container" with root fs located at /vm/container # lxc-create --dir=/vm/container -n container --template download
This will prompt you for distribution, release, and architecture. (Architecture must match host machine.)
Take this time to review its config in
/var/lib/lxc/container/config, and tell it to auto-start if you like:
# Auto-start the container on boot lxc.start.auto = 1
You'll also want to set up networking (if applicable):
# Networking lxc.network.type = veth lxc.network.flags = up lxc.network.link = br0 lxc.network.name = eth0 lxc.network.hwaddr = de:ad:be:ef:ba:be # or something sensible
// List containers, -f for fancy # lxc-ls -f
to ensure that your container has been successfully created; it should be listed. You can also list its root directory if you like. To start it in the background and obtain a root shell, do
// Start and attach a root shell # lxc-start -d -n container # lxc-attach -n container
Migrating a container between hosts
Start by shutting the container down:
Then make a tarball of the container's filesystem:
# tar --numeric-owner -czvf container.tar.gz /vm/container
Copy it to its target destination, along with the configs:
$ scp container.tar.gz new-host: $ scp -r /var/lib/lxc/container/ new-host:/var/lib/lxc/
Now carefully extract it. If you haven't already, provision storage and ethernet per the container creation section.
Yes, we really do want to stick it directly into
# tar --numeric-owner -xzvf container.tar.gz -C /
If you are migrating from an old version of LXC onto a newer one (e.g. migrating onto xylitol), update the config:
# lxc-update-config -c /vm/container/config
This will also create a config.backup; you should inspect the new config file to make sure the migration was successful.
Verify the container's existence:
# lxc-ls -f NAME STATE IPV4 IPV6 AUTOSTART ----------------------------------------- container STOPPED - - YES
Now just start it on up:
# lxc-start -d -n container
And test by trying an ssh in!