CloudStack Templates

This page explains how to create CloudStack templates from which CloudStack VMs are based.

Official documentation: https://docs.cloudstack.apache.org/en/latest/adminguide/templates.html

We require that cloud templates be prepared with cloud-init. cloud-init is a program which passes configuration data (e.g. user's SSH key) from a cloud provider to a VM at boot time. Most "major" GNU/Linux distributions have public cloud images which are (mostly) ready-to-use thanks to cloud-init.

Overview

There are two ways to create a template:

1. Convert the root disk volume of an existing VM into a template, or

2. Take a QCOW2 file, mount it over NBD, and modify files as necessary

The first option takes more time, but is generally easier. You can use either option; just keep in mind that some commands, such as systemctl, can only be executed in a live system.

Limitations

Downloading

If you are downloading a template over HTTP(S), there's one weird problem which I found out the hard way - the CloudStack Systems VMs' iptables rules are configured to block outgoing traffic to VLAN 134. I have no idea why CloudStack decides to do this, but you're going to have to work around those rules if you want to download a file hosted on a machine on-campus.

SSH into the Secondary Storage VM, e.g.

ssh -i /var/lib/cloudstack/management/.ssh/id_rsa -p 3922 root@169.254.12.1

Check if the first rule of the OUTPUT chain will accept all traffic:

iptables -L -vn

If not, add such a rule:

iptables -I OUTPUT 1 -j ACCEPT

UPDATE: I created a custom systemd service so that this happens automatically. It's in /etc/systemd/system/csc-accept-all-output.service in the system VM of type "secondarystoragevm" (currently called in "s-2-VM"). Here it is for reference:

[Unit]
Description=Accept all outgoing traffic
After=cloud-postinit.service

[Service]
Type=oneshot
ExecStart=/usr/sbin/iptables -I OUTPUT 1 -j ACCEPT
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Uploading

If you are uploading a QCOW2 image directly from your computer, you need to be on the campus network. For some reason the web UI tries to contact the management server directly, which of course has a private IP address. So either configure your browser to use a SOCKS proxy, or use the campus VPN.

Debian

We're going to start with Debian since it's the first template which I created. Other distributions are mostly similar.

From the CloudStack UI, as the admin user, click on Templates, then 'Register Template From URL'. Create a new template using this URL (change the release if necessary): https://cloud.debian.org/images/cloud/bullseye/latest/debian-11-genericcloud-amd64.qcow2. The following checkboxes should be checked: Extractable, HVM (don't make it public - we want members to use our modified templates instead).

Create a new VM using this template. Use the admin keypair during creation. Once it's ready, SSH into it using the admin keypair (on biloba or chamomile):

ssh -i /var/lib/cloudstack/management/.ssh/id_rsa debian@172.19.134.129

Replace the IP address as necessary.

IPv6 setup

We want the IPv6 address in our machines to be derived from the IPv4 address - for example, 172.19.134.129 becomes 2620:101:f000:4903::129. (Yes, I am aware that 129 in decimal is not the same as 129 in hex. But it makes it easier to remember.)

We're going to create some custom scripts which inject an IPv6 address at startup:

mkdir /etc/csc

# Here's a script to disable Router Advertisements (RA) and Duplicate Address Detection (DAD)
cat << 'EOF' > /etc/csc/ipv6-sysctl.sh
#!/bin/bash
set -ex
while read interface _; do    
        if [ $interface = lo ]; then
                continue
        fi
        sysctl net.ipv6.conf.$interface.accept_ra=0
        sysctl net.ipv6.conf.$interface.accept_dad=0
done < <(ip -brief link show)
EOF
chmod +x /etc/csc/ipv6-sysctl.sh

# Here's a script to derive the IPv6 address from the IPv4 address
cat << 'EOF' > /etc/csc/ipv6-addr.sh
#!/bin/bash
set -ex
INTERFACE=
IPV4_ADDRESS=
while read interface _ address; do
        if ! echo $address | grep -q '^172\.19\.134\.'; then
                continue
        fi
        INTERFACE=$interface
        IPV4_ADDRESS=$address
        break
done < <(ip -4 -brief addr show)
if [ -z "$INTERFACE" ]; then
        echo "Could not find primary interface" >&2
        exit 1
fi
NUM=$(echo $IPV4_ADDRESS | grep -oP '^172\.19\.134\.\K(\d+)')
IPV6_ADDRESS="2620:101:f000:4903::$NUM/64"
ip -6 addr add $IPV6_ADDRESS dev $INTERFACE
ip -6 route add default via 2620:101:f000:4903::1 dev $INTERFACE
EOF
chmod +x /etc/csc/ipv6-addr.sh

# Create some systemd services
cat <<EOF >/etc/systemd/system/csc-ipv6-sysctl.service
[Unit]
Description=Disable IPv6 RAs and DAD
# needed to avoid a dependency on basic.target
DefaultDependencies=no
After=network-pre.target
Before=networking.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/etc/csc/ipv6-sysctl.sh

[Install]
WantedBy=multi-user.target
EOF

cat <<EOF >/etc/systemd/system/csc-ipv6-addr.service
[Unit]
Description=Allocate IPv6 address
Requires=csc-ipv6-sysctl.service
After=csc-ipv6-sysctl.service
Requires=network.target
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/etc/csc/ipv6-addr.sh

[Install]
WantedBy=multi-user.target
EOF

# Enable the systemd services
systemctl daemon-reload
systemctl enable csc-ipv6-sysctl
systemctl enable csc-ipv6-addr

dhclient

OPen /etc/dhcp/dhclient.conf and remove the following fields from 'request': domain-name, domain-name-servers, domain-search, dhcp6.name-servers, dhcp6.domain-search, dhcp6.fqdn, dhcp6.sntp-servers, ntp-servers

resolv.conf

rm /etc/resolv.conf
cat <<EOF >/etc/resolv.conf
search      csclub.uwaterloo.ca uwaterloo.ca
options     rotate timeout:1 attempts:1 ndots:2

# CSC Nameservers
nameserver  2620:101:f000:4901:c5c::4
nameserver  2620:101:f000:7300:c5c::20
nameserver  129.97.134.4
nameserver  129.97.18.20

# IST Anycast (fallback)
#nameserver  129.97.2.1
#nameserver  129.97.2.2
EOF

chrony

Open /etc/chrony/chrony.conf, comment out the 'pool' line, and add the following lines:

#server   ntp.csclub.uwaterloo.ca
server   129.97.167.12
#server   ntp.student.cs.uwaterloo.ca
server   129.97.167.4
#server   ntp.cs.uwaterloo.ca
server   129.97.15.14
#server   ntp.cscf.uwaterloo.ca
server   129.97.15.15

Mirrors

Open /etc/apt/sources.list and replace deb.debian.org by mirror.csclub.uwaterloo.ca. Also replace security.debian.org by mirror.csclub.uwaterloo.ca. Comment out the deb-src lines.

MOTD

Add a custom MOTD in /etc/motd. Make sure to thank our sponsors (MEF and CSCF).

sshd

Add/set the following line in /etc/ssh/sshd_config:

PrintLastLog=no

Reset cloud-init

We need to undo the work done by cloud-init:

echo debian > /etc/hostname
cat <<EOF >/etc/hosts
127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters
EOF
cloud-init clean

Cleanup

Try to leave as little trace as possible:

apt clean

# Clear logs
rm -f /var/log/*.log
rm -f /var/log/syslog
rm -f /var/log/messages
journalctl --vacuum-size=1

# Clear history files
rm -f ~/.viminfo
rm -f ~/.bash_history
history -c
exit
# do this^ for the debian user too

Create the template

Power off the VM from the CloudStack UI, and follow the instructions here: http://docs.cloudstack.apache.org/en/latest/adminguide/templates.html#creating-a-template-from-an-existing-virtual-machine. (You're basically converting the root disk into a template.)

After you create the template, click on it from the UI, go to 'Settings', and delete SSH.PublicKey.

Ubuntu

Cloud image (replace release if necessary): https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-disk-kvm.img

All of the instructions for Debian apply, except for the changes below.

systemd-networkd

Ubuntu uses systemd-networkd instead of ifupdown, so in the custom systemd services which we create, replace 'networking.service' by 'systemd-networkd.service'.

For some reason systemd-networkd thinks that it's OK to accept router advertisements despite us explicitly disabling them at the kernel level, so we need to add the following snippet to /etc/cloud/cloud.cfg:

network:
  version: 2
  ethernets:
    id0:
      match:
        name: en*
      dhcp4: true
      accept-ra: false

While you're at it, add/set the following keys to /etc/cloud/cloud.cfg as well:

apt_preserve_sources_list: true
manage_etc_hosts: true

These preserve our changes to /etc/apt/sources.list and tell cloud-init to manage /etc/hosts.

systemd-timesyncd

Ubuntu uses systemd-timesyncd instead of chrony, so open /etc/systemd/timesyncd.conf and set the following:

# ntp.csclub.uwaterloo.ca ntp.student.cs.uwaterloo.ca
NTP=129.97.167.12 129.97.167.4
# ntp.cs.uwaterloo.ca ntp.cscf.uwaterloo.ca
FallbackNTP=129.97.15.14 129.97.15.15

Mirrors

Open /etc/apt/sources.list and replace 'zone1.clouds.archive.ubuntu.com' with 'mirror.csclub.uwaterloo.ca'. Also replace 'security.ubuntu.com' with 'mirror.csclub.uwaterloo.ca'.

MOTD

In addition to setting /etc/motd, disable some of these noisy MOTD headers:

chmod 640 /etc/update-motd.d/00-header
chmod 640 /etc/update-motd.d/10-help-text
chmod 640 /etc/update-motd.d/50-landscape-sysinfo
chmod 640 /etc/update-motd.d/50-motd-news
chmod 640 /etc/update-motd.d/88-esm-announce

Reset cloud-init

Follow the same instructions for Debian, but paste 'ubuntu' into /etc/hostname.

CentOS Stream

Download one of the GenericCloud images from here: https://cloud.centos.org/centos/8-stream/x86_64/images/

Warning: I've noticed that the default user in CentOS Stream 9 is called 'cloud-user' instead of 'centos'.

All of the instructions for Debian apply, except for the changes below.

No text editor is installed, so you'll want to install vim/nano.

NetworkManager

CentOS uses NetworkManager instead of ifupdown, so replace 'networking.service' by 'NetworkManager.service' in our custom systemd services.

In csc-ipv6-addr.service, replace 'network.target' by 'network-online.target' (otherwise the IPv4 address will not have been assigned by the time our script runs).

cloud-init has trouble reading DHCP data from NetworkManager for some reason, so install dhclient instead and add this line to the [Main] section of /etc/NetworkManager/NetworkManager.conf:

dhcp = dhclient

Before you logout of the VM, make sure to also rm -f /var/lib/NetworkManager/* to get rid of stale dhclient lease files.

chrony

The file to edit is /etc/chrony.conf, not /etc/chrony/chrony.conf.

Mirrors

In /etc/yum.repos.d, for each .repo file, comment out the 'mirrorlist=' line and add e.g.

baseurl=http://mirror.csclub.uwaterloo.ca/$contentdir/$stream/BaseOS/$basearch/os/

Replace 'BaseOS' by the name of the repo. For the Extras repo, write 'extras' as the repo name.

cloud.cfg configuration

There is a bug in the cloud-init configuration which was fixed in RHEL 8.2 but the fix doesn't seem to have made it to CentOS Stream 8 (it may already be fixed in CentOS Stream 9). See here and here.

In /etc/cloud/cloud.cfg, comment out the line 'ssh_genkeytypes: ~' and add this line:

    ssh_genkeytypes: ['rsa', 'ecdsa', 'ed25519']

Also add the following to /etc/cloud/cloud.cfg:

    manage_etc_hosts: true

Hosts configuration

Paste the following into /etc/hosts:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Paste 'localhost.localdomain' into /etc/hostname.

Cleanup

Run dnf clean all instead of apt clean.

Fedora

All of the instructions for CentOS Stream apply, except for the changes below.

Mount the QEMU image

Unfortunately cloud-init completely failed when I tried to use the Fedora Cloud image (https://alt.fedoraproject.org/cloud/) on a CloudStack VM. So we're going to download the QEMU image and mount it over NBD (instructions: https://gist.github.com/shamil/62935d9b456a6f9877b5). Note that Fedora uses btrfs, so you'll need to run something like mount -t btrfs /dev/nbd0p5 /mnt/fedora.

Add dhcp = dhclient to the [Main] section of /etc/NetworkManager/NetworkManager.conf in the mounted filesystem. Unmount the filesystem, remove the NBD device, and move/copy the modified QEMU image to /var/www/csc-cloud-images. You should now be able to download the image from the CloudStack UI using the URL http://biloba.cloud.csclub.uwaterloo.ca/csc-cloud-images/<name_of_file>. Make sure to edit the iptables rules in the System Storage VM first (see #Limitations).

Mirrors

For each file in /etc/yum.repos.d, comment out 'metalink' and set 'baseurl' to use the CSC mirror (except for cisco-openh264). For example, fedora.repo should use http://mirror.csclub.uwaterloo.ca/fedora/linux/releases/35/Everything/x86_64/os/.