Difference between revisions of "CloudStack Templates"

From CSCWiki
Jump to navigation Jump to search
m (Test)
Line 8: Line 8:
There are two ways to create a template:
There are two ways to create a template:
1. Convert the root disk volume of an existing VM into a template
1. Convert the root disk volume of an existing VM into a template, or
2. Take a QCOW2 file, mount it over NBD, and modify files as necessary
2. Take a QCOW2 file, mount it over NBD, and modify files as necessary

Revision as of 21:15, 27 May 2022

This page explains how to create CloudStack templates from which CloudStack VMs are based.

Official documentation: https://docs.cloudstack.apache.org/en/latest/adminguide/templates.html

We require that cloud templates be prepared with cloud-init. cloud-init is a program which passes configuration data (e.g. user's SSH key) from a cloud provider to a VM at boot time. Most "major" GNU/Linux distributions have public cloud images which are (mostly) ready-to-use thanks to cloud-init.


There are two ways to create a template:

1. Convert the root disk volume of an existing VM into a template, or

2. Take a QCOW2 file, mount it over NBD, and modify files as necessary

The first option takes more time, but is generally easier. You can use either option; just keep in mind that some commands, such as systemctl, can only be executed in a live system.



If you are downloading a template over HTTP(S), there's one weird problem which I found out the hard way - the CloudStack Systems VMs' iptables rules are configured to block outgoing traffic to VLAN 134. I have no idea why CloudStack decides to do this, but you're going to have to work around those rules if you want to download a file hosted on a machine on-campus.

SSH into the Secondary Storage VM, e.g.

ssh -i /var/lib/cloudstack/management/.ssh/id_rsa -p 3922 root@

Check if the first rule of the OUTPUT chain will accept all traffic:

iptables -L -vn

If not, add such a rule:

iptables -I OUTPUT 1 -j ACCEPT


If you are uploading a QCOW2 image directly from your computer, you need to be on the campus network. For some reason the web UI tries to contact the management server directly, which of course has a private IP address. So either configure your browser to use a SOCKS proxy, or use the campus VPN.


We're going to start with Debian since it's the first template which I created. Other distributions are mostly similar.

From the CloudStack UI, as the admin user, click on Templates, then 'Register Template From URL'. Create a new template using this URL (change the release if necessary): https://cloud.debian.org/images/cloud/bullseye/latest/debian-11-genericcloud-amd64.qcow2. The following checkboxes should be checked: Extractable, HVM (don't make it public - we want members to use our modified templates instead).

Create a new VM using this template. Use the admin keypair during creation. Once it's ready, SSH into it using the admin keypair (on biloba or chamomile):

ssh -i /var/lib/cloudstack/management/.ssh/id_rsa debian@

Replace the IP address as necessary.

IPv6 setup

We want the IPv6 address in our machines to be derived from the IPv4 address - for example, becomes 2620:101:f000:4903::129. (Yes, I am aware that 129 in decimal is not the same as 129 in hex. But it makes it easier to remember.)

We're going to create some custom scripts which inject an IPv6 address at startup:

mkdir /etc/csc

# Here's a script to disable Router Advertisements (RA) and Duplicate Address Detection (DAD)
cat << 'EOF' > /etc/csc/ipv6-sysctl.sh
set -ex
while read interface _; do    
        if [ $interface = lo ]; then
        sysctl net.ipv6.conf.$interface.accept_ra=0
        sysctl net.ipv6.conf.$interface.accept_dad=0
done < <(ip -brief link show)
chmod +x /etc/csc/ipv6-sysctl.sh

# Here's a script to derive the IPv6 address from the IPv4 address
cat << 'EOF' > /etc/csc/ipv6-addr.sh
set -ex
while read interface _ address; do
        if ! echo $address | grep -q '^172\.19\.134\.'; then
done < <(ip -4 -brief addr show)
if [ -z "$INTERFACE" ]; then
        echo "Could not find primary interface" >&2
        exit 1
NUM=$(echo $IPV4_ADDRESS | grep -oP '^172\.19\.134\.\K(\d+)')
ip -6 addr add $IPV6_ADDRESS dev $INTERFACE
ip -6 route add default via 2620:101:f000:4903::1 dev $INTERFACE
chmod +x /etc/csc/ipv6-addr.sh

# Create some systemd services
cat <<EOF >/etc/systemd/system/csc-ipv6-sysctl.service
Description=Disable IPv6 RAs and DAD
# needed to avoid a dependency on basic.target



cat <<EOF >/etc/systemd/system/csc-ipv6-addr.service
Description=Allocate IPv6 address



# Enable the systemd services
systemctl daemon-reload
systemctl enable csc-ipv6-sysctl
systemctl enable csc-ipv6-addr


OPen /etc/dhcp/dhclient.conf and remove the following fields from 'request': domain-name, domain-name-servers, domain-search, dhcp6.name-servers, dhcp6.domain-search, dhcp6.fqdn, dhcp6.sntp-servers, ntp-servers


rm /etc/resolv.conf
cat <<EOF >/etc/resolv.conf
search      csclub.uwaterloo.ca uwaterloo.ca
options     rotate timeout:1 attempts:1 ndots:2

# CSC Nameservers
nameserver  2620:101:f000:4901:c5c::4
nameserver  2620:101:f000:7300:c5c::20

# IST Anycast (fallback)


Open /etc/chrony/chrony.conf, comment out the 'pool' line, and add the following lines:

#server   ntp.csclub.uwaterloo.ca
#server   ntp.student.cs.uwaterloo.ca
#server   ntp.cs.uwaterloo.ca
#server   ntp.cscf.uwaterloo.ca


Open /etc/apt/sources.list and replace deb.debian.org by mirror.csclub.uwaterloo.ca. Also replace security.debian.org by mirror.csclub.uwaterloo.ca. Comment out the deb-src lines.


Add a custom MOTD in /etc/motd. Make sure to thank our sponsors (MEF and CSCF).


Add/set the following line in /etc/ssh/sshd_config:


Reset cloud-init

We need to undo the work done by cloud-init:

echo debian > /etc/hostname
cat <<EOF >/etc/hosts       localhost
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters
cloud-init clean


Try to leave as little trace as possible:

apt clean

# Clear logs
rm -f /var/log/*.log
rm -f /var/log/syslog
rm -f /var/log/messages
journalctl --vacuum-size=1

# Clear history files
rm -f ~/.viminfo
rm -f ~/.bash_history
history -c
# do this^ for the debian user too

Create the template

Power off the VM from the CloudStack UI, and follow the instructions here: http://docs.cloudstack.apache.org/en/latest/adminguide/templates.html#creating-a-template-from-an-existing-virtual-machine. (You're basically converting the root disk into a template.)

After you create the template, click on it from the UI, go to 'Settings', and delete SSH.PublicKey.


Cloud image (replace release if necessary): https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-disk-kvm.img

All of the instructions for Debian apply, except for the changes below.


Ubuntu uses systemd-networkd instead of ifupdown, so in the custom systemd services which we create, replace 'networking.service' by 'systemd-networkd.service'.

For some reason systemd-networkd thinks that it's OK to accept router advertisements despite us explicitly disabling them at the kernel level, so we need to add the following snippet to /etc/cloud/cloud.cfg:

  version: 2
        name: en*
      dhcp4: true
      accept-ra: false

While you're at it, add/set the following keys to /etc/cloud/cloud.cfg as well:

apt_preserve_sources_list: true
manage_etc_hosts: true

(These preserve our changes to /etc/apt/sources.list and /etc/hosts, respectively.)


Ubuntu uses systemd-timesyncd instead of chrony, so open /etc/systemd/timesyncd.conf and set the following:

# ntp.csclub.uwaterloo.ca ntp.student.cs.uwaterloo.ca
# ntp.cs.uwaterloo.ca ntp.cscf.uwaterloo.ca


Open /etc/apt/sources.list and replace 'zone1.clouds.archive.ubuntu.com' with 'mirror.csclub.uwaterloo.ca'. Also replace 'security.ubuntu.com' with 'mirror.csclub.uwaterloo.ca'.


In addition to setting /etc/motd, disable some of these noisy MOTD headers:

chmod 640 /etc/update-motd.d/00-header
chmod 640 /etc/update-motd.d/10-help-text
chmod 640 /etc/update-motd.d/50-landscape-sysinfo
chmod 640 /etc/update-motd.d/50-motd-news
chmod 640 /etc/update-motd.d/88-esm-announce

Reset cloud-init

Follow the same instructions for Debian, but paste 'ubuntu' into /etc/hostname.

CentOS Stream

Download one of the GenericCloud images from here: https://cloud.centos.org/centos/8-stream/x86_64/images/

Warning: I've noticed that the default user in CentOS Stream 9 is called 'cloud-user' instead of 'centos'.

All of the instructions for Debian apply, except for the changes below.

No text editor is installed, so you'll want to install vim/nano.


CentOS uses NetworkManager instead of ifupdown, so replace 'networking.service' by 'NetworkManager.service' in our custom systemd services.

In csc-ipv6-addr.service, replace 'network.target' by 'network-online.target' (otherwise the IPv4 address will not have been assigned by the time our script runs).

cloud-init has trouble reading DHCP data from NetworkManager for some reason, so install dhclient instead and add this line to the [Main] section of /etc/NetworkManager/NetworkManager.conf:

dhcp = dhclient

Before you logout of the VM, make sure to also rm -f /var/lib/NetworkManager/* to get rid of stale dhclient lease files.


The file to edit is /etc/chrony.conf, not /etc/chrony/chrony.conf.


In /etc/yum.repos.d, for each .repo file, comment out the 'mirrorlist=' line and add e.g.


Replace 'BaseOS' by the name of the repo. For the Extras repo, write 'extras' as the repo name.

cloud.cfg configuration

There is a bug in the cloud-init configuration which was fixed in RHEL 8.2 but the fix doesn't seem to have made it to CentOS Stream 8 (it may already be fixed in CentOS Stream 9). See here and here.

In /etc/cloud/cloud.cfg, comment out the line 'ssh_genkeytypes: ~' and add this line:

    ssh_genkeytypes: ['rsa', 'ecdsa', 'ed25519']

Also add the following to /etc/cloud/cloud.cfg:

    manage_etc_hosts: true

Hosts configuration

Paste the following into /etc/hosts:   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Paste 'localhost.localdomain' into /etc/hostname.


Run dnf clean all instead of apt clean.


All of the instructions for CentOS Stream apply, except for the changes below.

Mount the QEMU image

Unfortunately cloud-init completely failed when I tried to use the Fedora Cloud image (https://alt.fedoraproject.org/cloud/) on a CloudStack VM. So we're going to download the QEMU image and mount it over NBD (instructions: https://gist.github.com/shamil/62935d9b456a6f9877b5). Note that Fedora uses btrfs, so you'll need to run something like mount -t btrfs /dev/nbd0p5 /mnt/fedora.

Add dhcp = dhclient to the [Main] section of /etc/NetworkManager/NetworkManager.conf in the mounted filesystem. Unmount the filesystem, remove the NBD device, and move/copy the modified QEMU image to /var/www/csc-cloud-images. You should now be able to download the image from the CloudStack UI using the URL http://biloba.cloud.csclub.uwaterloo.ca/csc-cloud-images/<name_of_file>. Make sure to edit the iptables rules in the System Storage VM first (see #Limitations).


For each file in /etc/yum.repos.d, comment out 'metalink' and set 'baseurl' to use the CSC mirror (except for cisco-openh264). For example, fedora.repo should use http://mirror.csclub.uwaterloo.ca/fedora/linux/releases/35/Everything/x86_64/os/.