This page explains how to create CloudStack templates from which CloudStack VMs are based.
Official documentation: https://docs.cloudstack.apache.org/en/latest/adminguide/templates.html
We require that cloud templates be prepared with cloud-init. cloud-init is a program which passes configuration data (e.g. user's SSH key) from a cloud provider to a VM at boot time. Most "major" GNU/Linux distributions have public cloud images which are (mostly) ready-to-use thanks to cloud-init.
There are two ways to create a template:
1. Convert the root disk volume of an existing VM into a template, or
2. Take a QCOW2 file, mount it over NBD, and modify files as necessary
The first option takes more time, but is generally easier. You can use either option; just keep in mind that some commands, such as
systemctl, can only be executed in a live system.
If you are downloading a template over HTTP(S), there's one weird problem which I found out the hard way - the CloudStack Systems VMs' iptables rules are configured to block outgoing traffic to VLAN 134. I have no idea why CloudStack decides to do this, but you're going to have to work around those rules if you want to download a file hosted on a machine on-campus.
SSH into the Secondary Storage VM, e.g.
ssh -i /var/lib/cloudstack/management/.ssh/id_rsa -p 3922 email@example.com
Check if the first rule of the OUTPUT chain will accept all traffic:
iptables -L -vn
If not, add such a rule:
iptables -I OUTPUT 1 -j ACCEPT
UPDATE: I created a custom systemd service so that this happens automatically. It's in /etc/systemd/system/csc-accept-all-output.service in the system VM of type "secondarystoragevm" (currently called in "s-2-VM"). Here it is for reference:
[Unit] Description=Accept all outgoing traffic After=cloud-postinit.service [Service] Type=oneshot ExecStart=/usr/sbin/iptables -I OUTPUT 1 -j ACCEPT RemainAfterExit=yes [Install] WantedBy=multi-user.target
If you are uploading a QCOW2 image directly from your computer, you need to be on the campus network. For some reason the web UI tries to contact the management server directly, which of course has a private IP address. So either configure your browser to use a SOCKS proxy, or use the campus VPN.
We're going to start with Debian since it's the first template which I created. Other distributions are mostly similar.
From the CloudStack UI, as the admin user, click on Templates, then 'Register Template From URL'. Create a new template using this URL (change the release if necessary): https://cloud.debian.org/images/cloud/bullseye/latest/debian-11-genericcloud-amd64.qcow2. The following checkboxes should be checked: Extractable, HVM (don't make it public - we want members to use our modified templates instead).
Create a new VM using this template. Use the admin keypair during creation. Once it's ready, SSH into it using the admin keypair (on biloba or chamomile):
ssh -i /var/lib/cloudstack/management/.ssh/id_rsa firstname.lastname@example.org
Replace the IP address as necessary.
We want the IPv6 address in our machines to be derived from the IPv4 address - for example, 172.19.134.129 becomes 2620:101:f000:4903::129. (Yes, I am aware that 129 in decimal is not the same as 129 in hex. But it makes it easier to remember.)
We're going to create some custom scripts which inject an IPv6 address at startup:
mkdir /etc/csc # Here's a script to disable Router Advertisements (RA) and Duplicate Address Detection (DAD) cat << 'EOF' > /etc/csc/ipv6-sysctl.sh #!/bin/bash set -ex while read interface _; do if [ $interface = lo ]; then continue fi sysctl net.ipv6.conf.$interface.accept_ra=0 sysctl net.ipv6.conf.$interface.accept_dad=0 done < <(ip -brief link show) EOF chmod +x /etc/csc/ipv6-sysctl.sh # Here's a script to derive the IPv6 address from the IPv4 address cat << 'EOF' > /etc/csc/ipv6-addr.sh #!/bin/bash set -ex INTERFACE= IPV4_ADDRESS= while read interface _ address; do if ! echo $address | grep -q '^172\.19\.134\.'; then continue fi INTERFACE=$interface IPV4_ADDRESS=$address break done < <(ip -4 -brief addr show) if [ -z "$INTERFACE" ]; then echo "Could not find primary interface" >&2 exit 1 fi NUM=$(echo $IPV4_ADDRESS | grep -oP '^172\.19\.134\.\K(\d+)') IPV6_ADDRESS="2620:101:f000:4903::$NUM/64" ip -6 addr add $IPV6_ADDRESS dev $INTERFACE ip -6 route add default via 2620:101:f000:4903::1 dev $INTERFACE EOF chmod +x /etc/csc/ipv6-addr.sh # Create some systemd services cat <<EOF >/etc/systemd/system/csc-ipv6-sysctl.service [Unit] Description=Disable IPv6 RAs and DAD # needed to avoid a dependency on basic.target DefaultDependencies=no After=network-pre.target Before=networking.service [Service] Type=oneshot RemainAfterExit=yes ExecStart=/etc/csc/ipv6-sysctl.sh [Install] WantedBy=multi-user.target EOF cat <<EOF >/etc/systemd/system/csc-ipv6-addr.service [Unit] Description=Allocate IPv6 address Requires=csc-ipv6-sysctl.service After=csc-ipv6-sysctl.service Requires=network.target After=network.target [Service] Type=oneshot RemainAfterExit=yes ExecStart=/etc/csc/ipv6-addr.sh [Install] WantedBy=multi-user.target EOF # Enable the systemd services systemctl daemon-reload systemctl enable csc-ipv6-sysctl systemctl enable csc-ipv6-addr
OPen /etc/dhcp/dhclient.conf and remove the following fields from 'request': domain-name, domain-name-servers, domain-search, dhcp6.name-servers, dhcp6.domain-search, dhcp6.fqdn, dhcp6.sntp-servers, ntp-servers
rm /etc/resolv.conf cat <<EOF >/etc/resolv.conf search csclub.uwaterloo.ca uwaterloo.ca options rotate timeout:1 attempts:1 ndots:2 # CSC Nameservers nameserver 2620:101:f000:4901:c5c::4 nameserver 2620:101:f000:7300:c5c::20 nameserver 126.96.36.199 nameserver 188.8.131.52 # IST Anycast (fallback) #nameserver 184.108.40.206 #nameserver 220.127.116.11 EOF
Open /etc/chrony/chrony.conf, comment out the 'pool' line, and add the following lines:
#server ntp.csclub.uwaterloo.ca server 18.104.22.168 #server ntp.student.cs.uwaterloo.ca server 22.214.171.124 #server ntp.cs.uwaterloo.ca server 126.96.36.199 #server ntp.cscf.uwaterloo.ca server 188.8.131.52
Open /etc/apt/sources.list and replace
mirror.csclub.uwaterloo.ca. Also replace
mirror.csclub.uwaterloo.ca. Comment out the deb-src lines.
Add a custom MOTD in /etc/motd. Make sure to thank our sponsors (MEF and CSCF).
Add/set the following line in /etc/ssh/sshd_config:
We need to undo the work done by cloud-init:
echo debian > /etc/hostname cat <<EOF >/etc/hosts 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters EOF cloud-init clean
Try to leave as little trace as possible:
apt clean # Clear logs rm -f /var/log/*.log rm -f /var/log/syslog rm -f /var/log/messages journalctl --vacuum-size=1 # Clear history files rm -f ~/.viminfo rm -f ~/.bash_history history -c exit # do this^ for the debian user too
Create the template
Power off the VM from the CloudStack UI, and follow the instructions here: http://docs.cloudstack.apache.org/en/latest/adminguide/templates.html#creating-a-template-from-an-existing-virtual-machine. (You're basically converting the root disk into a template.)
After you create the template, click on it from the UI, go to 'Settings', and delete SSH.PublicKey.
Cloud image (replace release if necessary): https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-disk-kvm.img
All of the instructions for Debian apply, except for the changes below.
Ubuntu uses systemd-networkd instead of ifupdown, so in the custom systemd services which we create, replace 'networking.service' by 'systemd-networkd.service'.
For some reason systemd-networkd thinks that it's OK to accept router advertisements despite us explicitly disabling them at the kernel level, so we need to add the following snippet to /etc/cloud/cloud.cfg:
network: version: 2 ethernets: id0: match: name: en* dhcp4: true accept-ra: false
While you're at it, add/set the following keys to /etc/cloud/cloud.cfg as well:
apt_preserve_sources_list: true manage_etc_hosts: true
These preserve our changes to /etc/apt/sources.list and tell cloud-init to manage /etc/hosts.
Ubuntu uses systemd-timesyncd instead of chrony, so open /etc/systemd/timesyncd.conf and set the following:
# ntp.csclub.uwaterloo.ca ntp.student.cs.uwaterloo.ca NTP=184.108.40.206 220.127.116.11 # ntp.cs.uwaterloo.ca ntp.cscf.uwaterloo.ca FallbackNTP=18.104.22.168 22.214.171.124
Open /etc/apt/sources.list and replace 'zone1.clouds.archive.ubuntu.com' with 'mirror.csclub.uwaterloo.ca'. Also replace 'security.ubuntu.com' with 'mirror.csclub.uwaterloo.ca'.
In addition to setting /etc/motd, disable some of these noisy MOTD headers:
chmod 640 /etc/update-motd.d/00-header chmod 640 /etc/update-motd.d/10-help-text chmod 640 /etc/update-motd.d/50-landscape-sysinfo chmod 640 /etc/update-motd.d/50-motd-news chmod 640 /etc/update-motd.d/88-esm-announce
Follow the same instructions for Debian, but paste 'ubuntu' into /etc/hostname.
Download one of the GenericCloud images from here: https://cloud.centos.org/centos/8-stream/x86_64/images/
Warning: I've noticed that the default user in CentOS Stream 9 is called 'cloud-user' instead of 'centos'.
All of the instructions for Debian apply, except for the changes below.
No text editor is installed, so you'll want to install vim/nano.
CentOS uses NetworkManager instead of ifupdown, so replace 'networking.service' by 'NetworkManager.service' in our custom systemd services.
In csc-ipv6-addr.service, replace 'network.target' by 'network-online.target' (otherwise the IPv4 address will not have been assigned by the time our script runs).
cloud-init has trouble reading DHCP data from NetworkManager for some reason, so install dhclient instead and add this line to the
[Main] section of /etc/NetworkManager/NetworkManager.conf:
dhcp = dhclient
Before you logout of the VM, make sure to also
rm -f /var/lib/NetworkManager/* to get rid of stale dhclient lease files.
The file to edit is /etc/chrony.conf, not /etc/chrony/chrony.conf.
In /etc/yum.repos.d, for each .repo file, comment out the 'mirrorlist=' line and add e.g.
Replace 'BaseOS' by the name of the repo. For the Extras repo, write 'extras' as the repo name.
In /etc/cloud/cloud.cfg, comment out the line 'ssh_genkeytypes: ~' and add this line:
ssh_genkeytypes: ['rsa', 'ecdsa', 'ed25519']
Also add the following to /etc/cloud/cloud.cfg:
Paste the following into /etc/hosts:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
Paste 'localhost.localdomain' into /etc/hostname.
dnf clean all instead of
All of the instructions for CentOS Stream apply, except for the changes below.
Mount the QEMU image
Unfortunately cloud-init completely failed when I tried to use the Fedora Cloud image (https://alt.fedoraproject.org/cloud/) on a CloudStack VM. So we're going to download the QEMU image and mount it over NBD (instructions: https://gist.github.com/shamil/62935d9b456a6f9877b5). Note that Fedora uses btrfs, so you'll need to run something like
mount -t btrfs /dev/nbd0p5 /mnt/fedora.
dhcp = dhclient to the
[Main] section of /etc/NetworkManager/NetworkManager.conf in the mounted filesystem. Unmount the filesystem, remove the NBD device, and move/copy the modified QEMU image to /var/www/csc-cloud-images. You should now be able to download the image from the CloudStack UI using the URL http://biloba.cloud.csclub.uwaterloo.ca/csc-cloud-images/<name_of_file>. Make sure to edit the iptables rules in the System Storage VM first (see #Limitations).
For each file in /etc/yum.repos.d, comment out 'metalink' and set 'baseurl' to use the CSC mirror (except for cisco-openh264). For example, fedora.repo should use http://mirror.csclub.uwaterloo.ca/fedora/linux/releases/35/Everything/x86_64/os/.