CloudStack Templates: Difference between revisions

From CSCWiki
Jump to navigation Jump to search
mNo edit summary
mNo edit summary
Line 218: Line 218:
Cloud image (replace release if necessary): https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-disk-kvm.img
Cloud image (replace release if necessary): https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-disk-kvm.img


All of the instructions for Debian apply, except for the changes below.
=== systemd-networkd

=== systemd-networkd ===
Ubuntu uses systemd-networkd instead of ifupdown, so in the custom systemd services which we create, replace 'networking.service' by 'systemd-networkd.service'.
Ubuntu uses systemd-networkd instead of ifupdown, so in the custom systemd services which we create, replace 'networking.service' by 'systemd-networkd.service'.


Line 264: Line 266:
=== Reset cloud-init ===
=== Reset cloud-init ===
Follow the same instructions for Debian, but paste 'ubuntu' into /etc/hostname.
Follow the same instructions for Debian, but paste 'ubuntu' into /etc/hostname.

== CentOS Stream ==
Download one of the GenericCloud images from here: https://cloud.centos.org/centos/8-stream/x86_64/images/

<b>Warning</b>: I've noticed that the default user in CentOS Stream 9 is called 'cloud-user' instead of 'centos'.

All of the instructions for Debian apply, except for the changes below.

No text editor is installed, so you'll want to install vim/nano.

=== NetworkManager ===
CentOS uses NetworkManager instead of ifupdown, so replace 'networking.service' by 'NetworkManager.service' in our custom systemd services.

In csc-ipv6-addr.service, replace 'network.target' by 'network-online.target' (otherwise the IPv4 address will not have been assigned by the time our script runs).

cloud-init has trouble reading DHCP data from NetworkManager for some reason, so install dhclient instead and add this line to the <code>[Main]</code> section of /etc/NetworkManager/NetworkManager.conf:
<pre>
dhcp = dhclient
</pre>

=== chrony ===
The file to edit is /etc/chrony.conf, not /etc/chrony/chrony.conf.

=== Mirrors ===
In /etc/yum.repos.d, for each .repo file, comment out the 'mirrorlist=' line and e.g.
<pre>
baseurl=http://mirror.csclub.uwaterloo.ca/$contentdir/$stream/BaseOS/$basearch/os/
</pre>
Replace 'BaseOS' by the name of the repo. For the Extras repo, write 'extras' as the repo name.

=== cloud.cfg configuration ===
There is a bug in the cloud-init configuration which was fixed in RHEL 8.2 but which doesn't seem to have made it to CentOS Stream 8 (it may already be fixed in CentOS Stream 9). See [https://bugzilla.redhat.com/show_bug.cgi?id=1888761 here] and [https://bugzilla.redhat.com/show_bug.cgi?id=1963981 here].

In /etc/cloud/cloud.cfg, comment out the line 'ssh_genkeytypes: ~' and add this line:
<pre>
ssh_genkeytypes: ['rsa', 'ecdsa', 'ed25519']
</pre>
Also add the following to /etc/cloud/cloud.cfg:
<pre>
manage_etc_hosts: true
</pre>

=== Hosts configuration ===
Paste the following into /etc/hosts:
<pre>
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
</pre>
Paste 'localhost.localdomain' into /etc/hostname.

=== Cleanup ===
Run <code>dnf clean all</code> instead of <code>apt clean</code>.

Revision as of 20:22, 30 December 2021

This page explains how to create CloudStack templates from which CloudStack VMs are based.

Official documentation: https://docs.cloudstack.apache.org/en/latest/adminguide/templates.html

We require that cloud templates be prepared with cloud-init. cloud-init is a program which passes configuration data (e.g. user's SSH key) from a cloud provider to a VM at boot time. Most "major" GNU/Linux distributions have public cloud images which are (mostly) ready-to-use thanks to cloud-init.

Overview

There are two ways to create a template:

1. Convert the root disk volume of an existing VM into a template 2. Take a QCOW2 file, mount it over NBD, and modify files as necessary

The first option takes more time, but is generally easier. You can use either option; just keep in mind that some commands, such as systemctl, can only be executed in a live system.

Limitations

Downloading

If you are downloading a template over HTTP(S), there's one weird problem which I found out the hard way - the CloudStack Systems VMs' iptables rules are configured to block outgoing traffic to VLAN 134. I have no idea why CloudStack decides to do this, but you're going to have to work around those rules if you want to download a file hosted on a machine on-campus.

SSH into the Secondary Storage VM, e.g.

ssh -i /var/lib/cloudstack/management/.ssh/id_rsa -p 3922 root@169.254.12.1

Check if the first rule of the OUTPUT chain will accept all traffic:

iptables -L -vn

If not, add such a rule:

iptables -I OUTPUT 1 -j ACCEPT

Uploading

If you are uploading a QCOW2 image directly from your computer, you need to be on the campus network. For some reason the web UI tries to contact the management server directly, which of course has a private IP address. So either configure your browser to use a SOCKS proxy, or use the campus VPN.

Debian

We're going to start with Debian since it's the first template which I created. Other distributions are mostly similar.

From the CloudStack UI, as the admin user, click on Templates, then 'Register Template From URL'. Create a new template using this URL (change the release if necessary): https://cloud.debian.org/images/cloud/bullseye/latest/debian-11-genericcloud-amd64.qcow2. The following checkboxes should be checked: Extractable, HVM (don't make it public - we want members to use our modified templates instead).

Create a new VM using this template. Use the admin keypair during creation. Once it's ready, SSH into it using the admin keypair (on biloba or chamomile):

ssh -i /var/lib/cloudstack/management/.ssh/id_rsa debian@172.19.134.129

Replace the IP address as necessary.

IPv6 setup

We want the IPv6 address in our machines to be derived from the IPv4 address - for example, 172.19.134.129 becomes 2620:101:f000:4903::129. (Yes, I am aware that 129 in decimal is not the same as 129 in hex. But it makes it easier to remember.)

We're going to create some custom scripts which inject an IPv6 address at startup:

mkdir /etc/csc

# Here's a script to disable Router Advertisements (RA) and Duplicate Address Detection (DAD)
cat << 'EOF' > /etc/csc/ipv6-sysctl.sh
#!/bin/bash
set -ex
while read interface _; do    
        if [ $interface = lo ]; then
                continue
        fi
        sysctl net.ipv6.conf.$interface.accept_ra=0
        sysctl net.ipv6.conf.$interface.accept_dad=0
done < <(ip -brief link show)
EOF
chmod +x /etc/csc/ipv6-sysctl.sh

# Here's a script to derive the IPv6 address from the IPv4 address
cat << 'EOF' > /etc/csc/ipv6-addr.sh
#!/bin/bash
set -ex
INTERFACE=
IPV4_ADDRESS=
while read interface _ address; do
        if ! echo $address | grep -q '^172\.19\.134\.'; then
                continue
        fi
        INTERFACE=$interface
        IPV4_ADDRESS=$address
        break
done < <(ip -4 -brief addr show)
if [ -z "$INTERFACE" ]; then
        echo "Could not find primary interface" >&2
        exit 1
fi
NUM=$(echo $IPV4_ADDRESS | grep -oP '^172\.19\.134\.\K(\d+)')
IPV6_ADDRESS="2620:101:f000:4903::$NUM/64"
ip -6 addr add $IPV6_ADDRESS dev $INTERFACE
ip -6 route add default via 2620:101:f000:4903::1 dev $INTERFACE
EOF
chmod +x /etc/csc/ipv6-addr.sh

# Create some systemd services
cat <<EOF >/etc/systemd/system/csc-ipv6-sysctl.service
[Unit]
Description=Disable IPv6 RAs and DAD
# needed to avoid a dependency on basic.target
DefaultDependencies=no
After=network-pre.target
Before=networking.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/etc/csc/ipv6-sysctl.sh

[Install]
WantedBy=multi-user.target
EOF

cat <<EOF >/etc/systemd/system/csc-ipv6-addr.service
[Unit]
Description=Allocate IPv6 address
Requires=csc-ipv6-sysctl.service
After=csc-ipv6-sysctl.service
Requires=network.target
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/etc/csc/ipv6-addr.sh

[Install]
WantedBy=multi-user.target
EOF

# Enable the systemd services
systemctl daemon-reload
systemctl enable csc-ipv6-sysctl
systemctl enable csc-ipv6-addr

dhclient

OPen /etc/dhcp/dhclient.conf and remove the following fields from 'request': domain-name, domain-name-servers, domain-search, dhcp6.name-servers, dhcp6.domain-search, dhcp6.fqdn, dhcp6.sntp-servers, ntp-servers

resolv.conf

rm /etc/resolv.conf
cat <<EOF >/etc/resolv.conf
search      csclub.uwaterloo.ca uwaterloo.ca
options     rotate timeout:1 attempts:1 ndots:2

# CSC Nameservers
nameserver  2620:101:f000:4901:c5c::4
nameserver  2620:101:f000:7300:c5c::20
nameserver  129.97.134.4
nameserver  129.97.18.20

# IST Anycast (fallback)
#nameserver  129.97.2.1
#nameserver  129.97.2.2
EOF

chrony

Open /etc/chrony/chrony.conf, comment out the 'pool' line, and add the following lines:

#server   ntp.csclub.uwaterloo.ca
server   129.97.167.12
#server   ntp.student.cs.uwaterloo.ca
server   129.97.167.4
#server   ntp.cs.uwaterloo.ca
server   129.97.15.14
#server   ntp.cscf.uwaterloo.ca
server   129.97.15.15

Mirrors

Open /etc/apt/sources.list and replace deb.debian.org by mirror.csclub.uwaterloo.ca. Also replace security.debian.org by mirror.csclub.uwaterloo.ca. Comment out the deb-src lines.

MOTD

Add a custom MOTD in /etc/motd. Make sure to thank our sponsors (MEF and CSCF).

sshd

Add/set the following line in /etc/ssh/sshd_config:

PrintLastLog=no

Reset cloud-init

We need to undo the work done by cloud-init:

echo debian > /etc/hostname
cat <<EOF >/etc/hosts
127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters
EOF
cloud-init clean

Cleanup

Try to leave as little trace as possible:

apt clean

# Clear logs
rm -f /var/log/*.log
rm -f /var/log/syslog
rm -f /var/log/messages
journalctl --vacuum-size=1

# Clear history files
rm -f ~/.viminfo
rm -f ~/.bash_history
history -c
exit
# do this^ for the debian user too

Create the template

Power off the VM from the CloudStack UI, and follow the instructions here: http://docs.cloudstack.apache.org/en/latest/adminguide/templates.html#creating-a-template-from-an-existing-virtual-machine. (You're basically converting the root disk into a template.)

After you create the template, click on it from the UI, go to 'Settings', and delete SSH.PublicKey.

Ubuntu

Cloud image (replace release if necessary): https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64-disk-kvm.img

All of the instructions for Debian apply, except for the changes below.

systemd-networkd

Ubuntu uses systemd-networkd instead of ifupdown, so in the custom systemd services which we create, replace 'networking.service' by 'systemd-networkd.service'.

For some reason systemd-networkd thinks that it's OK to accept router advertisements despite us explicitly disabling them at the kernel level, so we need to add the following snippet to /etc/cloud/cloud.cfg:

network:
  version: 2
  ethernets:
    id0:
      match:
        name: en*
      dhcp4: true
      accept-ra: false

While you're at it, add/set the following keys to /etc/cloud/cloud.cfg as well:

apt_preserve_sources_list: true
manage_etc_hosts: true

(These preserve our changes to /etc/apt/sources.list and /etc/hosts, respectively.)

systemd-timesyncd

Ubuntu uses systemd-timesyncd instead of chrony, so open /etc/systemd/timesyncd.conf and set the following:

# ntp.csclub.uwaterloo.ca ntp.student.cs.uwaterloo.ca
NTP=129.97.167.12 129.97.167.4
# ntp.cs.uwaterloo.ca ntp.cscf.uwaterloo.ca
FallbackNTP=129.97.15.14 129.97.15.15

Mirrors

Open /etc/apt/sources.list and replace 'zone1.clouds.archive.ubuntu.com' with 'mirror.csclub.uwaterloo.ca'. Also replace 'security.ubuntu.com' with 'mirror.csclub.uwaterloo.ca'.

MOTD

In addition to setting /etc/motd, disable some of these noisy MOTD headers:

chmod 640 /etc/update-motd.d/00-header
chmod 640 /etc/update-motd.d/10-help-text
chmod 640 /etc/update-motd.d/50-landscape-sysinfo
chmod 640 /etc/update-motd.d/50-motd-news
chmod 640 /etc/update-motd.d/88-esm-announce

Reset cloud-init

Follow the same instructions for Debian, but paste 'ubuntu' into /etc/hostname.

CentOS Stream

Download one of the GenericCloud images from here: https://cloud.centos.org/centos/8-stream/x86_64/images/

Warning: I've noticed that the default user in CentOS Stream 9 is called 'cloud-user' instead of 'centos'.

All of the instructions for Debian apply, except for the changes below.

No text editor is installed, so you'll want to install vim/nano.

NetworkManager

CentOS uses NetworkManager instead of ifupdown, so replace 'networking.service' by 'NetworkManager.service' in our custom systemd services.

In csc-ipv6-addr.service, replace 'network.target' by 'network-online.target' (otherwise the IPv4 address will not have been assigned by the time our script runs).

cloud-init has trouble reading DHCP data from NetworkManager for some reason, so install dhclient instead and add this line to the [Main] section of /etc/NetworkManager/NetworkManager.conf:

dhcp = dhclient

chrony

The file to edit is /etc/chrony.conf, not /etc/chrony/chrony.conf.

Mirrors

In /etc/yum.repos.d, for each .repo file, comment out the 'mirrorlist=' line and e.g.

baseurl=http://mirror.csclub.uwaterloo.ca/$contentdir/$stream/BaseOS/$basearch/os/

Replace 'BaseOS' by the name of the repo. For the Extras repo, write 'extras' as the repo name.

cloud.cfg configuration

There is a bug in the cloud-init configuration which was fixed in RHEL 8.2 but which doesn't seem to have made it to CentOS Stream 8 (it may already be fixed in CentOS Stream 9). See here and here.

In /etc/cloud/cloud.cfg, comment out the line 'ssh_genkeytypes: ~' and add this line:

    ssh_genkeytypes: ['rsa', 'ecdsa', 'ed25519']

Also add the following to /etc/cloud/cloud.cfg:

    manage_etc_hosts: true

Hosts configuration

Paste the following into /etc/hosts:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Paste 'localhost.localdomain' into /etc/hostname.

Cleanup

Run dnf clean all instead of apt clean.