CSCWiki - User contributions [en]

New CSC Machine

2026-09-04T22:18:25Z

K95ma: /* General */ +

= Firmware Updates =

Vendors such as Dell provide firmware updates that should be applied before putting new machines into service. Even if the machine's warranty has expired, security updates are still made available.

It is recommended to use the following sequence when updating firmware on the Dell PowerEdge servers ([https://downloads.dell.com/solutions/general-solution-resources/White%20Papers/Recommended%20Workflow%20for%20Performing%20Firmware%20Updates%20on%20PowerEdge%20Servers.pdf]):

# iDRAC
# Lifecycle Controller
# BIOS
# Diagnostics
# OS Driver Pack
# RAID
# NIC
# PSU
# CPLD
# Other update
For consumer grade hardware, go to the motherboard vendor's website and find the way to upgrade the firmware.

= Booting =

* Put the TFTP image in place (if dist-arch pair installed before, you may skip this).
e.g. extract http://mirror.csclub.uwaterloo.ca/ubuntu/dists/oneiric/main/installer-amd64/current/images/netboot/netboot.tar.gz to caffeine:/srv/tftp/oneiric-amd64

* Force network boot in the BIOS. This may be called "Legacy LAN" or other such cryptic things. If this doesn't work, boot from CD or USB instead.

It is preferred to use the "alternate" Ubuntu installer image, based on debian-installer, instead of the Ubiquity installer. This installer supports software RAID and LVM out of the box, and will generally make your life easier. If installing Debian, this is the usual installer, so don't sweat it.

* Most of our newer servers (e.g. PowerEdge R815) need non-free firmware in order to boot. This means that if you are using a new netboot image, it is highly recommended to include the entire non-free firmware bundle in the boot image. See [https://wiki.debian.org/DebianInstaller/NetbootFirmware] for more information.
* For office terminals, create a boot USB (via dd, for example) and boot from USB.

= Installing =

== debian-installer ==

At least in expert mode, you can choose a custom mirror (top of the countries list) and give the path for mirror directly. This will make installation super-fast compared to installing from anywhere else.

Please install to LVM volumes, as this is our standard configuration on all machines where possible. It allows more flexible partitioning across available volumes. Since GRUB 2, even /boot may be on LVM; this is the preferred configuration for simplicity, except when legacy partitioning setups make this inconvenient.

You may enable unattended upgrades, but do not enable Canonical's remote management service or any such nonsense. This is mostly a straightforward Debian/Ubuntu install.

= After Installing =

Add the machine's name to ~git/public/hosts.git, and run the ansible playbook (https://git.uwaterloo.ca/csc/playbooks/blob/master/update-hosts.yml) to distribute the updated hosts file to all machines.

== Networking ==
Make sure to setup the IPMI on the mso-private VLAN (I think VLAN 520, check under networking page), and pleaseee set it up as a static IP.

Then fetch the MAC address from the normal network interface, then on Caffine there's a dhcp server running. You'll need to add it there ({{code|/etc/dhcp/dhcpd.conf}}), just look for where all the servers are, and add it there. Make sure to assign it an IP there, otherwise it ''won't'' be assigned one.

Then using IPAM give it a name and stuff, and a domain name.

== apt ==

Delete/clear the file <tt>/etc/apt/sources.list</tt> and paste something like the following into <tt>/etc/apt/sources.list.d/debian.sources</tt> (replace "bookworm" by the the current Debian stable codename):
<pre>
Types: deb
URIs: http://mirror.csclub.uwaterloo.ca/debian
Suites: bookworm bookworm-updates bookworm-backports
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Types: deb
URIs: http://mirror.csclub.uwaterloo.ca/debian-security
Suites: bookworm-security
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
</pre>

Install the CSC archive signing key:
<pre>
wget -O /etc/apt/keyrings/csclub.gpg http://debian.csclub.uwaterloo.ca/csclub.gpg
</pre>

Paste the following into <tt>/etc/apt/sources.list.d/csclub.sources</tt> (or copy from another host):
<pre>
Types: deb
URIs: http://debian.csclub.uwaterloo.ca
Suites: bookworm
Components: main
Signed-By: /etc/apt/keyrings/csclub.gpg
</pre>

In order to make Debian use packages in our repository by default, set our repository to the highest priority. Create <code>/etc/apt/preferences.d/99-csclub</code>: <syntaxhighlight>
Package: *
Pin: origin debian.csclub.uwaterloo.ca
Pin-Priority: 1001
</syntaxhighlight>You should now run <tt>apt-get update</tt> to reflect these changes.

For unattended upgrades in the future, install the <tt>unattended-upgrades</tt> package and copy <tt>/etc/apt/apt.conf</tt> from another host.

== Network ==

Note that debian 11 will use NetworkManager or <code>/etc/interfaces</code> by default if you install a desktop environment, which doesn't seem to do DHCPv6 nicely. For simplicity and consistency across machines, we will use <code>systemd-networkd</code>. First stop and disable NetworkManager:<syntaxhighlight lang="bash">
systemctl disable --now NetworkManager.service networking.service
apt autoremove NetworkManager
</syntaxhighlight>Then, create a network configuration file at <code>/etc/systemd/network/10-wired.network</code>:<syntaxhighlight>
[Match]
# Check the interface name using `ip a`
Name=enp3s0

[Network]
# DHCP for IPv4 should work just fine
DHCP=ipv4
# IPv6 doesn't seem to work properly. Manually set them here
Address=ALLOCATED_IPv6_ADDRESS
Gateway=IPv6_GATEWAY
</syntaxhighlight>Then start and enable <code>systemd-networkd.service</code>. Also remember to specify the campus DNS at <code>/etc/resolve.conf</code>. You can copy it from another CSC machine.

== Kerberos keys ==

If this is a reinstall of an existing host, copy back the SSH host keys and <tt>/etc/krb5.keytab</tt> from its former incarnation. Otherwise, create a new Kerberos principal and copy the keytab over, as follows (run from the host in question):<syntaxhighlight>
kadmin -p sysadmin/admin # or any other admin principal; the password for this one is the usual root password
addprinc -randkey host/[hostname].csclub.uwaterloo.ca
ktadd host/[hostname].csclub.uwaterloo.ca
</syntaxhighlight>This will generate a new principal (you can skip this step if one already exists) and add it to the local Kerberos keytab.

== Configuration ==

=== General ===
Install packages that we will need:<syntaxhighlight lang="bash">
apt install krb5-user nfs-common nslcd sudo-ldap
</syntaxhighlight>

Installing nsclcd will pop a dialog for configuring LDAP servers. Enter the following:

<syntaxhighlight lang="bash">
ldaps://ldap1.csclub.uwaterloo.ca ldaps://ldap2.csclub.uwaterloo.ca
</syntaxhighlight>

The following config files are needed to work in the CSC environment (examples given below for an office terminal; perhaps refer to another host if preferred).

<tt>/etc/nsswitch.conf</tt><syntaxhighlight>
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd: files systemd ldap
group: files systemd ldap
shadow: files ldap
gshadow: files ldap
sudoers: files ldap

hosts: files dns
networks: files

protocols: db files
services: db files
ethers: db files
rpc: db files
</syntaxhighlight><tt>/etc/ldap/ldap.conf</tt><syntaxhighlight>
#
# LDAP Defaults
#

# See ldap.conf(5) for details
# This file should be world readable but not world writable.

BASE dc=csclub, dc=uwaterloo, dc=ca
URI ldaps://ldap1.csclub.uwaterloo.ca ldaps://ldap2.csclub.uwaterloo.ca

SIZELIMIT 0

TLS_CACERT /etc/ssl/certs/ca-certificates.crt
TLS_CACERTFILE /etc/ssl/certs/ca-certificates.crt

SUDOERS_BASE ou=SUDOers,dc=csclub,dc=uwaterloo,dc=ca
</syntaxhighlight>Also make <tt>/etc/sudo-ldap.conf</tt> a symlink to the above. On debian, install <tt>sudo-ldap</tt> package too.

<tt>/etc/nslcd.conf</tt><syntaxhighlight>
# /etc/nslcd.conf
# nslcd configuration file. See nslcd.conf(5)
# for details.

# The user and group nslcd should run as.
uid nslcd
gid nslcd

# The location at which the LDAP server(s) should be reachable.
uri ldaps://ldap1.csclub.uwaterloo.ca
uri ldaps://ldap2.csclub.uwaterloo.ca

# The search base that will be used for all queries.
base dc=csclub, dc=uwaterloo, dc=ca

# The LDAP protocol version to use.
#ldap_version 3

# The DN to bind with for normal lookups.
#binddn cn=annonymous,dc=example,dc=net
#bindpw secret

# The DN used for password modifications by root.
#rootpwmoddn cn=admin,dc=example,dc=com

# SSL options
#ssl off
tls_reqcert demand
tls_cacertfile /etc/ssl/certs/ca-certificates.crt

# The search scope.
#scope sub

map group member uniqueMember
</syntaxhighlight><tt>/etc/krb5.conf</tt><syntaxhighlight>
[libdefaults]
default_realm = CSCLUB.UWATERLOO.CA
forwardable = true
proxiable = true
dns_lookup_kdc = false
dns_lookup_realm = false
allow_weak_crypto = true

[realms]
CSCLUB.UWATERLOO.CA = {
kdc = kdc1.csclub.uwaterloo.ca
kdc = kdc2.csclub.uwaterloo.ca
admin_server = kadmin.csclub.uwaterloo.ca
}
(rest omitted for brevity, see any CSC machine)
</syntaxhighlight>Notably, <tt>allow_weak_crypto</tt> is currently needed to mount <tt>/users</tt> (/music and <tt>/scratch</tt> is sec=sys and thus will always mount, even when krb5 is down and/or broken). Otherwise, you will get a mysterious "permission denied" error (even though the server claims to have authenticated the mount successfully).

Furthermore, the lines <tt>dns_lookup_kdc</tt> and <tt>dns_lookup_realm</tt> have been added - they are needed to stop the KDC from throwing its arms in the air and giving up if IST's DNS servers ever explode - an event that has happened in the recent past far more often than I'd like it to.

Change all lines in <tt>/etc/pam.d/common-*</tt> to have <tt>minimum_uid=10000</tt> so that Kerberos won't interfere with local users. Note that pam configs are notably different on syscom-only hosts. Look at an existing syscom-only host to see the difference.

Alter <tt>/etc/default/nfs-common</tt> <syntaxhighlight>
# Alter these lines:
NEED_STATD=1
NEED_GSSD=1
# -l for gssd is to allow legacy crypto suites
RPCGSSDOPTS="-v -l"
</syntaxhighlight>to enable <tt>statd</tt>, and more importantly <tt>gssd</tt> (needed for Kerberos NFS mounts). Start <code>rpc-statd.service</code> and <code>rpc-gssd.service</code> manually for now.

Add <tt>/users</tt>, <tt>/music</tt> and <tt>/scratch</tt> to <tt>/etc/fstab</tt> (as appropriate for the machine's role), make their mount points and mount them. Note that <tt>/scratch</tt> are sec=sys whereas <tt>/music</tt> and /users is sec=krb5p (with exceptions granted on a case-by-case basis for servers only, office terminals are always sec=krb5p for security reasons).

To allow single sign-on as <tt>root</tt> (primarily useful for pushing files to all machines simultaneously), put the following in <tt>/root/.k5login</tt>:
sysadmin/admin@CSCLUB.UWATERLOO.CA

Also copy the following files from another CSC host:
* <tt>/etc/ssh/ssh_config</tt> and <tt>/etc/ssh/sshd_config</tt> (for single sign-on)
* <tt>/etc/ssh/ssh_known_hosts</tt> (to remove hostkey warnings within our network)
* <tt>/etc/hosts</tt> (for host tab completion and emergency name resolution)
* <tt>/etc/resolv.conf</tt> (to use IST's nameservers and search csclub/uwaterloo domains. Only required if you are not using <tt>/etc/network/interfaces</tt> to configure DNS)

=== Audio ===

On an office terminal, copy <tt>/etc/pulse/default.pa</tt> from another office terminal.

If this is to be the machine that actually plays audio (currently <tt>nullsleep</tt>), the setup is slightly more complicated. You'll need to set up MPD and PipeWire to receive connections, and store the PulseAudio cookie in <tt>~audio</tt>, with appropriate permissions so that only the <tt>audio</tt> group can access it. If this is a new audio machine, you'll also need to change <tt>default.pa</tt> on all office terminals to point to it.

=== Password ===
Change the root password to the specified password in the usual place under the termcom user. If it's an office terminal, change the local user's password to the one specified in the usual place.

=== Prevent suspend and hibernation (Office Terminal) ===
Set <code>AllowSuspend</code>, <code>AllowHibernation</code>, <code>AllowSuspendThenHibernate</code> and <code>AllowHybridSleep</code> all to <code>no</code> in <code>/etc/systemd/sleep.conf</code>, and reboot.

== Records ==

You probably already created the host in the University IPAM system beforehand. If not, please do so.

Please also add the host to the [[Machine List]] here on the Wiki.

== Munin (System Monitoring) ==

If the new machine is not a container, you probably want to have it participate in the Munin cluster. Run <tt>apt-get install munin-node</tt> to install the monitoring client, then
edit the file /etc/munin/munin-node.conf. Look for a line that says <tt>allow ^127\.0\.0\.1$</tt> and add the following on a new line immediately below it:
<tt>allow ^129\.97\.134\.51$</tt> (this is the IP address for munin.csclub). Save the file, then <tt>/etc/init.d/munin-node restart</tt> and <tt>update-rc.d munin-node defaults</tt>.

Then, ssh into munin.csclub and edit the file /etc/munin/munin.conf and add the following lines to the end:
<tt>

[NEW-MACHINE-NAME.csclub] 
addr 129.97.134.### 
use_node_name yes</tt>

== Prometheus (System Monitoring) ==

We are currently using Prometheus to monitor our systems. On the new machine, install <tt>prometheus-node-exporter</tt> and <tt>stunnel</tt>.

Change <tt>/etc/default/prometheus-node-exporter</tt> to this:

ARGS="--web.listen-address=localhost:9101"

and start <tt>prometheus-node-exporter.service</tt>.

Then set up stunnel. Create <tt>/etc/stunnel/prometheus-node-exporter.conf</tt> with this content:

setuid = stunnel4
setgid = stunnel4
pid = /var/run/stunnel4/exporter.pid

debug = 7

[prometheus-node-exporter]
accept = 0.0.0.0:9100
connect = 127.0.0.1:9101
CAfile = /etc/stunnel/tls/server.crt
cert = /etc/stunnel/tls/node.crt
key = /etc/stunnel/tls/node.key
verifyPeer = yes

Copy <tt>/etc/stunnel/{node.crt, node.key, server.crt}</tt> from <tt>prometheus:/opt/prometheus/tls</tt> or the same location on other machines.

Finally, start <tt>stunnel4.service</tt>.

If it's a new machine, you'll also need to add it to the list of monitoring at <tt>prometheus:/opt/prometheus/prometheus.yml</tt>. Add it under a suitable label (or create a new label) in 'node_exporter' job.

= New Distribution =

If you're adding a new distribution, there a couple of steps you'll need to take in updating the CSClub Debian repository on [[Machine_List#sodium_benzoate|sodium-benzoate/mirror]].

The steps to add a new Debian release (in the examples, jessie) is as follows, modify as necessary:

=== Step 0: Create a GPG key ===

Use "gpg --gen-key" or something like that. Skip this if you already have one.

=== Step 1: Add to Uploaders ===

The /srv/debian/conf/uploaders file on mirror contains the list of people who can upload. Add your GPG key id to this file. Use "gpg --list-secret-keys" to find out the key ID. You also need to import your key into the mirror's gpg homedir as follows:

gpg --export $KEYID | sudo env GNUPGHOME=/srv/debian/gpg gpg --import

You only need to do this step once.

=== Step 2: Add Distro ===

Add a new section to /srv/debian/conf/distributions:

Origin: CSC
Label: Debian
Codename: '''jessie'''
Architectures: alpha amd64 i386 mips mipsel sparc powerpc armel source
Components: main contrib non-free
Uploaders: uploaders
Update: dell chrome
SignWith: yes
Log: '''jessie'''.log
--changes notifier

And update the '''Allow''' line in /srv/debian/conf/incoming:

Allow: '''jessie>jessie''' oldstable>squeeze stable>wheezy lucid>lucid maverick>maverick oneiric>oneiric precise>precise quantal>quantal

=== Step 3: Update from Sources ===

Run:

sudo env GNUPGHOME=/srv/debian/gpg /srv/debian/bin/rrr-update

If all went well you should see the new distribution listed at http://debian.csclub.uwaterloo.ca/dists/

=== Step 4: CSC Packages ===

Now that we've got our new distribution set up we need to generate our packages and have them uploaded. Namely, ceo and libpam-csc. For libpam-csc:

Get the package:

git clone https://git.csclub.uwaterloo.ca/public/libpam-csc.git
cd libpam-csc

Update change log:

EMAIL=[you]@csclub.uwaterloo.ca NAME="Your Name" dch -i

Update as necessary, i.e:

libpam-csc (1.10'''jessie0''') '''jessie'''; urgency=low

* Packaging for jessie.

-- Your Name <[you]@csclub.uwaterloo.ca> Thu, 10 Oct 2013 22:08:48 -0400

Build! (You may need to install various dependencies, which it will yell at you if you don't have.)

debuild -k'''YOURKEYID'''

Yay, it built now let's upload it to the repo. The build process which create a PACKAGE.changes file in the parent directory (replace PACKAGE with the actual package name).

Copy the dupload file from corn-syrup and dupload:

mv /etc/dupload /etc/dupload.bak
scp corn-syrup:/etc/dupload /etc/dupload
dupload libpam-csc_1.10jessie0_amd64.changes

Finally, log into mirror and type "sudo /srv/debian/bin/rrr-incoming". This is supposed to happen once every few minutes however it is always faster to run it manually.

And you're done. For CEO, see https://git.csclub.uwaterloo.ca/public/pyceo/src/branch/master/PACKAGING.md

New CSC Machine

2026-09-04T22:16:39Z

K95ma: /* General */ de

= Firmware Updates =

Vendors such as Dell provide firmware updates that should be applied before putting new machines into service. Even if the machine's warranty has expired, security updates are still made available.

It is recommended to use the following sequence when updating firmware on the Dell PowerEdge servers ([https://downloads.dell.com/solutions/general-solution-resources/White%20Papers/Recommended%20Workflow%20for%20Performing%20Firmware%20Updates%20on%20PowerEdge%20Servers.pdf]):

# iDRAC
# Lifecycle Controller
# BIOS
# Diagnostics
# OS Driver Pack
# RAID
# NIC
# PSU
# CPLD
# Other update
For consumer grade hardware, go to the motherboard vendor's website and find the way to upgrade the firmware.

= Booting =

* Put the TFTP image in place (if dist-arch pair installed before, you may skip this).
e.g. extract http://mirror.csclub.uwaterloo.ca/ubuntu/dists/oneiric/main/installer-amd64/current/images/netboot/netboot.tar.gz to caffeine:/srv/tftp/oneiric-amd64

* Force network boot in the BIOS. This may be called "Legacy LAN" or other such cryptic things. If this doesn't work, boot from CD or USB instead.

It is preferred to use the "alternate" Ubuntu installer image, based on debian-installer, instead of the Ubiquity installer. This installer supports software RAID and LVM out of the box, and will generally make your life easier. If installing Debian, this is the usual installer, so don't sweat it.

* Most of our newer servers (e.g. PowerEdge R815) need non-free firmware in order to boot. This means that if you are using a new netboot image, it is highly recommended to include the entire non-free firmware bundle in the boot image. See [https://wiki.debian.org/DebianInstaller/NetbootFirmware] for more information.
* For office terminals, create a boot USB (via dd, for example) and boot from USB.

= Installing =

== debian-installer ==

At least in expert mode, you can choose a custom mirror (top of the countries list) and give the path for mirror directly. This will make installation super-fast compared to installing from anywhere else.

Please install to LVM volumes, as this is our standard configuration on all machines where possible. It allows more flexible partitioning across available volumes. Since GRUB 2, even /boot may be on LVM; this is the preferred configuration for simplicity, except when legacy partitioning setups make this inconvenient.

You may enable unattended upgrades, but do not enable Canonical's remote management service or any such nonsense. This is mostly a straightforward Debian/Ubuntu install.

= After Installing =

Add the machine's name to ~git/public/hosts.git, and run the ansible playbook (https://git.uwaterloo.ca/csc/playbooks/blob/master/update-hosts.yml) to distribute the updated hosts file to all machines.

== Networking ==
Make sure to setup the IPMI on the mso-private VLAN (I think VLAN 520, check under networking page), and pleaseee set it up as a static IP.

Then fetch the MAC address from the normal network interface, then on Caffine there's a dhcp server running. You'll need to add it there ({{code|/etc/dhcp/dhcpd.conf}}), just look for where all the servers are, and add it there. Make sure to assign it an IP there, otherwise it ''won't'' be assigned one.

Then using IPAM give it a name and stuff, and a domain name.

== apt ==

Delete/clear the file <tt>/etc/apt/sources.list</tt> and paste something like the following into <tt>/etc/apt/sources.list.d/debian.sources</tt> (replace "bookworm" by the the current Debian stable codename):
<pre>
Types: deb
URIs: http://mirror.csclub.uwaterloo.ca/debian
Suites: bookworm bookworm-updates bookworm-backports
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Types: deb
URIs: http://mirror.csclub.uwaterloo.ca/debian-security
Suites: bookworm-security
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
</pre>

Install the CSC archive signing key:
<pre>
wget -O /etc/apt/keyrings/csclub.gpg http://debian.csclub.uwaterloo.ca/csclub.gpg
</pre>

Paste the following into <tt>/etc/apt/sources.list.d/csclub.sources</tt> (or copy from another host):
<pre>
Types: deb
URIs: http://debian.csclub.uwaterloo.ca
Suites: bookworm
Components: main
Signed-By: /etc/apt/keyrings/csclub.gpg
</pre>

In order to make Debian use packages in our repository by default, set our repository to the highest priority. Create <code>/etc/apt/preferences.d/99-csclub</code>: <syntaxhighlight>
Package: *
Pin: origin debian.csclub.uwaterloo.ca
Pin-Priority: 1001
</syntaxhighlight>You should now run <tt>apt-get update</tt> to reflect these changes.

For unattended upgrades in the future, install the <tt>unattended-upgrades</tt> package and copy <tt>/etc/apt/apt.conf</tt> from another host.

== Network ==

Note that debian 11 will use NetworkManager or <code>/etc/interfaces</code> by default if you install a desktop environment, which doesn't seem to do DHCPv6 nicely. For simplicity and consistency across machines, we will use <code>systemd-networkd</code>. First stop and disable NetworkManager:<syntaxhighlight lang="bash">
systemctl disable --now NetworkManager.service networking.service
apt autoremove NetworkManager
</syntaxhighlight>Then, create a network configuration file at <code>/etc/systemd/network/10-wired.network</code>:<syntaxhighlight>
[Match]
# Check the interface name using `ip a`
Name=enp3s0

[Network]
# DHCP for IPv4 should work just fine
DHCP=ipv4
# IPv6 doesn't seem to work properly. Manually set them here
Address=ALLOCATED_IPv6_ADDRESS
Gateway=IPv6_GATEWAY
</syntaxhighlight>Then start and enable <code>systemd-networkd.service</code>. Also remember to specify the campus DNS at <code>/etc/resolve.conf</code>. You can copy it from another CSC machine.

== Kerberos keys ==

If this is a reinstall of an existing host, copy back the SSH host keys and <tt>/etc/krb5.keytab</tt> from its former incarnation. Otherwise, create a new Kerberos principal and copy the keytab over, as follows (run from the host in question):<syntaxhighlight>
kadmin -p sysadmin/admin # or any other admin principal; the password for this one is the usual root password
addprinc -randkey host/[hostname].csclub.uwaterloo.ca
ktadd host/[hostname].csclub.uwaterloo.ca
</syntaxhighlight>This will generate a new principal (you can skip this step if one already exists) and add it to the local Kerberos keytab.

== Configuration ==

=== General ===
Install packages that we will need:<syntaxhighlight lang="bash">
apt install krb5-user nfs-common nslcd sudo-ldap
</syntaxhighlight>The following config files are needed to work in the CSC environment (examples given below for an office terminal; perhaps refer to another host if preferred).

<tt>/etc/nsswitch.conf</tt><syntaxhighlight>
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd: files systemd ldap
group: files systemd ldap
shadow: files ldap
gshadow: files ldap
sudoers: files ldap

hosts: files dns
networks: files

protocols: db files
services: db files
ethers: db files
rpc: db files
</syntaxhighlight><tt>/etc/ldap/ldap.conf</tt><syntaxhighlight>
#
# LDAP Defaults
#

# See ldap.conf(5) for details
# This file should be world readable but not world writable.

BASE dc=csclub, dc=uwaterloo, dc=ca
URI ldaps://ldap1.csclub.uwaterloo.ca ldaps://ldap2.csclub.uwaterloo.ca

SIZELIMIT 0

TLS_CACERT /etc/ssl/certs/ca-certificates.crt
TLS_CACERTFILE /etc/ssl/certs/ca-certificates.crt

SUDOERS_BASE ou=SUDOers,dc=csclub,dc=uwaterloo,dc=ca
</syntaxhighlight>Also make <tt>/etc/sudo-ldap.conf</tt> a symlink to the above. On debian, install <tt>sudo-ldap</tt> package too.

<tt>/etc/nslcd.conf</tt><syntaxhighlight>
# /etc/nslcd.conf
# nslcd configuration file. See nslcd.conf(5)
# for details.

# The user and group nslcd should run as.
uid nslcd
gid nslcd

# The location at which the LDAP server(s) should be reachable.
uri ldaps://ldap1.csclub.uwaterloo.ca
uri ldaps://ldap2.csclub.uwaterloo.ca

# The search base that will be used for all queries.
base dc=csclub, dc=uwaterloo, dc=ca

# The LDAP protocol version to use.
#ldap_version 3

# The DN to bind with for normal lookups.
#binddn cn=annonymous,dc=example,dc=net
#bindpw secret

# The DN used for password modifications by root.
#rootpwmoddn cn=admin,dc=example,dc=com

# SSL options
#ssl off
tls_reqcert demand
tls_cacertfile /etc/ssl/certs/ca-certificates.crt

# The search scope.
#scope sub

map group member uniqueMember
</syntaxhighlight><tt>/etc/krb5.conf</tt><syntaxhighlight>
[libdefaults]
default_realm = CSCLUB.UWATERLOO.CA
forwardable = true
proxiable = true
dns_lookup_kdc = false
dns_lookup_realm = false
allow_weak_crypto = true

[realms]
CSCLUB.UWATERLOO.CA = {
kdc = kdc1.csclub.uwaterloo.ca
kdc = kdc2.csclub.uwaterloo.ca
admin_server = kadmin.csclub.uwaterloo.ca
}
(rest omitted for brevity, see any CSC machine)
</syntaxhighlight>Notably, <tt>allow_weak_crypto</tt> is currently needed to mount <tt>/users</tt> (/music and <tt>/scratch</tt> is sec=sys and thus will always mount, even when krb5 is down and/or broken). Otherwise, you will get a mysterious "permission denied" error (even though the server claims to have authenticated the mount successfully).

Furthermore, the lines <tt>dns_lookup_kdc</tt> and <tt>dns_lookup_realm</tt> have been added - they are needed to stop the KDC from throwing its arms in the air and giving up if IST's DNS servers ever explode - an event that has happened in the recent past far more often than I'd like it to.

Change all lines in <tt>/etc/pam.d/common-*</tt> to have <tt>minimum_uid=10000</tt> so that Kerberos won't interfere with local users. Note that pam configs are notably different on syscom-only hosts. Look at an existing syscom-only host to see the difference.

Alter <tt>/etc/default/nfs-common</tt> <syntaxhighlight>
# Alter these lines:
NEED_STATD=1
NEED_GSSD=1
# -l for gssd is to allow legacy crypto suites
RPCGSSDOPTS="-v -l"
</syntaxhighlight>to enable <tt>statd</tt>, and more importantly <tt>gssd</tt> (needed for Kerberos NFS mounts). Start <code>rpc-statd.service</code> and <code>rpc-gssd.service</code> manually for now.

Add <tt>/users</tt>, <tt>/music</tt> and <tt>/scratch</tt> to <tt>/etc/fstab</tt> (as appropriate for the machine's role), make their mount points and mount them. Note that <tt>/scratch</tt> are sec=sys whereas <tt>/music</tt> and /users is sec=krb5p (with exceptions granted on a case-by-case basis for servers only, office terminals are always sec=krb5p for security reasons).

To allow single sign-on as <tt>root</tt> (primarily useful for pushing files to all machines simultaneously), put the following in <tt>/root/.k5login</tt>:
sysadmin/admin@CSCLUB.UWATERLOO.CA

Also copy the following files from another CSC host:
* <tt>/etc/ssh/ssh_config</tt> and <tt>/etc/ssh/sshd_config</tt> (for single sign-on)
* <tt>/etc/ssh/ssh_known_hosts</tt> (to remove hostkey warnings within our network)
* <tt>/etc/hosts</tt> (for host tab completion and emergency name resolution)
* <tt>/etc/resolv.conf</tt> (to use IST's nameservers and search csclub/uwaterloo domains. Only required if you are not using <tt>/etc/network/interfaces</tt> to configure DNS)

=== Audio ===

On an office terminal, copy <tt>/etc/pulse/default.pa</tt> from another office terminal.

If this is to be the machine that actually plays audio (currently <tt>nullsleep</tt>), the setup is slightly more complicated. You'll need to set up MPD and PipeWire to receive connections, and store the PulseAudio cookie in <tt>~audio</tt>, with appropriate permissions so that only the <tt>audio</tt> group can access it. If this is a new audio machine, you'll also need to change <tt>default.pa</tt> on all office terminals to point to it.

=== Password ===
Change the root password to the specified password in the usual place under the termcom user. If it's an office terminal, change the local user's password to the one specified in the usual place.

=== Prevent suspend and hibernation (Office Terminal) ===
Set <code>AllowSuspend</code>, <code>AllowHibernation</code>, <code>AllowSuspendThenHibernate</code> and <code>AllowHybridSleep</code> all to <code>no</code> in <code>/etc/systemd/sleep.conf</code>, and reboot.

== Records ==

You probably already created the host in the University IPAM system beforehand. If not, please do so.

Please also add the host to the [[Machine List]] here on the Wiki.

== Munin (System Monitoring) ==

If the new machine is not a container, you probably want to have it participate in the Munin cluster. Run <tt>apt-get install munin-node</tt> to install the monitoring client, then
edit the file /etc/munin/munin-node.conf. Look for a line that says <tt>allow ^127\.0\.0\.1$</tt> and add the following on a new line immediately below it:
<tt>allow ^129\.97\.134\.51$</tt> (this is the IP address for munin.csclub). Save the file, then <tt>/etc/init.d/munin-node restart</tt> and <tt>update-rc.d munin-node defaults</tt>.

Then, ssh into munin.csclub and edit the file /etc/munin/munin.conf and add the following lines to the end:
<tt>

[NEW-MACHINE-NAME.csclub] 
addr 129.97.134.### 
use_node_name yes</tt>

== Prometheus (System Monitoring) ==

We are currently using Prometheus to monitor our systems. On the new machine, install <tt>prometheus-node-exporter</tt> and <tt>stunnel</tt>.

Change <tt>/etc/default/prometheus-node-exporter</tt> to this:

ARGS="--web.listen-address=localhost:9101"

and start <tt>prometheus-node-exporter.service</tt>.

Then set up stunnel. Create <tt>/etc/stunnel/prometheus-node-exporter.conf</tt> with this content:

setuid = stunnel4
setgid = stunnel4
pid = /var/run/stunnel4/exporter.pid

debug = 7

[prometheus-node-exporter]
accept = 0.0.0.0:9100
connect = 127.0.0.1:9101
CAfile = /etc/stunnel/tls/server.crt
cert = /etc/stunnel/tls/node.crt
key = /etc/stunnel/tls/node.key
verifyPeer = yes

Copy <tt>/etc/stunnel/{node.crt, node.key, server.crt}</tt> from <tt>prometheus:/opt/prometheus/tls</tt> or the same location on other machines.

Finally, start <tt>stunnel4.service</tt>.

If it's a new machine, you'll also need to add it to the list of monitoring at <tt>prometheus:/opt/prometheus/prometheus.yml</tt>. Add it under a suitable label (or create a new label) in 'node_exporter' job.

= New Distribution =

If you're adding a new distribution, there a couple of steps you'll need to take in updating the CSClub Debian repository on [[Machine_List#sodium_benzoate|sodium-benzoate/mirror]].

The steps to add a new Debian release (in the examples, jessie) is as follows, modify as necessary:

=== Step 0: Create a GPG key ===

Use "gpg --gen-key" or something like that. Skip this if you already have one.

=== Step 1: Add to Uploaders ===

The /srv/debian/conf/uploaders file on mirror contains the list of people who can upload. Add your GPG key id to this file. Use "gpg --list-secret-keys" to find out the key ID. You also need to import your key into the mirror's gpg homedir as follows:

gpg --export $KEYID | sudo env GNUPGHOME=/srv/debian/gpg gpg --import

You only need to do this step once.

=== Step 2: Add Distro ===

Add a new section to /srv/debian/conf/distributions:

Origin: CSC
Label: Debian
Codename: '''jessie'''
Architectures: alpha amd64 i386 mips mipsel sparc powerpc armel source
Components: main contrib non-free
Uploaders: uploaders
Update: dell chrome
SignWith: yes
Log: '''jessie'''.log
--changes notifier

And update the '''Allow''' line in /srv/debian/conf/incoming:

Allow: '''jessie>jessie''' oldstable>squeeze stable>wheezy lucid>lucid maverick>maverick oneiric>oneiric precise>precise quantal>quantal

=== Step 3: Update from Sources ===

Run:

sudo env GNUPGHOME=/srv/debian/gpg /srv/debian/bin/rrr-update

If all went well you should see the new distribution listed at http://debian.csclub.uwaterloo.ca/dists/

=== Step 4: CSC Packages ===

Now that we've got our new distribution set up we need to generate our packages and have them uploaded. Namely, ceo and libpam-csc. For libpam-csc:

Get the package:

git clone https://git.csclub.uwaterloo.ca/public/libpam-csc.git
cd libpam-csc

Update change log:

EMAIL=[you]@csclub.uwaterloo.ca NAME="Your Name" dch -i

Update as necessary, i.e:

libpam-csc (1.10'''jessie0''') '''jessie'''; urgency=low

* Packaging for jessie.

-- Your Name <[you]@csclub.uwaterloo.ca> Thu, 10 Oct 2013 22:08:48 -0400

Build! (You may need to install various dependencies, which it will yell at you if you don't have.)

debuild -k'''YOURKEYID'''

Yay, it built now let's upload it to the repo. The build process which create a PACKAGE.changes file in the parent directory (replace PACKAGE with the actual package name).

Copy the dupload file from corn-syrup and dupload:

mv /etc/dupload /etc/dupload.bak
scp corn-syrup:/etc/dupload /etc/dupload
dupload libpam-csc_1.10jessie0_amd64.changes

Finally, log into mirror and type "sudo /srv/debian/bin/rrr-incoming". This is supposed to happen once every few minutes however it is always faster to run it manually.

And you're done. For CEO, see https://git.csclub.uwaterloo.ca/public/pyceo/src/branch/master/PACKAGING.md

New CSC Machine

2026-09-04T22:12:06Z

K95ma: /* Networking */ +

= Firmware Updates =

Vendors such as Dell provide firmware updates that should be applied before putting new machines into service. Even if the machine's warranty has expired, security updates are still made available.

It is recommended to use the following sequence when updating firmware on the Dell PowerEdge servers ([https://downloads.dell.com/solutions/general-solution-resources/White%20Papers/Recommended%20Workflow%20for%20Performing%20Firmware%20Updates%20on%20PowerEdge%20Servers.pdf]):

# iDRAC
# Lifecycle Controller
# BIOS
# Diagnostics
# OS Driver Pack
# RAID
# NIC
# PSU
# CPLD
# Other update
For consumer grade hardware, go to the motherboard vendor's website and find the way to upgrade the firmware.

= Booting =

* Put the TFTP image in place (if dist-arch pair installed before, you may skip this).
e.g. extract http://mirror.csclub.uwaterloo.ca/ubuntu/dists/oneiric/main/installer-amd64/current/images/netboot/netboot.tar.gz to caffeine:/srv/tftp/oneiric-amd64

* Force network boot in the BIOS. This may be called "Legacy LAN" or other such cryptic things. If this doesn't work, boot from CD or USB instead.

It is preferred to use the "alternate" Ubuntu installer image, based on debian-installer, instead of the Ubiquity installer. This installer supports software RAID and LVM out of the box, and will generally make your life easier. If installing Debian, this is the usual installer, so don't sweat it.

* Most of our newer servers (e.g. PowerEdge R815) need non-free firmware in order to boot. This means that if you are using a new netboot image, it is highly recommended to include the entire non-free firmware bundle in the boot image. See [https://wiki.debian.org/DebianInstaller/NetbootFirmware] for more information.
* For office terminals, create a boot USB (via dd, for example) and boot from USB.

= Installing =

== debian-installer ==

At least in expert mode, you can choose a custom mirror (top of the countries list) and give the path for mirror directly. This will make installation super-fast compared to installing from anywhere else.

Please install to LVM volumes, as this is our standard configuration on all machines where possible. It allows more flexible partitioning across available volumes. Since GRUB 2, even /boot may be on LVM; this is the preferred configuration for simplicity, except when legacy partitioning setups make this inconvenient.

You may enable unattended upgrades, but do not enable Canonical's remote management service or any such nonsense. This is mostly a straightforward Debian/Ubuntu install.

= After Installing =

Add the machine's name to ~git/public/hosts.git, and run the ansible playbook (https://git.uwaterloo.ca/csc/playbooks/blob/master/update-hosts.yml) to distribute the updated hosts file to all machines.

== Networking ==
Make sure to setup the IPMI on the mso-private VLAN (I think VLAN 520, check under networking page), and pleaseee set it up as a static IP.

Then fetch the MAC address from the normal network interface, then on Caffine there's a dhcp server running. You'll need to add it there ({{code|/etc/dhcp/dhcpd.conf}}), just look for where all the servers are, and add it there. Make sure to assign it an IP there, otherwise it ''won't'' be assigned one.

Then using IPAM give it a name and stuff, and a domain name.

== apt ==

Delete/clear the file <tt>/etc/apt/sources.list</tt> and paste something like the following into <tt>/etc/apt/sources.list.d/debian.sources</tt> (replace "bookworm" by the the current Debian stable codename):
<pre>
Types: deb
URIs: http://mirror.csclub.uwaterloo.ca/debian
Suites: bookworm bookworm-updates bookworm-backports
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

Types: deb
URIs: http://mirror.csclub.uwaterloo.ca/debian-security
Suites: bookworm-security
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
</pre>

Install the CSC archive signing key:
<pre>
wget -O /etc/apt/keyrings/csclub.gpg http://debian.csclub.uwaterloo.ca/csclub.gpg
</pre>

Paste the following into <tt>/etc/apt/sources.list.d/csclub.sources</tt> (or copy from another host):
<pre>
Types: deb
URIs: http://debian.csclub.uwaterloo.ca
Suites: bookworm
Components: main
Signed-By: /etc/apt/keyrings/csclub.gpg
</pre>

In order to make Debian use packages in our repository by default, set our repository to the highest priority. Create <code>/etc/apt/preferences.d/99-csclub</code>: <syntaxhighlight>
Package: *
Pin: origin debian.csclub.uwaterloo.ca
Pin-Priority: 1001
</syntaxhighlight>You should now run <tt>apt-get update</tt> to reflect these changes.

For unattended upgrades in the future, install the <tt>unattended-upgrades</tt> package and copy <tt>/etc/apt/apt.conf</tt> from another host.

== Network ==

Note that debian 11 will use NetworkManager or <code>/etc/interfaces</code> by default if you install a desktop environment, which doesn't seem to do DHCPv6 nicely. For simplicity and consistency across machines, we will use <code>systemd-networkd</code>. First stop and disable NetworkManager:<syntaxhighlight lang="bash">
systemctl disable --now NetworkManager.service networking.service
apt autoremove NetworkManager
</syntaxhighlight>Then, create a network configuration file at <code>/etc/systemd/network/10-wired.network</code>:<syntaxhighlight>
[Match]
# Check the interface name using `ip a`
Name=enp3s0

[Network]
# DHCP for IPv4 should work just fine
DHCP=ipv4
# IPv6 doesn't seem to work properly. Manually set them here
Address=ALLOCATED_IPv6_ADDRESS
Gateway=IPv6_GATEWAY
</syntaxhighlight>Then start and enable <code>systemd-networkd.service</code>. Also remember to specify the campus DNS at <code>/etc/resolve.conf</code>. You can copy it from another CSC machine.

== Kerberos keys ==

If this is a reinstall of an existing host, copy back the SSH host keys and <tt>/etc/krb5.keytab</tt> from its former incarnation. Otherwise, create a new Kerberos principal and copy the keytab over, as follows (run from the host in question):<syntaxhighlight>
kadmin -p sysadmin/admin # or any other admin principal; the password for this one is the usual root password
addprinc -randkey host/[hostname].csclub.uwaterloo.ca
ktadd host/[hostname].csclub.uwaterloo.ca
</syntaxhighlight>This will generate a new principal (you can skip this step if one already exists) and add it to the local Kerberos keytab.

== Configuration ==

=== General ===
Install packages that we will need:<syntaxhighlight lang="bash">
apt install krb5-user nfs-common nslcd sudo-ldap
# This package are automatically installed already, but we need to install our version so that NFS can connect to our crappy NetApp server
apt install --reinstall libk5crypto3
</syntaxhighlight>The following config files are needed to work in the CSC environment (examples given below for an office terminal; perhaps refer to another host if preferred).

<tt>/etc/nsswitch.conf</tt><syntaxhighlight>
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd: files systemd ldap
group: files systemd ldap
shadow: files ldap
gshadow: files ldap
sudoers: files ldap

hosts: files dns
networks: files

protocols: db files
services: db files
ethers: db files
rpc: db files
</syntaxhighlight><tt>/etc/ldap/ldap.conf</tt><syntaxhighlight>
#
# LDAP Defaults
#

# See ldap.conf(5) for details
# This file should be world readable but not world writable.

BASE dc=csclub, dc=uwaterloo, dc=ca
URI ldaps://ldap1.csclub.uwaterloo.ca ldaps://ldap2.csclub.uwaterloo.ca

SIZELIMIT 0

TLS_CACERT /etc/ssl/certs/ca-certificates.crt
TLS_CACERTFILE /etc/ssl/certs/ca-certificates.crt

SUDOERS_BASE ou=SUDOers,dc=csclub,dc=uwaterloo,dc=ca
</syntaxhighlight>Also make <tt>/etc/sudo-ldap.conf</tt> a symlink to the above. On debian, install <tt>sudo-ldap</tt> package too.

<tt>/etc/nslcd.conf</tt><syntaxhighlight>
# /etc/nslcd.conf
# nslcd configuration file. See nslcd.conf(5)
# for details.

# The user and group nslcd should run as.
uid nslcd
gid nslcd

# The location at which the LDAP server(s) should be reachable.
uri ldaps://ldap1.csclub.uwaterloo.ca
uri ldaps://ldap2.csclub.uwaterloo.ca

# The search base that will be used for all queries.
base dc=csclub, dc=uwaterloo, dc=ca

# The LDAP protocol version to use.
#ldap_version 3

# The DN to bind with for normal lookups.
#binddn cn=annonymous,dc=example,dc=net
#bindpw secret

# The DN used for password modifications by root.
#rootpwmoddn cn=admin,dc=example,dc=com

# SSL options
#ssl off
tls_reqcert demand
tls_cacertfile /etc/ssl/certs/ca-certificates.crt

# The search scope.
#scope sub

map group member uniqueMember
</syntaxhighlight><tt>/etc/krb5.conf</tt><syntaxhighlight>
[libdefaults]
default_realm = CSCLUB.UWATERLOO.CA
forwardable = true
proxiable = true
dns_lookup_kdc = false
dns_lookup_realm = false
allow_weak_crypto = true

[realms]
CSCLUB.UWATERLOO.CA = {
kdc = kdc1.csclub.uwaterloo.ca
kdc = kdc2.csclub.uwaterloo.ca
admin_server = kadmin.csclub.uwaterloo.ca
}
(rest omitted for brevity, see any CSC machine)
</syntaxhighlight>Notably, <tt>allow_weak_crypto</tt> is currently needed to mount <tt>/users</tt> (/music and <tt>/scratch</tt> is sec=sys and thus will always mount, even when krb5 is down and/or broken). Otherwise, you will get a mysterious "permission denied" error (even though the server claims to have authenticated the mount successfully).

Furthermore, the lines <tt>dns_lookup_kdc</tt> and <tt>dns_lookup_realm</tt> have been added - they are needed to stop the KDC from throwing its arms in the air and giving up if IST's DNS servers ever explode - an event that has happened in the recent past far more often than I'd like it to.

Change all lines in <tt>/etc/pam.d/common-*</tt> to have <tt>minimum_uid=10000</tt> so that Kerberos won't interfere with local users. Note that pam configs are notably different on syscom-only hosts. Look at an existing syscom-only host to see the difference.

Alter <tt>/etc/default/nfs-common</tt> <syntaxhighlight>
# Alter these lines:
NEED_STATD=1
NEED_GSSD=1
# -l for gssd is to allow legacy crypto suites
RPCGSSDOPTS="-v -l"
</syntaxhighlight>to enable <tt>statd</tt>, and more importantly <tt>gssd</tt> (needed for Kerberos NFS mounts). Start <code>rpc-statd.service</code> and <code>rpc-gssd.service</code> manually for now.

Add <tt>/users</tt>, <tt>/music</tt> and <tt>/scratch</tt> to <tt>/etc/fstab</tt> (as appropriate for the machine's role), make their mount points and mount them. Note that <tt>/scratch</tt> are sec=sys whereas <tt>/music</tt> and /users is sec=krb5p (with exceptions granted on a case-by-case basis for servers only, office terminals are always sec=krb5p for security reasons).

To allow single sign-on as <tt>root</tt> (primarily useful for pushing files to all machines simultaneously), put the following in <tt>/root/.k5login</tt>:
sysadmin/admin@CSCLUB.UWATERLOO.CA

Also copy the following files from another CSC host:
* <tt>/etc/ssh/ssh_config</tt> and <tt>/etc/ssh/sshd_config</tt> (for single sign-on)
* <tt>/etc/ssh/ssh_known_hosts</tt> (to remove hostkey warnings within our network)
* <tt>/etc/hosts</tt> (for host tab completion and emergency name resolution)
* <tt>/etc/resolv.conf</tt> (to use IST's nameservers and search csclub/uwaterloo domains. Only required if you are not using <tt>/etc/network/interfaces</tt> to configure DNS)

=== Audio ===

On an office terminal, copy <tt>/etc/pulse/default.pa</tt> from another office terminal.

If this is to be the machine that actually plays audio (currently <tt>nullsleep</tt>), the setup is slightly more complicated. You'll need to set up MPD and PipeWire to receive connections, and store the PulseAudio cookie in <tt>~audio</tt>, with appropriate permissions so that only the <tt>audio</tt> group can access it. If this is a new audio machine, you'll also need to change <tt>default.pa</tt> on all office terminals to point to it.

=== Password ===
Change the root password to the specified password in the usual place under the termcom user. If it's an office terminal, change the local user's password to the one specified in the usual place.

=== Prevent suspend and hibernation (Office Terminal) ===
Set <code>AllowSuspend</code>, <code>AllowHibernation</code>, <code>AllowSuspendThenHibernate</code> and <code>AllowHybridSleep</code> all to <code>no</code> in <code>/etc/systemd/sleep.conf</code>, and reboot.

== Records ==

You probably already created the host in the University IPAM system beforehand. If not, please do so.

Please also add the host to the [[Machine List]] here on the Wiki.

== Munin (System Monitoring) ==

If the new machine is not a container, you probably want to have it participate in the Munin cluster. Run <tt>apt-get install munin-node</tt> to install the monitoring client, then
edit the file /etc/munin/munin-node.conf. Look for a line that says <tt>allow ^127\.0\.0\.1$</tt> and add the following on a new line immediately below it:
<tt>allow ^129\.97\.134\.51$</tt> (this is the IP address for munin.csclub). Save the file, then <tt>/etc/init.d/munin-node restart</tt> and <tt>update-rc.d munin-node defaults</tt>.

Then, ssh into munin.csclub and edit the file /etc/munin/munin.conf and add the following lines to the end:
<tt>

[NEW-MACHINE-NAME.csclub] 
addr 129.97.134.### 
use_node_name yes</tt>

== Prometheus (System Monitoring) ==

We are currently using Prometheus to monitor our systems. On the new machine, install <tt>prometheus-node-exporter</tt> and <tt>stunnel</tt>.

Change <tt>/etc/default/prometheus-node-exporter</tt> to this:

ARGS="--web.listen-address=localhost:9101"

and start <tt>prometheus-node-exporter.service</tt>.

Then set up stunnel. Create <tt>/etc/stunnel/prometheus-node-exporter.conf</tt> with this content:

setuid = stunnel4
setgid = stunnel4
pid = /var/run/stunnel4/exporter.pid

debug = 7

[prometheus-node-exporter]
accept = 0.0.0.0:9100
connect = 127.0.0.1:9101
CAfile = /etc/stunnel/tls/server.crt
cert = /etc/stunnel/tls/node.crt
key = /etc/stunnel/tls/node.key
verifyPeer = yes

Copy <tt>/etc/stunnel/{node.crt, node.key, server.crt}</tt> from <tt>prometheus:/opt/prometheus/tls</tt> or the same location on other machines.

Finally, start <tt>stunnel4.service</tt>.

If it's a new machine, you'll also need to add it to the list of monitoring at <tt>prometheus:/opt/prometheus/prometheus.yml</tt>. Add it under a suitable label (or create a new label) in 'node_exporter' job.

= New Distribution =

If you're adding a new distribution, there a couple of steps you'll need to take in updating the CSClub Debian repository on [[Machine_List#sodium_benzoate|sodium-benzoate/mirror]].

The steps to add a new Debian release (in the examples, jessie) is as follows, modify as necessary:

=== Step 0: Create a GPG key ===

Use "gpg --gen-key" or something like that. Skip this if you already have one.

=== Step 1: Add to Uploaders ===

The /srv/debian/conf/uploaders file on mirror contains the list of people who can upload. Add your GPG key id to this file. Use "gpg --list-secret-keys" to find out the key ID. You also need to import your key into the mirror's gpg homedir as follows:

gpg --export $KEYID | sudo env GNUPGHOME=/srv/debian/gpg gpg --import

You only need to do this step once.

=== Step 2: Add Distro ===

Add a new section to /srv/debian/conf/distributions:

Origin: CSC
Label: Debian
Codename: '''jessie'''
Architectures: alpha amd64 i386 mips mipsel sparc powerpc armel source
Components: main contrib non-free
Uploaders: uploaders
Update: dell chrome
SignWith: yes
Log: '''jessie'''.log
--changes notifier

And update the '''Allow''' line in /srv/debian/conf/incoming:

Allow: '''jessie>jessie''' oldstable>squeeze stable>wheezy lucid>lucid maverick>maverick oneiric>oneiric precise>precise quantal>quantal

=== Step 3: Update from Sources ===

Run:

sudo env GNUPGHOME=/srv/debian/gpg /srv/debian/bin/rrr-update

If all went well you should see the new distribution listed at http://debian.csclub.uwaterloo.ca/dists/

=== Step 4: CSC Packages ===

Now that we've got our new distribution set up we need to generate our packages and have them uploaded. Namely, ceo and libpam-csc. For libpam-csc:

Get the package:

git clone https://git.csclub.uwaterloo.ca/public/libpam-csc.git
cd libpam-csc

Update change log:

EMAIL=[you]@csclub.uwaterloo.ca NAME="Your Name" dch -i

Update as necessary, i.e:

libpam-csc (1.10'''jessie0''') '''jessie'''; urgency=low

* Packaging for jessie.

-- Your Name <[you]@csclub.uwaterloo.ca> Thu, 10 Oct 2013 22:08:48 -0400

Build! (You may need to install various dependencies, which it will yell at you if you don't have.)

debuild -k'''YOURKEYID'''

Yay, it built now let's upload it to the repo. The build process which create a PACKAGE.changes file in the parent directory (replace PACKAGE with the actual package name).

Copy the dupload file from corn-syrup and dupload:

mv /etc/dupload /etc/dupload.bak
scp corn-syrup:/etc/dupload /etc/dupload
dupload libpam-csc_1.10jessie0_amd64.changes

Finally, log into mirror and type "sudo /srv/debian/bin/rrr-incoming". This is supposed to happen once every few minutes however it is always faster to run it manually.

And you're done. For CEO, see https://git.csclub.uwaterloo.ca/public/pyceo/src/branch/master/PACKAGING.md

NixOS

2026-09-02T20:58:00Z

K95ma: /* Setting up the system flake */ fix template variable names

We're trying to explore different options for running services, and '''NixOS on Proxmox containers''' is one of them. Here's how it is supposed to work:

* '''Malleable''': it should be relatively easy to modify an existing service config, update software version, and migrate one software to another, as everything are written in Nix configuration files.
* '''Recoverable''': NixOS keeps old copies of the system, which can be reverted if we see any immediate issues.
* '''Discoverable''': Services are located in one canonical, centralized location. This reduces the time needed to find the specific config for a specific software, and also makes it easy for someone to know what services are running.
* '''Replicable''': As NixOS service configuration files are written in a human readable format, anyone wishing to use the "normal" way to configure their service should be able to understand how to setup their service in a similar way.
* '''Trackable''': Easy to manage and track changes using Git, maybe even with CI.

=== Setting up a Proxmox VM ===

If there isn't a template already, use https://hydra.nixos.org/job/nixos/release-26.05/nixos.proxmoxLXC.x86_64-linux (replace 26.05 with the latest release)

Use "Create CT", add a SSH public key, and use the vmbr0 bridge. Start the container and look at its IP on the network tab. You should then be able to SSH into the container with a command like:

<syntaxhighlight lang="sh">
ssh -J [WatIAM]@neotame.csclub.uwaterloo.ca root@129.97.[ACT.UAL]
</syntaxhighlight>

==== Setting up the system flake ====

Because the new proxmoxLXC tarball appears to not have ''anything'' under {{code|/etc/nixos/}}, a system flake needs to be created from scratch. First run {{code|nix-channel --update}} then get into a shell with your favorite editor, e.g. {{code|nix-shell -p vim}}. Then create a new empty directory, enter it, and run:

<syntaxhighlight lang="sh">
nix flake init --extra-experimental-features nix-command --extra-experimental-features flakes
</syntaxhighlight>

In {{code|flake.nix}}:

<syntaxhighlight lang="nix">
{
inputs = {
nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-26.05";
};

outputs = { nixpkgs, ... } @inputs: {
nixosConfigurations.@@HOSTNAME@@ = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
specialArgs = { inherit inputs; };
modules = [ ./main.nix ];
};
};
}
</syntaxhighlight>

in {{code|main.nix}}:
<syntaxhighlight lang="nix">
{ modulesPath, pkgs, ... }:
{
imports = [
(modulesPath + "/virtualisation/proxmox-lxc.nix")
];
boot.isContainer = true;
# Supress systemd units that don't work because of LXC
systemd.suppressedSystemUnits = [
"dev-mqueue.mount"
"sys-kernel-debug.mount"
"sys-fs-fuse-connections.mount"
];
system.stateVersion = "26.05";
nix.settings.experimental-features = [ "nix-command" "flakes" ];
users.users."root".openssh.authorizedKeys.keys = [
# syscom public key
"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+l5+EvUkm/+I96y4mOrgmQA4h40UDHB59xCok8q23zNgzeRNZQFT9dx9vLZXSuYaZaecaBkC6IF+jHSvQRAbAhPnVPzlabbBOmIuZek6vWEjvr726UJhitW+HV3KNy1BMU6FMUxIboYbo1/zSpQTiyOzcQ+KWiFETxMKYl5bejlLdZUyl6OPYIAp5fwFRVneQHUkhJ/+QufzJz+HNVSCtrPGfNfoVsFiu6zzBDH0EwVxs9Ks2alyE0GT3jYGq8CkFB5ejdgt9FjlW1QG2F3eIsYVZOQ5vd/4KxqnAXcM2zkCyRlSaFNCRG3G3bOWBndc/gp9b+duRXfZyNguo3SVBZpQ9jDYxCdxyvKJVFKMOHDuVjIyE56J/vLB+SI5bRdb/zyPo5psqZVuSgt91WjpH1izGEkYYQjeOhyk57OOLPqIWyXvCoxCM1NkYy8Ld3JebRo0KwEFsNtaq0JUuCtgitfdM9qc3UQHVaKfU1PVKhYoLRtUYq18LY/7jvdRvAMM= root@xylitol"
];
users.users.@@USER@@ =
{
isNormalUser = true;
extraGroups = [ "wheel" ];
openssh.authorizedKeys.keys = [
"@@KEY1@@"
"@@KEY2@@"
];
shell = pkgs.fish;
};
programs.fish.enable = true;
programs.git.enable = true;
services.openssh = {
enable = true;
settings.PasswordAuthentication = false;
settings.KbdInteractiveAuthentication = false;
settings.PermitRootLogin = "without-password";
};
security.sudo.wheelNeedsPassword = false;
environment.systemPackages = with pkgs; [ vim fish ];
}
</syntaxhighlight>

Then run {{code|nixos-rebuild switch --flake .}} and the system flake should be setup. This can be used to create a non-root user that performs administration.

NixOS

2026-09-02T20:57:00Z

K95ma: add more instructions

We're trying to explore different options for running services, and '''NixOS on Proxmox containers''' is one of them. Here's how it is supposed to work:

* '''Malleable''': it should be relatively easy to modify an existing service config, update software version, and migrate one software to another, as everything are written in Nix configuration files.
* '''Recoverable''': NixOS keeps old copies of the system, which can be reverted if we see any immediate issues.
* '''Discoverable''': Services are located in one canonical, centralized location. This reduces the time needed to find the specific config for a specific software, and also makes it easy for someone to know what services are running.
* '''Replicable''': As NixOS service configuration files are written in a human readable format, anyone wishing to use the "normal" way to configure their service should be able to understand how to setup their service in a similar way.
* '''Trackable''': Easy to manage and track changes using Git, maybe even with CI.

=== Setting up a Proxmox VM ===

If there isn't a template already, use https://hydra.nixos.org/job/nixos/release-26.05/nixos.proxmoxLXC.x86_64-linux (replace 26.05 with the latest release)

Use "Create CT", add a SSH public key, and use the vmbr0 bridge. Start the container and look at its IP on the network tab. You should then be able to SSH into the container with a command like:

<syntaxhighlight lang="sh">
ssh -J [WatIAM]@neotame.csclub.uwaterloo.ca root@129.97.[ACT.UAL]
</syntaxhighlight>

==== Setting up the system flake ====

Because the new proxmoxLXC tarball appears to not have ''anything'' under {{code|/etc/nixos/}}, a system flake needs to be created from scratch. First run {{code|nix-channel --update}} then get into a shell with your favorite editor, e.g. {{code|nix-shell -p vim}}. Then create a new empty directory, enter it, and run:

<syntaxhighlight lang="sh">
nix flake init --extra-experimental-features nix-command --extra-experimental-features flakes
</syntaxhighlight>

In {{code|flake.nix}}:

<syntaxhighlight lang="nix">
{
inputs = {
nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-26.05";
};

outputs = { nixpkgs, ... } @inputs: {
nixosConfigurations.[hostname] = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
specialArgs = { inherit inputs; };
modules = [ ./main.nix ];
};
};
}
</syntaxhighlight>

in {{code|main.nix}}:
<syntaxhighlight lang="nix">
{ modulesPath, pkgs, ... }:
{
imports = [
(modulesPath + "/virtualisation/proxmox-lxc.nix")
];
boot.isContainer = true;
# Supress systemd units that don't work because of LXC
systemd.suppressedSystemUnits = [
"dev-mqueue.mount"
"sys-kernel-debug.mount"
"sys-fs-fuse-connections.mount"
];
system.stateVersion = "26.05";
nix.settings.experimental-features = [ "nix-command" "flakes" ];
users.users."root".openssh.authorizedKeys.keys = [
# syscom public key
"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+l5+EvUkm/+I96y4mOrgmQA4h40UDHB59xCok8q23zNgzeRNZQFT9dx9vLZXSuYaZaecaBkC6IF+jHSvQRAbAhPnVPzlabbBOmIuZek6vWEjvr726UJhitW+HV3KNy1BMU6FMUxIboYbo1/zSpQTiyOzcQ+KWiFETxMKYl5bejlLdZUyl6OPYIAp5fwFRVneQHUkhJ/+QufzJz+HNVSCtrPGfNfoVsFiu6zzBDH0EwVxs9Ks2alyE0GT3jYGq8CkFB5ejdgt9FjlW1QG2F3eIsYVZOQ5vd/4KxqnAXcM2zkCyRlSaFNCRG3G3bOWBndc/gp9b+duRXfZyNguo3SVBZpQ9jDYxCdxyvKJVFKMOHDuVjIyE56J/vLB+SI5bRdb/zyPo5psqZVuSgt91WjpH1izGEkYYQjeOhyk57OOLPqIWyXvCoxCM1NkYy8Ld3JebRo0KwEFsNtaq0JUuCtgitfdM9qc3UQHVaKfU1PVKhYoLRtUYq18LY/7jvdRvAMM= root@xylitol"
];
users.users.[user] =
{
isNormalUser = true;
extraGroups = [ "wheel" ];
openssh.authorizedKeys.keys = [
"[key1]"
"[key2]"
];
shell = pkgs.[shell];
};
programs.fish.enable = true;
programs.git.enable = true;
services.openssh = {
enable = true;
settings.PasswordAuthentication = false;
settings.KbdInteractiveAuthentication = false;
settings.PermitRootLogin = "without-password";
};
security.sudo.wheelNeedsPassword = false;
environment.systemPackages = with pkgs; [ vim fish ];
}
</syntaxhighlight>

Then run {{code|nixos-rebuild switch --flake .}} and the system flake should be setup. This can be used to create a non-root user that performs administration.

Template:Todo

2026-09-02T01:45:52Z

K95ma: include only+

[[[:Category:Todo|{{{text|todo}}}]]]<includeonly>[[Category:Todo]]</includeonly><noinclude>
We should really be using the template [https://www.mediawiki.org/wiki/Template:Fix Fix] from somewhere but im lazy {{todo}}
</noinclude>

Proxmox

2026-09-02T01:43:57Z

K95ma: /* SSH */ fix

The Proxmox Vitural Environment is a cluster hosted on sorbitol, tahini and teriyaki. The GUI can be accessed via the hostname of any machine on port 8006, e.g. [https://citric-acid.csclub.uwaterloo.ca:8006 https://tahini.csclub.uwaterloo.ca:8006].

== LDAP ==
To setup LDAP, from `Server View`, open the `Datacenter` page. Then go to `Permissions -> Realms`.

Then go to the LDAP page, add ldap realm, and input ldap1.csclub.uwaterloo.ca, and ldap2.csclub.uwaterloo.ca as the servers. Then make sure to sync both groups, and users.

Label it `csclub`, and make sure to log in using that realm when logging in from web ui.

== Joining the Cluster ==
To join the cluster, go to the existing CSC Cluster, from `Server View`, open the `Datacenter` page. Then go to `Cluster` and copy the join information.

On the new node, from `Server View`, open the `Datacenter` page. Then go to `Cluster`, then `Join Cluster` and paste the join information. You will need to enter the root password of the host where you copied the join information.

== SSH ==

When creating a new container/VM, ensure that syscom has access by adding the following SSH public key to {{code|/root/.ssh/authorized_keys}}:

<syntaxhighlight lang="sh">
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+l5+EvUkm/+I96y4mOrgmQA4h40UDHB59xCok8q23zNgzeRNZQFT9dx9vLZXSuYaZaecaBkC6IF+jHSvQRAbAhPnVPzlabbBOmIuZek6vWEjvr726UJhitW+HV3KNy1BMU6FMUxIboYbo1/zSpQTiyOzcQ+KWiFETxMKYl5bejlLdZUyl6OPYIAp5fwFRVneQHUkhJ/+QufzJz+HNVSCtrPGfNfoVsFiu6zzBDH0EwVxs9Ks2alyE0GT3jYGq8CkFB5ejdgt9FjlW1QG2F3eIsYVZOQ5vd/4KxqnAXcM2zkCyRlSaFNCRG3G3bOWBndc/gp9b+duRXfZyNguo3SVBZpQ9jDYxCdxyvKJVFKMOHDuVjIyE56J/vLB+SI5bRdb/zyPo5psqZVuSgt91WjpH1izGEkYYQjeOhyk57OOLPqIWyXvCoxCM1NkYy8Ld3JebRo0KwEFsNtaq0JUuCtgitfdM9qc3UQHVaKfU1PVKhYoLRtUYq18LY/7jvdRvAMM= root@xylitol
</syntaxhighlight>

== Virtual Machines ==
When making virtual machines in proxmox please first consider whether a full vm is needed, or an lxc will do.

Then go ahead, choose the node (reccomended to choose teriyaki, or tahini), and set a name.

When setting the name, if it is for a member please use `[watiam]-whatever-they-want`. If it's for something else, please name it clearly. Then when you get to os [storage is still TBD, check syscom irc, same with ISO image].

Add Qemu Agent, otherwise rest are fine with defaults.

Disks [TBD]

Network TBD

== Networking ==
There are two ways to do networking: network bridge and NAT. Network bridge will put the container/virtual machine on the CSC network (basically side-by-side to proxmox itself), while NAT will encapsulate the container/VM inside a private subnet that is only visible to proxmox host itself.

For services that only exposes HTTP/HTTPS, NAT is more desirable since multiple services can share a host nginx instance, only requiring the host IP to have 80/443 port opened to the Internet, thus saving some IP address in our pool and save some trips to the IST for firewall exemption. But for services that requires custom ports to be opened (for example, BigBlueButton requires a range of UDP ports to be exposed for relaying video streams), using the network bridge and giving the container/VM its own public IP might be easier.

Currently, <code>vmbr0</code> is used for bridged network and <code>vmbr1</code> is used for NAT (see [https://pve.proxmox.com/wiki/Network_Configuration#sysadmin_network_masquerading Proxmox's wiki on NAT networking] for setup instruction). <code>vmbr0</code> uses the CSC DHCP server, so you can use DHCP there, but <code>vmbr1</code> requires manual IP assignment.

Note that only using <code>vmbr1</code> requires you to use SSH ProxyJump via citric-acid to access the inner container, as it wouldn't have a public IP.

NixOS

2026-09-02T01:43:25Z

K95ma: fix

NixOS

2026-09-02T01:42:58Z

K95ma: +

Matrix

2026-09-02T00:54:42Z

K95ma: +

We are currently setting up a test server for Matrix. We use Synapse. If everything goes well, we will set up a production Matrix server.

== Server Setup ==

We are currently running Matrix on a [[Proxmox]] LXE container {{code|Proxmox}} on {{code|tahini}}.

== Matrix Installation ==

We are using NixOS, so the config to setup both Synapse and PostgreSQL is hosted at https://git.csclub.uwaterloo.ca/k95ma/matrix-nixos/.

The reverse proxy, Caddy is configured somewhere.{{todo}}

== Testing ==

Go to https://app.cinny.in/login/csclub.uwaterloo.ca, and click "Continue with CSC".

== Bridging ==

We are currently running mautrix-discord. Bridge to IRC via heisenbridge may be added in the future.

=== Discord ===

{{code|@beefbot:csclub.uwaterloo.ca}} is a Matrix account that is used to configure the actual mautrix-discord. It is logged into mautrix-discord via login-token bot (token saved, if it expires, login to CSC Exec and reset it). The login password of beefbot (the matrix account) is stored in the CSC vault on [[Vaultwarden]].

{{code|Matrix Bridge#7477}} is the Discord bot that reads the messages on Discord and sends them to Matrix via puppeting. It needs read access to the channels for the Discord -> Matrix part to work properly.

* Create the Matrix channel to bridge to
* Invite {{code|@discordbot:csclub.uwaterloo.ca}} to the channel
* Invite the Discord bot to the bridged server using {{code|1=https://discord.com/oauth2/authorize?client_id=...&permissions=309774583872&scope=bot}}. client_id is found by logging in to the CSC Exec account on Discord and going to developer portal to find the ID of the Matrix Bridge bot.
* Login to User {{code|beefbot}} (password in VaultWarden, idk if there is a better place to put it) on our matrix instance
* ''Inside'' the matrix channel, send, {{code|!discord bridge <channel ID>}} (the channel ID on Discord, the Discord bot must be able to see that channel)
* Create a Discord Webhook URL for that channel on Discord's side, and then DM the discord bot (via beefbot on matrix) with {{code|!discord set-relay !26DmcJd3cQ...:csclub.uwaterloo.ca --url https://discord.com/...}} with the first argument replaced by the actual room identifier and the second parameter the webhook URL.

Category:Todo

2026-09-02T00:54:29Z

K95ma: +

tracks pages with the template [[Template:todo]]

Template:Todo

2026-09-02T00:53:30Z

K95ma: +template

[[[:Category:Todo|{{{text|todo}}}]]]

Proxmox

2026-09-02T00:43:07Z

K95ma: + ssh key note

The Proxmox Vitural Environment is a cluster hosted on sorbitol, tahini and teriyaki. The GUI can be accessed via the hostname of any machine on port 8006, e.g. [https://citric-acid.csclub.uwaterloo.ca:8006 https://tahini.csclub.uwaterloo.ca:8006].

== LDAP ==
To setup LDAP, from `Server View`, open the `Datacenter` page. Then go to `Permissions -> Realms`.

Then go to the LDAP page, add ldap realm, and input ldap1.csclub.uwaterloo.ca, and ldap2.csclub.uwaterloo.ca as the servers. Then make sure to sync both groups, and users.

Label it `csclub`, and make sure to log in using that realm when logging in from web ui.

== Joining the Cluster ==
To join the cluster, go to the existing CSC Cluster, from `Server View`, open the `Datacenter` page. Then go to `Cluster` and copy the join information.

On the new node, from `Server View`, open the `Datacenter` page. Then go to `Cluster`, then `Join Cluster` and paste the join information. You will need to enter the root password of the host where you copied the join information.

== SSH ==

When creating a new container/VM, ensure that syscom has access by adding the following SSH public key to {{code|/root/.ssh/authorized_keys}}:

<syntaxhighlight>
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+l5+EvUkm/+I96y4mOrgmQA4h40UDHB59xCok8q23zNgzeRNZQFT9dx9vLZXSuYaZaecaBkC6IF+jHSvQRAbAhPnVPzlabbBOmIuZek6vWEjvr726UJhitW+HV3KNy1BMU6FMUxIboYbo1/zSpQTiyOzcQ+KWiFETxMKYl5bejlLdZUyl6OPYIAp5fwFRVneQHUkhJ/+QufzJz+HNVSCtrPGfNfoVsFiu6zzBDH0EwVxs9Ks2alyE0GT3jYGq8CkFB5ejdgt9FjlW1QG2F3eIsYVZOQ5vd/4KxqnAXcM2zkCyRlSaFNCRG3G3bOWBndc/gp9b+duRXfZyNguo3SVBZpQ9jDYxCdxyvKJVFKMOHDuVjIyE56J/vLB+SI5bRdb/zyPo5psqZVuSgt91WjpH1izGEkYYQjeOhyk57OOLPqIWyXvCoxCM1NkYy8Ld3JebRo0KwEFsNtaq0JUuCtgitfdM9qc3UQHVaKfU1PVKhYoLRtUYq18LY/7jvdRvAMM= root@xylitol
</syntaxhighlight>

== Virtual Machines ==
When making virtual machines in proxmox please first consider whether a full vm is needed, or an lxc will do.

Then go ahead, choose the node (reccomended to choose teriyaki, or tahini), and set a name.

When setting the name, if it is for a member please use `[watiam]-whatever-they-want`. If it's for something else, please name it clearly. Then when you get to os [storage is still TBD, check syscom irc, same with ISO image].

Add Qemu Agent, otherwise rest are fine with defaults.

Disks [TBD]

Network TBD

== Networking ==
There are two ways to do networking: network bridge and NAT. Network bridge will put the container/virtual machine on the CSC network (basically side-by-side to proxmox itself), while NAT will encapsulate the container/VM inside a private subnet that is only visible to proxmox host itself.

For services that only exposes HTTP/HTTPS, NAT is more desirable since multiple services can share a host nginx instance, only requiring the host IP to have 80/443 port opened to the Internet, thus saving some IP address in our pool and save some trips to the IST for firewall exemption. But for services that requires custom ports to be opened (for example, BigBlueButton requires a range of UDP ports to be exposed for relaying video streams), using the network bridge and giving the container/VM its own public IP might be easier.

Currently, <code>vmbr0</code> is used for bridged network and <code>vmbr1</code> is used for NAT (see [https://pve.proxmox.com/wiki/Network_Configuration#sysadmin_network_masquerading Proxmox's wiki on NAT networking] for setup instruction). <code>vmbr0</code> uses the CSC DHCP server, so you can use DHCP there, but <code>vmbr1</code> requires manual IP assignment.

Note that only using <code>vmbr1</code> requires you to use SSH ProxyJump via citric-acid to access the inner container, as it wouldn't have a public IP.

MySQL

2026-09-01T23:02:06Z

K95ma: K95ma moved page MySQL to MariaDB: use the actual thing we use

#REDIRECT [[MariaDB]]

MariaDB

2026-09-01T23:02:06Z

K95ma: K95ma moved page MySQL to MariaDB: use the actual thing we use

== For members ==
<Currently both MariaDB and Postgres shows a 503 Service Unavailable>

Note: the database on caffeine is actually MariaDB, not MySQL. Although they are mostly compatible, there are some incompatibilities to be aware of. See [https://mariadb.com/kb/en/mariadb-vs-mysql-compatibility/ MariaDB versus MySQL: Compatibility] for details.

=== Creating databases ===

Users can create their own MySQL databases through [[ceo]]. Users emailing syscom asking for a MySQL database should be directed to do so. The process is as follows:

# SSH into any [[Machine_List|CSC machine]].
# Run <tt>ceo</tt>.
# Select "Create MySQL database" and follow the instructions.
# Login info will be stored in <tt>ceo-mysql-info</tt> in your home directory.
# You can now connect to the MySQL database (from [[Machine_List#caffeine|caffeine]] only).

=== Deleting databases ===

Users can delete their own MySQL databases.

SSH into [[Machine_List#caffeine|caffeine]].
mysql -u yourusernamehere -p
Enter password: ******
DROP DATABASE database name goes here
Login info and database name was created on database creation in <tt>ceo-mysql-info</tt> in your home directory.

== For syscom ==

=== Creating a database manually ===
To create a MySQL database manually on caffeine, first connect to the database as root:

$ mysql -uroot -p
Enter password: ******

Then run the following SQL statements:

CREATE USER 'someuser'@'localhost' IDENTIFIED VIA unix_socket;
CREATE USER 'someuser'@'%' IDENTIFIED BY 'longrandompassword';
CREATE DATABASE someuser;
GRANT ALL PRIVILEGES ON someusername.* to 'someuser'@'localhost' IDENTIFIED VIA unix_socket;
GRANT ALL PRIVILEGES ON someusername.* to 'someuser'@'%';

This will allow users to connect locally without a password, and connect remotely with a password.

For random passwords run <code>pwgen -s 20 1</code>. For the administrative passwords see /users/sysadmin/passwords/mysql.

Write a file (usually ~club/mysql) to the club's homedir readable only by them containing the following:

Username: clubuserid
Password: longrandompassword
Hostname: localhost

Try not to send passwords via plaintext email.

=== Replication ===

See the history of this page for information on the previous replication setup.

=== Backups ===

We use [https://mariadb.com/kb/en/mariabackup-overview/ mariabackup] to take periodic backups. It is currently installed and configured on both caffeine and coffee.

==== Installation ====
In the example below, we will be installing mariabackup on coffee, and sending the backups to corn-syrup.

First, install the mariadb-backup package:
<pre>
apt install mariadb-backup
</pre>

Next, create an SSH key pair for the mysql user:
<pre>
mkdir /var/mariadb
chown mysql:mysql /var/mariadb
su -s /bin/bash mysql
cd /var/mariadb
mkdir .ssh
chmod 700 .ssh
# Choose /var/mariadb/.ssh/id_ed25519 for the path
ssh-keygen -t ed25519
</pre>

Paste the public key (/var/mariadb/.ssh/id_ed25519.pub) into /users/syscom/.ssh/authorized_keys on corn-syrup:
<pre>
restrict ssh-ed25519 AAAAC3Nza... mysql@coffee
</pre>
Also create the folder <code>/users/syscom/backups/coffee/mariabackup</code>. We will store the backups here.

We will use a hacky bash script to try to emulate the same behaviour as pgBackRest. We will compress and stream each backup to a folder on corn-syrup in the format <code>1701678356-F</code>, where the number is a Unix epoch timestamp and the letter at the end is one of F, D or I (for full, differential or incremental backups). Full backups do not depend on any other backups. Differential backups depend on the latest full backup before them. Incremental backups depend on the latest backup before them (of any type).

On coffee, paste the following into e.g. /var/mariadb/bin/backup-mariadb.sh:
<pre>
#!/bin/bash

RETENTION_FULL=2
RETENTION_DIFF=4
SSH_KEY=/var/mariadb/.ssh/id_ed25519
SSH_USER=syscom
SSH_HOST=corn-syrup
SSH_FOLDER=/users/$SSH_USER/backups/$(hostname)/mariabackup
SSH_ARGS="-i $SSH_KEY -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
SSH="ssh $SSH_ARGS $SSH_USER@$SSH_HOST"

set -euxo pipefail
# $USER doesn't seem to be defined when we run this from cron
if [ "$(id -un)" != mysql ]; then
echo "This script should run as the mysql user" >&2
exit 1
fi
if [ $# -ne 1 ]; then
echo "Usage: $0 <full|diff|incr>" >&2
exit 1
fi
backup_type=$1
if [ "$backup_type" = full ]; then
backup_type_letter=F
elif [ "$backup_type" = diff ]; then
backup_type_letter=D
elif [ "$backup_type" = incr ]; then
backup_type_letter=I
else
echo "Backup type must be one of 'full', 'diff' or 'incr'" >&2
exit 1
fi
if ! pgrep mariadbd >/dev/null; then
echo "MariaDB is not running" >&2
exit 1
fi
if pgrep mariabackup >/dev/null; then
echo "mariabackup is already running" >&2
exit 1
fi
# Delete temporary files left behind by previous run, if there are any
$SSH -- "rm -rf $SSH_FOLDER/*.tmp"
# Get a list of all backups in chronological order
mapfile -t backups < <($SSH -- "/bin/ls -1 $SSH_FOLDER | grep -P '^\\d+-[FDI]$' | sort")
incremental_basedir_args=
old_checkpoint_dir=$(mktemp -d)
new_checkpoint_dir=$(mktemp -d)
trap "rm -rf $old_checkpoint_dir $new_checkpoint_dir" EXIT
if [ "$backup_type" = diff -o "$backup_type" = incr ]; then
# Find a backup which we can use as a base.
# For incr, this can be any type; for diff, this must be a full backup.
base_backup=
for ((i=${#backups[@]}-1; i>=0; i--)); do
backup=${backups[i]}
if [ $backup_type = incr ] || [[ $backup =~ -F$ ]]; then
base_backup=$backup
break
fi
done
if [ -z "$base_backup" ]; then
echo "Could not find base backup for $backup_type type" >&2
exit 1
fi
# Copy the xtrabackup_checkpoints file from the base backup into a
# temporary directory, and use it in the mariabackup command.
scp $SSH_ARGS "$SSH_USER@$SSH_HOST:$SSH_FOLDER/$base_backup/xtrabackup_*" $old_checkpoint_dir/
incremental_basedir_args="--incremental-basedir=$old_checkpoint_dir"
fi
compress_level=6
if [ $backup_type = full ]; then
# Use a lower compression level to go faster
compress_level=5
fi
foldername="$(date +%s)-$backup_type_letter"
# First copy to a temporary dir, then rename the temporary dir to the
# desired dir name (in case our process gets killed)
mariabackup --user=mysql --backup $incremental_basedir_args --stream=xbstream --extra-lsndir=$new_checkpoint_dir \
| nice zstd -$compress_level -T4 \
| $SSH -- "cd $SSH_FOLDER && mkdir $foldername.tmp && cat > $foldername.tmp/data.xb.zst"
scp $SSH_ARGS $new_checkpoint_dir/* $SSH_USER@$SSH_HOST:$SSH_FOLDER/$foldername.tmp/
$SSH -- "mv $SSH_FOLDER/$foldername.tmp $SSH_FOLDER/$foldername"

# Delete old backups
if [ $backup_type = incr ]; then
# We don't delete backups when making an incr backup, since we only
# have retention limits for full and diff
exit
fi
if [ $backup_type = full ]; then
retention=$RETENTION_FULL
else
retention=$RETENTION_DIFF
fi
num_backups_of_same_type=1
backups_to_delete=()
for ((i=${#backups[@]}-1; i>=0; i--)); do
backup=${backups[i]}
if ! [[ $backup =~ -${backup_type_letter}$ ]]; then
continue
fi
((num_backups_of_same_type++))
if [ $num_backups_of_same_type -lt $retention ]; then
continue
fi
if [ $backup_type = full ]; then
# Delete everything before the last full backup which we want to
# keep
pat='^'
else
# Delete all the diff and incr backups before the last diff backup
# which we want to keep
pat='-[DI]$'
fi
for ((j=$i-1; j>=0; j--)); do
backup=${backups[j]}
if [[ $backup =~ $pat ]]; then
backups_to_delete+=($backup)
fi
done
break
done
if [ ${#backups_to_delete[@]} -eq 0 ]; then
echo "No backups to delete" >&2
exit
fi
$SSH -- "cd $SSH_FOLDER && rm -r ${backups_to_delete[@]}"
</pre>
The script should be invoked with exactly one argument which must be one of "full", "diff" or "incr".

==== Cron ====
We are going to use systemd timers because they are much nicer to use than cron. Install /usr/local/bin/csc-systemd-email and /etc/systemd/system/csc-email-on-failure@.service on the target machine so that we get emails for failed jobs (there should be a copy of this on caffeine).

Paste the following into /etc/systemd/system/mariadb-backup@.service:
<pre>
[Unit]
Description=MariaDB backup (%i)
Documentation=https://wiki.csclub.uwaterloo.ca/MySQL#Backups

[Service]
Type=oneshot
User=mysql
ExecStart=/var/mariadb/bin/backup-mariadb.sh %i

[Unit]
OnFailure=csc-email-on-failure@%n.service
</pre>

Paste the following into /etc/systemd/system/mariadb-backup-full.timer:
<pre>
[Unit]
Description=MariaDB backup (full)

[Timer]
# Full back up at 00:20 every Sunday and Wednesday
OnCalendar=Sun,Wed *-*-* 00:20:00
Unit=mariadb-backup@full.service
Persistent=true

[Install]
WantedBy=timers.target
</pre>

Paste the following into /etc/systemd/system/mariadb-backup-diff.timer:
<pre>
[Unit]
Description=MariaDB backup (diff)

[Timer]
# Differential backup at 00:35 every day
OnCalendar=*-*-* 00:35:00
Unit=mariadb-backup@diff.service
Persistent=true

[Install]
WantedBy=timers.target
</pre>

Paste the following into /etc/systemd/system/mariadb-backup-incr.timer:
<pre>
[Unit]
Description=MariaDB backup (incr)

[Timer]
# Incremental backup at the 50th minute of every hour
OnCalendar=*-*-* *:50:00
Unit=mariadb-backup@incr.service
Persistent=true

[Install]
WantedBy=timers.target
</pre>

Finally, enable and start the timers:
<pre>
systemctl daemon-reload
systemctl enable --now mariadb-backup-full.timer
systemctl enable --now mariadb-backup-diff.timer
systemctl enable --now mariadb-backup-incr.timer
</pre>

==== Restore ====
Paste the following into e.g. /var/mariadb/bin/restore-mariadb.sh:
<pre>
#!/bin/bash

SSH_KEY=/var/mariadb/.ssh/id_ed25519
SSH_USER=syscom
SSH_HOST=corn-syrup
SSH_FOLDER=/users/$SSH_USER/backups/$(hostname)/mariabackup
SSH_ARGS="-i $SSH_KEY -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
SSH="ssh $SSH_ARGS $SSH_USER@$SSH_HOST"

set -euxo pipefail
shopt -s dotglob
if [ "$(id -un)" != mysql ]; then
echo "This script should run as the mysql user" >&2
exit 1
fi
if [ $# -gt 1 ]; then
echo "Usage: $0 [0123456789-I]" >&2
exit 1
fi
if pgrep mariadbd >/dev/null; then
echo "Please stop MariaDB first" >&2
exit 1
fi
# Get a list of all backups in chronological order
mapfile -t backups < <($SSH -- "/bin/ls -1 $SSH_FOLDER | grep -P '^\\d+-[FDI]$' | sort")
if [ ${#backups[@]} -eq 0 ]; then
echo "No backups found" >&2
exit 1
fi
if [ $# -eq 1 ]; then
last_backup_idx=
for ((i=${#backups[@]}-1; i>=0; i--)); do
if [ ${backups[i]} = "$1" ]; then
last_backup_idx=$i
break
fi
done
if [ -z "$last_backup_idx" ]; then
echo "Could not find $1 on remote" >&2
exit 1
fi
else
last_backup_idx=$(( ${#backups[@]} - 1 ))
fi
last_full_backup_idx=
for ((i=$last_backup_idx; i>=0; i--)); do
if [[ ${backups[i]} =~ -F$ ]]; then
last_full_backup_idx=$i
break
fi
done
if [ -z "$last_full_backup_idx" ]; then
echo "Could not find full backup for ${backups[last_backup_idx]}" >&2
exit 1
fi
backups_to_use=()
if [[ ${backups[last_backup_idx]} =~ -F$ ]]; then
# If we're restoring a full backup, we only need that one backup
backups_to_use=(${backups[last_backup_idx]})
elif [[ ${backups[last_backup_idx]} =~ -D$ ]]; then
# If we're restoring a diff backup, we only need that one backup and the
# first full backup before it
backups_to_use=(${backups[last_full_backup_idx]} ${backups[last_backup_idx]})
else
# If we're restoring an incr backup, we need all the backups from it to
# the first diff backup before it, and the first full backup before that.
# If there is no diff backup between it and the last full backup, then
# we need everything between it and the last full backup.
for ((i=$last_backup_idx; i>=$last_full_backup_idx; i--)); do
backups_to_use=(${backups[i]} ${backups_to_use[@]})
if [[ ${backups[i]} =~ -D$ ]]; then
backups_to_use=(${backups[last_full_backup_idx]} ${backups_to_use[@]})
break
fi
done
fi
base_dir=$(mktemp -d)
incr_dir=$(mktemp -d)
trap "rm -rf $base_dir $incr_dir" EXIT
for backup in ${backups_to_use[@]}; do
if [[ $backup =~ -F$ ]]; then
backup_dir=$base_dir
else
backup_dir=$incr_dir
fi
$SSH -- "cat $SSH_FOLDER/$backup/data.xb.zst" | zstd -d | mbstream -x -C $backup_dir
incremental_dir_args=
if [ $backup_dir = $incr_dir ]; then
incremental_dir_args="--incremental-dir=$incr_dir"
fi
mariabackup --prepare --target-dir=$base_dir $incremental_dir_args
if [ $backup_dir = $incr_dir ]; then
rm -rf $incr_dir/*
fi
done
if [ "$(/bin/ls -1 /var/lib/mysql | wc -l)" -gt 0 ]; then
read -p "Everything under /var/lib/mysql will be deleted. Continue (y/n)? " yn
yn=${yn,,} # convert to lower case
if [ "$yn" = y -o "$yn" = yes ]; then
rm -rf /var/lib/mysql/*
else
echo "Aborting." >&2
exit 1
fi
fi
mariabackup --move-back --target-dir=$base_dir
echo "Restoration succeeded, please restart MariaDB"
</pre>
Make sure to stop MariaDB before restoring a backup. If this script is invoked without any arguments, the latest backup found on corn-syrup will be used; a single argument may also be specified, which must be the name of one of the backup folders stored on corn-syrup.

[[Category:Software]]

Ceo

2026-08-31T01:41:50Z

K95ma: /* Making Changes */ fix link

[[Image:Pyceo.png|thumb|300px|right|<tt>pyceo</tt>'s main menu screen]]

ceo is the CSC member creation and administration interface. It was originally written in perl by persons of mysterious-ness, was re-written in python by Michael Spang in early 2007, and re-written again (in Python) by Syscom in 2020-2021. The source-code for ceo can be found in git: [https://git.csclub.uwaterloo.ca/public/pyceo https://git.csclub.uwaterloo.ca/public/pyceo].

= Instructions/Usage =
ceo can be accessed by running the "ceo" command from a terminal, or terminal emulator.
By default, a curses-based menu interface is presented. Use the arrow keys to navigate;
on many screens, pressing a letter will select the next menu item beginning with that letter.

=== Command-line Mode ===
Run <tt>ceo --help</tt> to see a list of command-line utilities.

== Adding a New Member ==
After a new member has paid the membership fee and signed the Machine Usage Policy forms, a new member account is added to the CSC system by selecting "New Member" in ceo and following the on-screen instructions. The new member's username is to be identical to their WatIAM username, if applicable. For WatIAM users, the name and program fields will automatically be filled after a username is provided.

== Renewing/Extending a Membership ==
A membership can be renewed or extended by selecting "Renew Membership" in the ceo interface.

== Hosted Clubs ==
Clubs are hosted free of charge. To create a new club account use the "New Club" option in the ceo interface.

=== Club Representatives ===
At this time, there is no limit to the number of representatives a club may have, but representative accounts must be registered with the "New Club Rep" option, and renewed with the "Renew Club Rep" option.

=== Other Club Features ===
For access to features beyond basic hosting (ie, databases), one of the club representatives will need to email the Systems Committee to have this set up.

= raymo's guide on how to fix things after screwing up =

== Changing a member to a nonmember (club rep) and vice-versa ==

ssh hfcs
kinit # if you don't already have [[Kerberos#raymo's guide to keytabs|keytabs]] set up
ldapvi -Y GSSAPI
Use <code>/<username></code> to search for the user in vi and change <code>term</code> to <code>nonMemberTerm</code> (or vice-versa) for the relevant terms. When you're done deleting the file should no longer contain the username. Save and quit (<code>:wq</code>) and press <code>y</code> when prompted.

== Deleting a member ==

* '''RULE: Never do this without good reason. We should NEVER delete accounts or groups that have been used before.'''

If you accidentally created a club rep as a regular member instead, see the [previous section|Ceo#Changing a member to a nonmember (club rep) and vice-versa]. For another reason that doesn't break the '''RULE''' above, first follow the steps in the change membership section above, up to and including <code>ldapvi</code>, then delete both the user and group LDAP records. These are separated by blank lines. When you're done deleting the file should no longer contain the username. Save and quit as if changing membership. Then:
ssh auth1
sudo kadmin.local
delprinc <username>
ssh phosphoric acid
sudo rm -rfI /users/<username>
Unsubscribe the user from [https://mailman.csclub.uwaterloo.ca/postorius/lists/syscom.csclub.uwaterloo.ca/members/member/ csc-general on mailman]

= Feature Requests and Ideas =

* Create a graphical and/or online version of ceo
* Add new members to fuse and plugdev groups

= Contributing to CEO =

== Preliminary Steps ==
=== Generate a GPG Key ===
In order to sign the ceo packages you will need to generate yourself a GPG key if you do not already have one. Assuming you do not run

gpg --gen-key

Choose option (2) DSA (sign only). Choose no expiration when prompted and then your full name and email when asked. It will ask you to confirm the information and then for a passphrase.

=== Add Your Key To Mirror ===
ssh mirror.csclub.uwaterloo.ca
gpg --list-keys

Locate the 8-character id string. For example "16E37635" in
/users/m2ellis/.gnupg/pubring.gpg
---------------------------------
pub 1024D/'''16E37635''' 2010-08-19
uid Michael Ellis <m2ellis@csclub.uwaterloo.ca>

Now you must add this id into the file /srv/debian/conf/uploaders on mirror
sudo vim /srv/debian/conf/uploaders

Now in another terminal run
gpg --export --armor $KEYID

Now on mirror run
sudo -s
GNUPGHOME=/srv/debian/gpg gpg --import

Then paste the output from gpg --export --armor $KEYID and end with CTRL-D. It should give you a confirmation, example
gpg: key 16E37635: public key "Michael Ellis <m2ellis@csclub.uwaterloo.ca>" imported
gpg: Total number processed: 1
gpg: imported: 1

== Making Changes ==
The source-code for ceo can be found in git: [https://git.csclub.uwaterloo.ca/public/pyceo public/pyceo]. To checkout the code run

git clone ~git/public/pyceo.git

When you are done making your change you need to update the changelog with dch. Assuming this is a minor incremental change run

dch -i

Add a description of your change and then save and quit. Once you are sure of your changes commit them to the git repository and push them (test them first!).

Make sure to set a distribution, like distribution UNRELEASED is NOT allowed. So change it to whatever distribution you're deploying to in the <code>debian/changelog</code> file

Then you'll need to make a tar.gz file<syntaxhighlight lang="bash">
tar czf ceo_<major>.<minor>.<patch>.orig.tar.gz pyceo/
</syntaxhighlight>To build the package run debuild

debuild

This will generate the *.deb files in the parent directory.

This may try to auto-sign with whatever your first GPG key is, you can safely exist and debsign yourself.

=== Uploading Changes to Mirror ===
After you make the package, you'll need to sign it, this can be via debsign. (You can find the .changes file in the parent directory)<syntaxhighlight lang="bash">
debsign -k [GPG Key ID] [package].changes

</syntaxhighlight>

In the directory containing the *.deb and *.changes files run<syntaxhighlight lang="bash">
dput -c /path/to/dput.cf ceo_<major>.<minor>.<patch>-<distribution>1_amd64.changes
</syntaxhighlight>Here is a sample dput.cf<syntaxhighlight lang="ini">
[DEFAULT]
default_host_main = debian.csclub

[debian.csclub]
fqdn = potassium-benzoate.csclub.uwaterloo.ca
method = scp
incoming = /srv/debian/incoming
run_install = 1
pre_upload_command = /bin/true
login = o32patel
</syntaxhighlight>Then ssh to mirror and run
reprepro --confdir /srv/debian/conf/ processincoming basic # use --ignore=longkeyid if you uploaded a long GPG id

The package should now be uploaded and you can update in the usual way with apt-get/aptitude.

== How to deploy CEO ==
Firstly, cry

Secondly, check the usual place for all the passwords, install ceo. Install ceod if needed, and edit the configs to make sure the correct servers are contacted for the relevant services. Like install ceod on the management node of cloudstack.

Then setup the kerberos tickets

Then go to `/etc/csc`, and fill in all the ceo.ini, and ceod.ini configs
[[Category:Software]]

User:K95ma

2026-08-13T03:32:43Z

K95ma:

Termcom person

Vaultwarden

2026-08-13T03:26:46Z

K95ma: +

Can be found at; https://pass.csclub.uwaterloo.ca/

You have to manually make an account by pressing 'Create Account'

It's doesn't integrate with OAuth, or LDAP so we can't auto-create the accounts (note: is this true? per https://bitwarden.com/help/directory-sync/ and https://github.com/dani-garcia/vaultwarden#features we should be able to set something up)

It runs on tahini, as a docker-compose container in a [[Proxmox]] VM. To update its version, get access to it, then edit the docker-compose.yaml to the [https://hub.docker.com/r/vaultwarden/server/tags latest stable version] and pull and up:

<syntaxhighlight lang="bash">
cd /root/docker/vault
vim docker-compose.yaml
docker compose pull
docker compose up -d
</syntaxhighlight>

Matrix

2026-08-12T23:38:33Z

K95ma: /* Discord */ fix

Matrix

2026-08-12T23:37:53Z

K95ma: /* Discord */ link

Matrix

2026-08-12T23:37:36Z

K95ma: /* Bridging */ expand

Matrix

2026-08-12T23:21:22Z

K95ma: /* Bridging */ +

Firewall

2026-08-07T03:14:07Z

K95ma: /* General Use */ fix wording

Our networks are behind the University's Campus firewall. This means that traffic to us is automatically dropped at the edge of campus unless we have exceptions added for it.

= Current Exceptions =

== Office Terminals ==

No exceptions in the campus firewall.

== General Use ==

These ports are allowed:

* Port 22 (SSH), plus additional SSH ports on taurine (21, 53, 80, 81, 443, 8000, 8080)
* Ports 60000–61000 (Mosh)
* Ports 28000-28500 (TCP/UDP general use)

== Webserver ==

* caffeine has ports 22 (SSH), 80 (HTTP), 443 (HTTPS), 11068 (HTTP for rridge) and UDP 60000–61000 (Mosh)
* wiki is now a CNAME for caffeine
* git has ports 80 (HTTP) and 443 (HTTPS)
* nextcloud has ports 80 (HTTP) and 443 (HTTPS)

== Mail ==

* mail has ports 25 (SMTP), 80 (HTTP), 143 (IMAP), 443 (HTTPS), 587 (MAIL SUBMISSION), 993 (IMAPS)
* mailman has ports 80 (HTTP) and 443 (HTTPS)

== Mirror ==

* mirror has ports 21 (FTP), 22 (SSH), 80 (HTTP), 443 (HTTPS), 873 (RSYNC)

== IPv6 Test ==

* ds.test-ipv6, mtu1280.test-ipv6 have ports 80 (HTTP), 443 (HTTPS)
* v6ns1.test-ipv6 has port 53 (DNS)

== Cloud ==
* cloud.csclub.uwaterloo.ca (and csclub.cloud, and all of their subdomains): 80 (HTTP) and 443 (HTTPS)
* riboflavin (and possibly other cloud machines) have port 22 exempted from the campus firewall, but they are currently blocked via iptables on the hosts

== BigBlueButton ==

* bbb has TCP ports 80 (HTTP) and 443 (HTTPS)
* bbb has UDP ports 16384 - 32768 (for WebRTC)

== drone.io (ci) ==

* This has now moved to Kubernetes (same IP address as cloud.csclub.uwaterloo.ca).

== progcom ==
* progcom has TCP ports 80 (HTTP) and 443 (HTTPS)

== Other Web Services ==

rt, munin, prometheus

* 80 (HTTP) and 443 (HTTPS)

= Adding Exceptions =

Create a ticket on the [https://uwaterloo.atlassian.net/servicedesk/customer/portal/2/group/413/create/805 UWaterloo Help Portal]. Use the syscom account unless you already have an account and IST knows it.

Machine List

2026-08-07T03:13:19Z

K95ma: /* General-Use Servers */ +

Most of our machines are in the E7, F7, G7 and H7 racks (as of Jan. 2022) in the MC 3015 server room. There is an additional rack in the DC 3558 machine room on the third floor. Our office terminals are in the CSC office, in MC 3036/3037.

= Web Server =
You are highly encouraged to avoid running anything that's not directly related to your CSC webspace on our web server. We have plenty of general-use machines; please use those instead. You can even edit web pages from any other machine--usually the only reason you'd *need* to be on caffeine is for database access.

== ''caffeine'' ==

Caffeine is the Computer Science Club's web server. It serves websites, databases for websites, and a large amount of other services.

''(Redundant active backup coming soon...)''

==== Specs ====

* QEMU virtual machine hosted on [[Machine List#Teriyaki|Teriyaki]]
** 32 vCPUs
** 64GB of RAM

==== Services ====

* Club and member web sites with [https://www.apache.org/ Apache]
* [[MySQL]] databases
* [[PostgreSQL]] databases
* [[ceo]] daemon

==''mathnews''==

[[#xylitol|xylitol]] hosts a systemd-nspawn container which serves as the mathNEWS webserver. It is administered by mathNEWS, as a pilot for providing containers to select groups who have more specialized demands than the general-use infrastructure can meet.

= General-Use Servers =

These machines can be used for (nearly) anything you like (though be polite and remember that these are shared machines). Recall that when you signed the Machine Usage Agreement, you promised not to use these machines to generate profit (so no cryptocurrency mining).

For computationally-intensive jobs (CPU/memory bound) we recommend running on high-fructose-corn-syrup, carbonated-water, sorbitol, mannitol, or corn-syrup, listed in roughly decreasing order of available resources. For low-intensity interactive jobs, such as IRC clients, we recommend running on neotame. '''If you have a long-running computationally intensive job, it's good to nice[https://en.wikipedia.org/wiki/Nice_(Unix)] your process, and possibly let syscom know too.'''

'''Note: [[Firewall#General Use|firewall]] applies to these servers'''

== ''corn-syrup'' ==

Dell PowerEdge 2950

==== Specs ====

* 2 × Intel Xeon E5405 (2.00 GHz, 4 cores each)
* 32 GB RAM
* eth0 ("Gb0") mac addr 00:24:e8:52:41:27
* eth1 ("Gb1") mac addr 00:24:e8:52:41:29
* IPMI mac addr 00:24:e8:52:41:2b
* 3 × Western-Digital 160GB SATA hard drive (445 GB software RAID0 array)

==== Notes ====

* Use eth0/Gb0 for the mathstudentorgsnet connection

==== Services ====

* Hosts 1 TB <tt>[[scratch|/scratch]]</tt> and exports via NFS (sec=krb5)

== ''high-fructose-corn-syrup'' ==

High-fructose-corn-syrup (or hfcs) is a large SuperMicro server. It's been in CSC service since April 2012.

==== Specs ====

* 4x AMD Opteron 6272 (2.4 GHz, 16 cores each)
* 192 GB RAM
* Supermicro H8QGi+-F Motherboard Quad 1944-pin Socket [http://csclub.uwaterloo.ca/misc/manuals/motherboard-H8QGI+-F.pdf (Manual)]
* 500 GB Seagate Barracuda
* Supermicro Case Rackmount CSE-748TQ-R1400B 4U [http://csclub.uwaterloo.ca/misc/manuals/SC748.pdf (Manual)]
'''Notes'''

* Missing moba IO shield (as of January 2024)

== ''carbonated-water'' ==

carbonated-water is a Dell R815 provided by CSCF.

==== Specs ====

* 4x AMD Opteron 6176 processors (2.3 GHz, 12 cores each)
* 128GB RAM

== ''neotame'' ==

neotame is a SuperMicro server funded by MEF. It is the successor to taurine.

'''We strongly discourage running computationally-intensive jobs''' on neotame as many users run interactive applications such as IRC clients on it and any significant service degradation will be more likely to affect other users (who will probably notice right away).

==== Specs ====

* 2x Intel Xeon E5-2630 v4 processors (2.2 GHz, 10 cores/20 threads each)
* 64GB RAM
'''Notes'''
* SSH server also listens on ports 21, 22, 53, 80, 81, 443, 8000, 8080 for your convenience.
== ''mannitol'' ==

mannitol is a SuperMicro server funded by MEF. CUDA is available on this node.

==== Specs ====

* 2x Intel Xeon E5-2630 v4 processors (2.2 GHz, 10 cores/20 threads each)
* 64GB RAM
* NVIDIA GeForce RTX 3050 6G

= Office Terminals =

It's possible to SSH into these machines, but we discourage you from trying to use these machines when you're not sitting in front of them. They are bounced at least every time our login manager, lightdm, throws a tantrum (which is several times a day). These are for use inside our physical office.

== ''cyanide'' ==
cyanide is a [https://support.apple.com/kb/sp710 Mac Mini (Late 2014)], identical in specification to powernap

=== Spec ===

* Intel i7-4578U (4) @ 3.500GHz
* 16GB RAM
* Intel Iris Graphics 5100
* 256GB On-board SSD

==''powernap''==
powernap is a [https://support.apple.com/kb/sp710 Mac Mini (Late 2014)].

=== Spec ===

* Intel i7-4578U (4) @ 3.500GHz
* 16GB RAM
* Intel Iris Graphics 5100
* 256GB On-board SSD

=== Speaker ===
powernap has the office speakers (a pair of nice studio monitors) currently connected to it.

=== Services ===
* MPD for playing music. Only office/termcom/syscom can log into powernap. Use `ncmpcpp` to control MPD.
** TODO: this is not the case anymore
* Bluetooth audio receiver. Only syscom can control bluetooth pairing. Use `bluetoothctl` to control bluetooth.

Music is located in `/music` on the office terminals.

= Progcom Only =
The Programme Committee has access to a VM on corn-syrup called 'progcom'. They have sudo rights in this VM so they may install and run their own software inside it. This VM should only be accessible by members of progcom or syscom.

The CI/CD stuff for the csclub.uwaterloo.ca runs on this vm (drone).

= Codey Bot Only =
codey-prod: QEMU virtual machine hosted on [[Machine List#Teriyaki|Teriyaki]]

codey-staging, codey-dev: Used to ran on Cloudstack, currently not deployed on Proxmox due to a lack of Codey Bot devs.

= Syscom Only =

The following systems are only be accessible to members of the [[Systems Committee]] for a variety of reasons; the most common of which being that some of these machines host [[Kerberos]] authentication services for the CSC.

==''xylitol''==

xylitol is a Dell PowerEdge R815 donated by CSCF. It is primarily a container host for services previously hosted on aspartame and dextrose, including munin, rt, mathnews, auth1, and dns1. It was provisioned with the intent to replace both of those hosts.

==== Specs ====

* Dual AMD Opteron 6176 (2.3 GHz, 48 cores total)
* 128GB RAM
* 500GB volume group on RAID1 SSD (xylitol-mirrored)
* 500ish-GB volume group on RAID10 HDD (xylitol-raidten)

==== Services ====

* [[Machine List#auth1|auth1]]
* [[Machine List#chat|chat]]
* [[Machine List#mail|mail]]
* mailman3
* [[Machine List#munin|munin]]
* [[BigBlueButton|bigbluebutton3]]
* dns1
* keycloak
* mathnews
* mattermost

==''auth1''==

Container on [[#xylitol|xylitol]].

==== Services ====

*[[LDAP]] primary
*[[Kerberos]] primary

==''chat''==

Container on [[#xylitol|xylitol]].

==== Services ====

* The Lounge web IRC client (https://chat.csclub.uwaterloo.ca)

==''mail''==

mail is the CSC's mail server. It hosts mail delivery, imap(s), smtp(s), and mailman. It is also syscom-only. It is a [[Virtualization#Linux_Containers|Linux container]] at present.

TODO: "HA"-ish configuration

==== Specs ====

* currently hosted on [[#xylitol|xylitol]]

==== Services ====

*[[Mail]] services
* mailman (web interface at [http://mailman.csclub.uwaterloo.ca/])
*[[Webmail]]
*[[ceo]] daemon

==''munin''==

munin is a syscom-only monitoring and accounting machine. It is a [[Virtualization#Linux_Containers|Linux container]] at present.

==== Specs ====

* currently hosted on [[#xylitol|xylitol]]

==== Services ====

*[http://munin.csclub.uwaterloo.ca munin] systems monitoring daemon
==''phosphoric-acid''==

phosphoric-acid is a Dell PowerEdge R815 donated by CSCF and is a clone of xylitol. It may be used to provide redundant cloud services in the future.

==== Specs ====

* (clone of Xylitol)

==== Services ====

*[[#coffee|coffee]]
*[[Observability|prometheus]]

==''coffee''==

Virtual machine running on phosphoric-acid.

==== Services ====

*[[Database#MySQL|MySQL]]
*[[Database#Postgres|Postgres]]

==''cobalamin''==

Dell PowerEdge 2950 donated to us by FEDS. Located in the Science machine room on the first floor of Physics, on Science Computing Rack 2. NICs are plugged into A1 and A2 on the adjacent rack. Acts as a backup server for many things.

TODO: should replace with another Syscom server when Science Computing clears out the rack (ETA before 09/2024)

==== Specs ====

* 1 × Intel Xeon E5420 (2.50 GHz, 4 cores)
* 16GB RAM
* Broadcom NetworkXtreme II
* 2x73GB Hard Drives, hardware RAID1
** Soon to be 2x1TB in MegaRAID1
*http://www.dell.com/support/home/ca/en/cabsdt1/product-support/servicetag/51TYRG1/configuration

==== Services ====

* Containers: [[#auth2|auth2]] (kerberos)

==== Notes ====

* '''TODO: Mega unreliable.''' (Goes down once every few weeks... due to power outages in the PHYS server room)
** It is plugged into a UPS but the UPS has dead batteries.
* The network card requires non-free drivers. Be sure to use an installation disc with non-free.

* We have separate IP ranges for cobalamin and its containers because the machine is located in a different building. They are:
** VLAN ID 506 (csc-data1): 129.97.18.16/29; gateway 129.97.18.17; mask 255.255.255.240
** VLAN ID 504 (csc-ipmi): 172.19.5.24/29; gateway 172.19.5.25; mask 255.255.255.248
* Physical access to the PHYS server rooms can be acquired by visiting Science Computing in PHYS 2006.

==''auth2''==

Container on [[#cobalamin|cobalamin]].

==== Services ====

*[[LDAP]] secondary
*[[Kerberos]] secondary

MAC Address: c2:c0:00:00:00:a2

==''potassium-benzoate''==

potassium-benzoate is our mirror server, funded by MEF.

==== Specs ====

* 36 drive Supermicro chassis (SSG-6048R-E1CR36L)
* 2 x Intel Xeon E5-2695 v4 (18 cores, 2.10GHz)
* 64 GB (4 x 16GB) of DDR4 (2133Mhz) ECC RDIMM RAM
* 2 x 1 TB Samsung Evo 850 SSD drives
* 17 x 4 TB Western Digital Gold drives (separate funding from MEF)
* 9 x 18TB Seagate Exos X18 (8 ZFS, Z2,1 hot-spare)
* 10 Gbps SFP+ card (loaned from CSCF)
* 50 Gbps Mellanox QSFP card (from ginkgo; currently unconnected)

Spec before 2025-03-27:
* 1 x Intel Xeon E5-2630 v3 (8 cores, 2.40 GHz)

==== Network Connections ====

potassium-benzoate has two connections to our network:

* 1 Gbps to our switch (used for management)
* 2 x 10 Gbps (LACP bond) to mc-rt-3015-mso-a (for mirror)

Mirror's bandwidth is limited to 1 Gbps on each of the 4 campus internet links. Mirror's bandwidth is not limited on campus.

==== Services ====

*[[Mirror]]
*[[Talks]] mirror
*[[Debian_Repository|CSClub packages repository]]

= User Cloud (Proxmox) =
A [[Proxmox]] Cluster consistent of 3 nodes

==== Services: ====

* [[Machine List#caffeine|caffeine]]
* [https://pass.uwaterloo.ca/ pass.uwaterloo.ca], a university-wide password manager hosted by CSC as a demo service for all Nexus (ADFS) user.
* reverse-proxy
* k3s (in progress)
* [[Matrix]]
* pmc (Website for the Pure Math Club)

== ''tahini'' ==
Server was funded via SLEF. Part of Proxmox Project, and mass migration.

==== Specs ====
* 2x AMD EPYC 9654 96-Core Processor
* 256gb DDR5 ram

== ''teriyaki'' ==
Server was funded via SLEF. Part of Proxmox Project, and mass migration. Located in DC 3558

==== Specs ====
* 2x AMD EPYC 9654 96-Core Processor
* 256gb DDR5 ram

== ''sorbitol'' ==

sorbitol is a SuperMicro server funded by MEF. Used for Proxmox.

==== Specs ====

* 2x Intel Xeon E5-2630 v4 processors (2.2 GHz, 10 cores/20 threads each)
* 64GB RAM

= Syscom Cloud (Proxmox) =
Syscom only [[Proxmox]] Cluster consistent of 3 nodes

==== Services: ====

* test-ipv6 (test-ipv6.csclub.uwaterloo.ca; a test-ipv6.com mirror)

== rum ==
Dell PowerEdge R730 on loan by CSCF to replace our older Dells.

==== Specs ====

* 2x Intel Xeon E5-2697 v3
* 256 GB DDR4
* 2x 2TB Kingston KC3000 (KIN-SKC3000D2048G)
** Mounted on Startech Dual M.2 PCIE SSD Adapter Cards (STA-PEX8M2E2)

== tequila ==
Dell PowerEdge R730 on loan by CSCF to replace our older Dells. Located in DC 3558.

==== Specs ====

* 2x Intel Xeon E5-2697 v3
* 256 GB DDR4
* LSI 9300-8e HBA + 2 NetApp DS4324 HDD shelves (24-disks each)
** 24 x 2TB HDDs (assorted brands/models)
** Dual IOM3 controllers.

== vodka ==
Dell PowerEdge R730 on loan by CSCF to replace our older Dells.

=== Specs ===

* 2x Intel Xeon E5-2697 v3
* 256 GB DDR4
* 2x 2TB Kingston KC3000 (KIN-SKC3000D2048G)
** Mounted on Startech Dual M.2 PCIE SSD Adapter Cards (STA-PEX8M2E2)

= Storage =
See also [[Filer]]

== ranch ==
Dell PowerEdge R730 on loan by CSCF to replace our older Dells.

=== Specs ===

* 2x Intel Xeon E5-2697 v3
* 256 GB DDR4
* 2 NetApp DS4324 HDD shelves (24-disks each)
** 24 x 2TB HDDs (assorted brands/models)
** Dual IOM3 controllers.

= UPS =

All of the machines in the MC 3015 machine room are connected to one of our UPSs.

All of our UPSs can be monitored via CSCF:

* MC3015-UPS-B2
* mc-3015-e7-ups-1.cs.uwaterloo.ca (rbc55, batteries replaced July 2014) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-e7-ups-1&var-Interval=30m)
* mc-3015-f7-ups-1.cs.uwaterloo.ca (rbc55, batteries replaced Feb 2017) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-f7-ups-1&var-Interval=30m)
* mc-3015-g7-ups-1.cs.uwaterloo.ca (su5000t, batteries replaced 2010) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-g7-ups-1&var-Interval=30m)
* mc-3015-g7-ups-2.cs.uwaterloo.ca (unknown) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-g7-ups-2&var-Interval=30m)
* mc-3015-h7-ups-1.cs.uwaterloo.ca (su5000t, batteries replaced 2004) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-h7-ups-1&var-Interval=30m)
* mc-3015-h7-ups-2.cs.uwaterloo.ca (unknown) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-h7-ups-2&var-Interval=30m)

We will receive email alerts for any issues with the UPS. Their status can be monitored via [[SNMP]].

TODO: Fix labels & verify info is correct & figure out why we can't talk to cacti.
= Other =

== ps3 ==
This is just a very wide PS3, the model that supported running Linux natively before it was removed. Firmware was updated to remove this feature, however it can still be done via. homebrew.

'''Specs'''

* It's a PS3.

'''2022-10-24''' - Thermal paste replaced + firmware updated to latest supported version, also modded.

==''binaerpilot''==

This is a Gumstix Overo Tide CPU on a Tobi expansion board. It is currently attached to corn-syrup in the machine room and even more currently turned off until someone can figure out what is wrong with it.

==== Specs ====

* TI OMAP 3530 750Mhz (ARM Cortex-A8)
* 512MB RAM

==''anamanaguchi''==

This is a Gumstix Overo Tide CPU on a Chestnut43 expansion board. It is currently in the hardware drawer in the CSC.

==== Specs ====

* TI OMAP 3530 750Mhz (ARM Cortex-A8)
* 512MB RAM

'''NOTE: May have disappeared at some point'''

==''digital cutter''==

See [[Digital Cutter|here]].
==== Specs ====

* Intel Core i7-6700k
* 2x8GB RAM
* 1x 64GB SanDisk SDSSDP064G SSD
* Cup Holder (DVD drive has power, but not connected to mother board)

Matrix

2025-10-29T02:00:38Z

K95ma: /* Discord Instructions */ fix

Matrix

2025-10-29T01:57:23Z

K95ma: /* Discord Instructions */ italic

Matrix

2025-10-29T01:56:15Z

K95ma: /* Discord Instructions */ fix

Matrix

2025-10-29T01:55:56Z

K95ma: /* Discord Instructions */ +

Template:Code

2025-10-29T01:55:11Z

K95ma: copied template from https://en.wikipedia.org/w/index.php?title=Template:Code&action=history; see url for attribution

{{#tag:syntaxhighlight|{{{code|{{{1}}}}}}|lang={{{lang|{{{2|text}}}}}}|class={{{class|}}}|style={{{style|}}}|inline=1}}

Matrix

2025-10-29T01:53:31Z

K95ma: +

Matrix

2025-10-26T21:18:23Z

K95ma: /* Matrix Installation */ -

Matrix

2025-10-26T20:59:44Z

K95ma: +

NixOS

2025-09-29T04:23:38Z

K95ma: +

Matrix

2025-09-18T02:50:09Z

K95ma: +

We are currently setting up a test server for Matrix. We use Synapse. If everything goes well, we will set up a production Matrix server.

== Server Setup ==

We are currently running Matrix on a Proxmox LXE container on <code>citric-acid</code>. Ask Siracha for the credentials to access Proxmox and the VM.

== Matrix Installation ==

We are using NixOS, so use the following config (might be a bit messy) to setup both Synapse, Nginx, and PostgreSQL:

'''Note''': We could have used <code>recommendedProxySettings = true</code> which sets most of the <code>X-Forwarded-For</code> headers correctly. However, this is a reverse proxy behind a reverse proxy (Nginx on citric acid -> Nginx on the container -> Synapse) so <code>X-Forwarded-Proto</code> has to be always set to https. I'm sure there is a better way to do this.

<pre>
{ pkgs, lib, config, ... }:
let
fqdn = "matrix.${config.networking.domain}";
clientConfig = {
"m.homeserver".base_url = "https://${fqdn}";
"m.identity_server" = {};
};
serverConfig."m.server" = "${fqdn}:443";
mkWellKnown = data: ''
add_header Content-Type application/json;
add_header Access-Control-Allow-Origin *;
return 200 '${builtins.toJSON data}';
'';
extraCfg = pkgs.writeText "synapse-extra-config.yaml" ''
'';
in {
networking.firewall = {
enable = true;
allowedTCPPorts = [ 80 8008 ];
};
networking.domain = "csclub.uwaterloo.ca";

services.postgresql = {
enable = true;
initialScript = pkgs.writeText "synapse-init.sql" ''
CREATE ROLE "matrix-synapse" WITH LOGIN PASSWORD 'synapse';
CREATE DATABASE "matrix-synapse" WITH OWNER "matrix-synapse"
TEMPLATE template0
LC_COLLATE = "C"
LC_CTYPE = "C";
'';
dataDir = "/data/postgresql";
};

services.nginx = {
enable = true;
# recommendedProxySettings = true;
virtualHosts = {
"csclub.uwaterloo.ca" = {
locations."/".extraConfig = ''
return 404;
'';
locations."= /.well-known/matrix/server".extraConfig = mkWellKnown serverConfig;
locations."= /.well-known/matrix/client".extraConfig = mkWellKnown clientConfig;
};
"${fqdn}" = {
locations."/".extraConfig = ''
return 404;
'';
locations."/_matrix".proxyPass = "http://[::1]:8008";
locations."/_matrix".extraConfig = ''
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
'';
locations."/_synapse/client".proxyPass = "http://[::1]:8008";
locations."/_synapse/client".extraConfig = ''
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
'';
};
};
};

services.matrix-synapse = {
enable = true;
extraConfigFiles = [ extraCfg ];
dataDir = "/data/matrix-synapse";
settings = {
server_name = config.networking.domain;
public_baseurl = "https://matrix.csclub.uwaterloo.ca/";
listeners = [
{
port = 8008;
bind_addresses = [ "::1" ];
type = "http";
tls = false;
x_forwarded = true;
resources = [
{
names = [ "client" ]; # no federation
compress = true;
}
];
}
];

oidc_providers = [
{
idp_id = "keycloak";
idp_name = "CSC";
issuer = "https://keycloak.csclub.uwaterloo.ca/realms/csc";
client_id = "synapse";
client_secret = "xxxx"; # fill the client secret from keycloak here
scopes = [ "openid" "profile" ];
user_mapping_provider.config = {
localpart_template = "{{ user.preferred_username }}";
display_name_template = "{{ user.name }}";
};

}
];
registration_shared_secret = "xxxxx";
enable_registration = true;
enable_registration_captcha = false;
registration_requires_token = true;
};
};
}
</pre>

== Testing ==

Go to https://app.cinny.in/login/matrix.csclub.uwaterloo.ca, and click "Continue with CSC".

Proxmox

2025-09-16T02:22:47Z

K95ma: +

The Proxmox Vitural Environment (as of 2025-09-15) lives on the citric-acid machine and can be accessed via https://citric-acid.csclub.uwaterloo.ca:8006.

== Setting up Proxmox ==
To setup proxmox, from `Server View`, open the `Datacenter` page. Then go to `Permissions -> Realms`.

Then just make sure pam is setup lol

== Networking ==
There are two ways to do networking: network bridge and NAT. Network bridge will put the container/virtual machine on the CSC network (basically side-by-side to proxmox itself), while NAT will encapsulate the container/VM inside a private subnet that is only visible to proxmox host itself.

For services that only exposes HTTP/HTTPS, NAT is more desirable since multiple services can share a host nginx instance, only requiring the host IP to have 80/443 port opened to the Internet, thus saving some IP address in our pool and save some trips to the IST for firewall exemption. But for services that requires custom ports to be opened (for example, BigBlueButton requires a range of UDP ports to be exposed for relaying video streams), using the network bridge and giving the container/VM its own public IP might be easier.

Currently, <code>vmbr0</code> is used for bridged network and <code>vmbr1</code> is used for NAT (see [https://pve.proxmox.com/wiki/Network_Configuration#sysadmin_network_masquerading Proxmox's wiki on NAT networking] for setup instruction). <code>vmbr0</code> uses the CSC DHCP server, so you can use DHCP there, but <code>vmbr1</code> requires manual IP assignment.

Note that only using <code>vmbr1</code> requires you to use SSH ProxyJump via citric-acid to access the inner container, as it wouldn't have a public IP.

Keycloak

2025-09-16T01:20:31Z

K95ma: fix

We are using [https://www.keycloak.org/ Keycloak] for web SSO (Single Sign-On). Clients may use Keycloak for authenticating users via SAML or OIDC (OpenID Connect).

* Admin login: https://keycloak.csclub.uwaterloo.ca/admin
* Regular user login: https://keycloak.csclub.uwaterloo.ca/realms/csc/account
* OIDC Auto Discovery URL: https://keycloak.csclub.uwaterloo.ca/realms/csc/.well-known/openid-configuration

== Prerequisites ==
OK so before we get started, there's this really useful feature in Keycloak called "Conditional user attribute" which allows you to create a flow which branches based on attributes a user may have. For some reason, this is enabled in the test suite for Keycloak, but is not available from the main application. So we're going to compile and inject it ourselves.

Clone https://git.csclub.uwaterloo.ca/public/keycloak-spi and run <code>mvn clean package</code>. This will create a JAR file called csc-keycloak-spi.jar in the target directory; upload this to somewhere where it can be easily downloaded, e.g. your www directory.

== Database setup ==
Go to biloba or chamomile, run <code>mysql</code>, and run the following:
<pre>
CREATE USER 'keycloak' IDENTIFIED BY 'replace_this_password';
CREATE DATABASE keycloak CHARACTER SET utf8 COLLATE utf8_unicode_ci;
GRANT ALL PRIVILEGES ON keycloak.* TO 'keycloak';
FLUSH PRIVILEGES;
</pre>

== Kubernetes setup ==
We are running Keycloak on Kubernetes. This introduces some complications because it gets reverse proxied twice, and we also can't (or at least shouldn't) modify the filesystem of the Pod where it's running, since that Pod can get destroyed at any time. We still need to load that JAR file we just created, though, so we're going to place it into a PersistentVolume instead. We're going to do this by first creating a PersistentVolumeClaim, then claiming it in a temporary Pod which we'll use for shell access:
<pre>
cat <<EOF | kubectl apply -f
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
namespace: syscom
name: keycloak-spi-pvc
spec:
storageClassName: cloudstack-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Mi
---
apiVersion: v1
kind: Pod
metadata:
namespace: syscom
name: temp-pod
spec:
containers:
- name: temp
image: alpine
volumeMounts:
- mountPath: "/data"
name: keycloak-spi-pv
stdin: true
stdinOnce: true
tty: true
volumes:
- name: keycloak-spi-pv
persistentVolumeClaim:
claimName: keycloak-spi-pvc
EOF
</pre>
Run <code>kubectl -n syscom get pods</code> a few times to check if the pod is ready; once it is, attach to it:
<pre>
kubectl -n syscom exec -it temp-pod -- sh
cd /data
mkdir keycloak-spi
chmod a+w keycloak-spi
cd keycloak-spi
wget https://csclub.uwaterloo.ca/~merenber/csc-keycloak-spi.jar
exit
</pre>
Now delete the pod since we don't need it anymore:
<pre>
kubectl -n syscom delete pod temp-pod
</pre>
Create some secrets (use the MySQL password which you chose earlier):
<pre>
kubectl -n syscom create secret generic keycloak-secret \
--from-literal=DB_USER=some_user \
--from-literal=DB_PASSWORD=some_password \
--from-literal=KEYCLOAK_USER=some_user \
--from-literal=KEYCLOAK_PASSWORD=some_password
</pre>
Now apply the main manifest:
<pre>
kubectl apply -f https://git.csclub.uwaterloo.ca/cloud/manifests/raw/branch/master/keycloak.yaml
</pre>

== DNS setup ==
From Infoblox, make keycloak.csclub.uwaterloo.ca a CNAME for rr-public-cloud.csclub.uwaterloo.ca; that record points to biloba and chamomile, which know how to reverse proxy requests to Kubernetes.

== NGINX setup ==
Pretty standard stuff:
<pre>
server {
listen 80;
listen [::]:80;
server_name keycloak.csclub.uwaterloo.ca;
return 301 https://$host$request_uri;
}

server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name keycloak.csclub.uwaterloo.ca;
ssl_certificate /etc/ssl/private/csclub.uwaterloo.ca.chain;
ssl_certificate_key /etc/ssl/private/csclub.uwaterloo.ca.key;

location / {
proxy_pass http://k8s;
}
include proxy_params;

access_log /var/log/nginx/keycloak-access.log;
error_log /var/log/nginx/keycloak-error.log;
}
</pre>
Also make sure you have the following snippet in /etc/nginx/proxy_params:
<pre>
# Increase buffer size
# See https://ma.ttias.be/nginx-proxy-upstream-sent-big-header-reading-response-header-upstream/
proxy_buffer_size 128k;
proxy_buffers 4 256k;
</pre>
Don't forget to enable the site and reload NGINX on both chamomile and biloba.

== Web UI setup ==
If all went well, you should now be able to visit https://keycloak.csclub.uwaterloo.ca from your browser.
Create a new realm called 'csc'. Set the Display Name to 'Computer Science Club'.

=== Realm settings ===
From the web UI, go to 'Realm Settings', click the Login tab, and disable 'Login with email'.

Now click on the 'Tokens' tab. Set 'SSO Session Idle' and 'SSO Session Max' to 120 days each. You can optionally do this for the Master realm as well.

Now click on the 'Email' tab.

* Set 'Host' to 'mail.csclub.uwaterloo.ca'.
* Set 'From Display Name' to 'Keycloak'.
* Set 'From' to 'no-reply@csclub.uwaterloo.ca'.
* Set 'Reply To Display Name' to 'Systems Committee'.
* Set 'Reply To' to 'syscom@csclub.uwaterloo.ca'.
* Set 'Envelope From' to 'keycloak@csclub.uwaterloo.ca'.
* Enable StartTLS.

=== Authentication Settings ===
Click on 'Authentication' from the panel on the left-hand side.

==== Flows ====
Click on the 'Flows' tab and select the 'Browser' flow from the dropdown. Create a copy called 'CSC Browser' and make it look like this:
<pre>
Cookie (Alternative)
CSC Browser Forms (Alternative)
Username Password Form (Required)
CSC Membership Check (Conditional)
Condition - User Attribute (Required)
Deny Access (Required)
OTP Form (Required)
</pre>
The condition should be whether a user has the attribute 'shadowExpire' set to '1'. Set the error message in 'Deny Access' to something reasonable (this gets displayed to users).

Screenshot:
[[File:Keycloak_CSC_Browser_Flow.png]]

Select the 'First Broker Login' flow and create a copy called 'CSC First Broker Login'. This will be invoked when a user signs in with an Identity Provider, and the IdP user ID is not linked to a Keycloak user. Make the flow look like this:
<pre>
Review Profile (Disabled)
User Creation or Linking (Required)
Detect Existing Broker User (Required)
Automatically Set Existing User (Required)
CSC Membership Check (Conditional)
Condition - User Attribute (Required)
Deny Access (Required)
</pre>

Finally, create a new flow called 'CSC Browser - IdP post-login'. This will be invoked when a user signs in with an IdP, and the IdP user ID has already been linked to a Keycloak user. Make it look like this:
<pre>
CSC Membership Check - IdP Post-login (Conditional)
Condition - User Attribute (Membership expired - IdP post-login) (Required)
Deny Access (Denied because membership expired - IdP post-login) (Required)
Allow Access
</pre>
Note that the last step needs to be 'Allow Access'.

==== Bindings ====
Click on the 'Bindings' tab and set 'Browser Flow' to 'CSC Browser'.

==== Required Actions ====
Click on the 'Required Actions' tab and disable the following:

* Update Password
* Update Profile
* Delete Account

=== User Federation ===
Click on 'User Federation' from the left-hand panel and select 'ldap'. Configure it as follows:

* Edit Mode: READ_ONLY
* Vendor: Other
* Username LDAP attribute: uid
* RDN LDAP attribute: uid
* UUID LDAP attribute: entryUUID
* User Object Classes: member
* Connection URL: ldaps://ldap1.csclub.uwaterloo.ca ldaps://ldap2.csclub.uwaterloo.ca
* Users DN: ou=People,dc=csclub,dc=uwaterloo,dc=ca
* Custom User LDAP Filter: (objectClass=member)
* Search Scope: One Level
* Bind Type: none

Under Sync Settings, enable Changed Users Full Sync. Set the Changed Users Sync Period to something reasonable (should be at least once a day).

Click on Mappers, click 'email', and set the LDAP Attribute to 'mailLocalAddress'. Also set the LDAP Attribute for 'First Name' to 'givenName'.

Create a new mapper of type 'user-attribute-ldap-mapper'. Set 'User Model Attribute' and 'LDAP Attribute' to 'shadowExpire'. Enable 'Always Read Value From LDAP'.

Now click the 'Synchronize all users' button from the main LDAP configuration page. This will take a few minutes so be patient.

=== Identity Providers ===
We are going to use [[ADFS]] for identity brokering. The idea is to allow members to login using their CSC credentials or their UW credentials.

==== SAML Passthrough ====
I created a monstrosity called [https://git.csclub.uwaterloo.ca/merenber/saml-passthrough saml-passthrough]. The idea is to protect a SAML IdP behind another SAML IdP (ADFS). So basically SAML-inside-SAML.

The reason why we need this is because we can't change the AssertionConsumerService endpoint URL which we originally submitted to IST (if we really wanted to, we could submit another form, but that requires assistance from a faculty member, and it's just a big PiTA). This also allows us to multiplex ADFS information to multiple Keycloak realms, if necessary.

Anywho, the instructions for setting it up are in the README (it's just a FastCGI application). It should be running on caffeine right now under /srv/saml-passthrough. It's configured to download SP metadata from Keycloak at startup, so if you update the SP certificate in Keycloak, make sure to restart the saml-passthrough service.

Now, back to the Keycloak UI. Click 'Identity Providers' from the left-hand side, and add a SAML provider. Set the following:

* Alias: adfs
* Display Name: University of Waterloo (ADFS)
* First Login Flow: CSC First Login Flow
* Post Login Flow: CSC Browser - IdP post-login
* Sync Mode: legacy
* Service Provider Entity ID: https://keycloak.csclub.uwaterloo.ca/auth/realms/csc
* Single Sign-On Service URL: https://csclub.uwaterloo.ca/keycloak/saml/sso
* Principal Type: Attribute [Friendly Name]
* Principal Attribute: uid
* HTTP-POST Binding Response: ON
* Want AuthnRequests Signed: ON
* Want Assertions Signed: ON
* Validate Signature: ON
* Validating X509 certificates: download the content of https://csclub.uwaterloo.ca/keycloak/saml/metadata and copy and paste the value inside the <code><X509Certificate></code> tag.

That X509 certificate will expire in January 2032; before then, make sure you do the following:

* create a new keypair in /srv/saml-passthrough on caffeine
* restart the saml-passthrough service
* update the 'Validating X509 certificates' field in Keycloak

Keycloak's SP certificate (which can be viewed from Realm Settings -> Keys -> Providers -> rsa-generated) will expire in December 2031; before then, make sure you upload a new keypair (just choose 'rsa-generated' from the 'Add keystore' dropdown), then restart the saml-passthrough service on caffeine.

== Testing ==
Here's a test OIDC application which you can use to verify that everything is working: https://www.keycloak.org/app/#url=https://keycloak.csclub.uwaterloo.ca/&realm=csc&client=test

Keycloak

2025-09-16T01:05:24Z

K95ma: /* Testing */ fix

We are using [https://www.keycloak.org/ Keycloak] for web SSO (Single Sign-On). Clients may use Keycloak for authenticating users via SAML or OIDC (OpenID Connect).

* Admin login: https://keycloak.csclub.uwaterloo.ca/auth/admin
* Regular user login: https://keycloak.csclub.uwaterloo.ca/auth/realms/csc/account
* OIDC Auto Discovery URL: https://keycloak.csclub.uwaterloo.ca/auth/realms/csc/.well-known/openid-configuration

== Prerequisites ==
OK so before we get started, there's this really useful feature in Keycloak called "Conditional user attribute" which allows you to create a flow which branches based on attributes a user may have. For some reason, this is enabled in the test suite for Keycloak, but is not available from the main application. So we're going to compile and inject it ourselves.

Clone https://git.csclub.uwaterloo.ca/public/keycloak-spi and run <code>mvn clean package</code>. This will create a JAR file called csc-keycloak-spi.jar in the target directory; upload this to somewhere where it can be easily downloaded, e.g. your www directory.

== Database setup ==
Go to biloba or chamomile, run <code>mysql</code>, and run the following:
<pre>
CREATE USER 'keycloak' IDENTIFIED BY 'replace_this_password';
CREATE DATABASE keycloak CHARACTER SET utf8 COLLATE utf8_unicode_ci;
GRANT ALL PRIVILEGES ON keycloak.* TO 'keycloak';
FLUSH PRIVILEGES;
</pre>

== Kubernetes setup ==
We are running Keycloak on Kubernetes. This introduces some complications because it gets reverse proxied twice, and we also can't (or at least shouldn't) modify the filesystem of the Pod where it's running, since that Pod can get destroyed at any time. We still need to load that JAR file we just created, though, so we're going to place it into a PersistentVolume instead. We're going to do this by first creating a PersistentVolumeClaim, then claiming it in a temporary Pod which we'll use for shell access:
<pre>
cat <<EOF | kubectl apply -f
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
namespace: syscom
name: keycloak-spi-pvc
spec:
storageClassName: cloudstack-storage
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Mi
---
apiVersion: v1
kind: Pod
metadata:
namespace: syscom
name: temp-pod
spec:
containers:
- name: temp
image: alpine
volumeMounts:
- mountPath: "/data"
name: keycloak-spi-pv
stdin: true
stdinOnce: true
tty: true
volumes:
- name: keycloak-spi-pv
persistentVolumeClaim:
claimName: keycloak-spi-pvc
EOF
</pre>
Run <code>kubectl -n syscom get pods</code> a few times to check if the pod is ready; once it is, attach to it:
<pre>
kubectl -n syscom exec -it temp-pod -- sh
cd /data
mkdir keycloak-spi
chmod a+w keycloak-spi
cd keycloak-spi
wget https://csclub.uwaterloo.ca/~merenber/csc-keycloak-spi.jar
exit
</pre>
Now delete the pod since we don't need it anymore:
<pre>
kubectl -n syscom delete pod temp-pod
</pre>
Create some secrets (use the MySQL password which you chose earlier):
<pre>
kubectl -n syscom create secret generic keycloak-secret \
--from-literal=DB_USER=some_user \
--from-literal=DB_PASSWORD=some_password \
--from-literal=KEYCLOAK_USER=some_user \
--from-literal=KEYCLOAK_PASSWORD=some_password
</pre>
Now apply the main manifest:
<pre>
kubectl apply -f https://git.csclub.uwaterloo.ca/cloud/manifests/raw/branch/master/keycloak.yaml
</pre>

== DNS setup ==
From Infoblox, make keycloak.csclub.uwaterloo.ca a CNAME for rr-public-cloud.csclub.uwaterloo.ca; that record points to biloba and chamomile, which know how to reverse proxy requests to Kubernetes.

== NGINX setup ==
Pretty standard stuff:
<pre>
server {
listen 80;
listen [::]:80;
server_name keycloak.csclub.uwaterloo.ca;
return 301 https://$host$request_uri;
}

server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name keycloak.csclub.uwaterloo.ca;
ssl_certificate /etc/ssl/private/csclub.uwaterloo.ca.chain;
ssl_certificate_key /etc/ssl/private/csclub.uwaterloo.ca.key;

location / {
proxy_pass http://k8s;
}
include proxy_params;

access_log /var/log/nginx/keycloak-access.log;
error_log /var/log/nginx/keycloak-error.log;
}
</pre>
Also make sure you have the following snippet in /etc/nginx/proxy_params:
<pre>
# Increase buffer size
# See https://ma.ttias.be/nginx-proxy-upstream-sent-big-header-reading-response-header-upstream/
proxy_buffer_size 128k;
proxy_buffers 4 256k;
</pre>
Don't forget to enable the site and reload NGINX on both chamomile and biloba.

== Web UI setup ==
If all went well, you should now be able to visit https://keycloak.csclub.uwaterloo.ca from your browser.
Create a new realm called 'csc'. Set the Display Name to 'Computer Science Club'.

=== Realm settings ===
From the web UI, go to 'Realm Settings', click the Login tab, and disable 'Login with email'.

Now click on the 'Tokens' tab. Set 'SSO Session Idle' and 'SSO Session Max' to 120 days each. You can optionally do this for the Master realm as well.

Now click on the 'Email' tab.

* Set 'Host' to 'mail.csclub.uwaterloo.ca'.
* Set 'From Display Name' to 'Keycloak'.
* Set 'From' to 'no-reply@csclub.uwaterloo.ca'.
* Set 'Reply To Display Name' to 'Systems Committee'.
* Set 'Reply To' to 'syscom@csclub.uwaterloo.ca'.
* Set 'Envelope From' to 'keycloak@csclub.uwaterloo.ca'.
* Enable StartTLS.

=== Authentication Settings ===
Click on 'Authentication' from the panel on the left-hand side.

==== Flows ====
Click on the 'Flows' tab and select the 'Browser' flow from the dropdown. Create a copy called 'CSC Browser' and make it look like this:
<pre>
Cookie (Alternative)
CSC Browser Forms (Alternative)
Username Password Form (Required)
CSC Membership Check (Conditional)
Condition - User Attribute (Required)
Deny Access (Required)
OTP Form (Required)
</pre>
The condition should be whether a user has the attribute 'shadowExpire' set to '1'. Set the error message in 'Deny Access' to something reasonable (this gets displayed to users).

Screenshot:
[[File:Keycloak_CSC_Browser_Flow.png]]

Select the 'First Broker Login' flow and create a copy called 'CSC First Broker Login'. This will be invoked when a user signs in with an Identity Provider, and the IdP user ID is not linked to a Keycloak user. Make the flow look like this:
<pre>
Review Profile (Disabled)
User Creation or Linking (Required)
Detect Existing Broker User (Required)
Automatically Set Existing User (Required)
CSC Membership Check (Conditional)
Condition - User Attribute (Required)
Deny Access (Required)
</pre>

Finally, create a new flow called 'CSC Browser - IdP post-login'. This will be invoked when a user signs in with an IdP, and the IdP user ID has already been linked to a Keycloak user. Make it look like this:
<pre>
CSC Membership Check - IdP Post-login (Conditional)
Condition - User Attribute (Membership expired - IdP post-login) (Required)
Deny Access (Denied because membership expired - IdP post-login) (Required)
Allow Access
</pre>
Note that the last step needs to be 'Allow Access'.

==== Bindings ====
Click on the 'Bindings' tab and set 'Browser Flow' to 'CSC Browser'.

==== Required Actions ====
Click on the 'Required Actions' tab and disable the following:

* Update Password
* Update Profile
* Delete Account

=== User Federation ===
Click on 'User Federation' from the left-hand panel and select 'ldap'. Configure it as follows:

* Edit Mode: READ_ONLY
* Vendor: Other
* Username LDAP attribute: uid
* RDN LDAP attribute: uid
* UUID LDAP attribute: entryUUID
* User Object Classes: member
* Connection URL: ldaps://ldap1.csclub.uwaterloo.ca ldaps://ldap2.csclub.uwaterloo.ca
* Users DN: ou=People,dc=csclub,dc=uwaterloo,dc=ca
* Custom User LDAP Filter: (objectClass=member)
* Search Scope: One Level
* Bind Type: none

Under Sync Settings, enable Changed Users Full Sync. Set the Changed Users Sync Period to something reasonable (should be at least once a day).

Click on Mappers, click 'email', and set the LDAP Attribute to 'mailLocalAddress'. Also set the LDAP Attribute for 'First Name' to 'givenName'.

Create a new mapper of type 'user-attribute-ldap-mapper'. Set 'User Model Attribute' and 'LDAP Attribute' to 'shadowExpire'. Enable 'Always Read Value From LDAP'.

Now click the 'Synchronize all users' button from the main LDAP configuration page. This will take a few minutes so be patient.

=== Identity Providers ===
We are going to use [[ADFS]] for identity brokering. The idea is to allow members to login using their CSC credentials or their UW credentials.

==== SAML Passthrough ====
I created a monstrosity called [https://git.csclub.uwaterloo.ca/merenber/saml-passthrough saml-passthrough]. The idea is to protect a SAML IdP behind another SAML IdP (ADFS). So basically SAML-inside-SAML.

The reason why we need this is because we can't change the AssertionConsumerService endpoint URL which we originally submitted to IST (if we really wanted to, we could submit another form, but that requires assistance from a faculty member, and it's just a big PiTA). This also allows us to multiplex ADFS information to multiple Keycloak realms, if necessary.

Anywho, the instructions for setting it up are in the README (it's just a FastCGI application). It should be running on caffeine right now under /srv/saml-passthrough. It's configured to download SP metadata from Keycloak at startup, so if you update the SP certificate in Keycloak, make sure to restart the saml-passthrough service.

Now, back to the Keycloak UI. Click 'Identity Providers' from the left-hand side, and add a SAML provider. Set the following:

* Alias: adfs
* Display Name: University of Waterloo (ADFS)
* First Login Flow: CSC First Login Flow
* Post Login Flow: CSC Browser - IdP post-login
* Sync Mode: legacy
* Service Provider Entity ID: https://keycloak.csclub.uwaterloo.ca/auth/realms/csc
* Single Sign-On Service URL: https://csclub.uwaterloo.ca/keycloak/saml/sso
* Principal Type: Attribute [Friendly Name]
* Principal Attribute: uid
* HTTP-POST Binding Response: ON
* Want AuthnRequests Signed: ON
* Want Assertions Signed: ON
* Validate Signature: ON
* Validating X509 certificates: download the content of https://csclub.uwaterloo.ca/keycloak/saml/metadata and copy and paste the value inside the <code><X509Certificate></code> tag.

That X509 certificate will expire in January 2032; before then, make sure you do the following:

* create a new keypair in /srv/saml-passthrough on caffeine
* restart the saml-passthrough service
* update the 'Validating X509 certificates' field in Keycloak

Keycloak's SP certificate (which can be viewed from Realm Settings -> Keys -> Providers -> rsa-generated) will expire in December 2031; before then, make sure you upload a new keypair (just choose 'rsa-generated' from the 'Add keystore' dropdown), then restart the saml-passthrough service on caffeine.

== Testing ==
Here's a test OIDC application which you can use to verify that everything is working: https://www.keycloak.org/app/#url=https://keycloak.csclub.uwaterloo.ca/&realm=csc&client=test

User:K95ma

2025-09-13T01:54:11Z

K95ma: Created page with "Termcom guy"

Termcom guy

LDAP

2025-09-10T00:58:05Z

K95ma: /* Querying LDAP */ fix command

We use [http://www.openldap.org/ OpenLDAP] for directory services. Our primary LDAP server is [[Machine_List#auth1|auth1]] and our secondary LDAP server is [[Machine_List#auth2|auth2]].

=== ehashman's Guide to Setting up OpenLDAP on Debian ===

Welcome to my nightmare.

==== What is LDAP? ====

<blockquote>'''LDAP:''' Lightweight Directory Access Protocol

An open, vendor-neutral, industry standard application protocol for accessing and maintaining distributed directory information services over an Internet Protocol (IP) network. — [https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol Wikipedia: LDAP]
</blockquote>
In this case, "directory" refers to the user directory, like on an old-school Rolodex. Many groups use LDAP to maintain their user directory, including the University (the "WatIAM" identity management system), the Computer Science Club, and even the UW Amateur Radio Club.

This is a guide documenting how to set up LDAP on a Debian Linux system.

==== First steps ====

<ul>
<li>Ensure that openldap is installed on the machine:
<pre># apt-get install slapd ldap-utils</pre></li>
<li>Debian will do a lot of magic and set up a skeleton LDAP server and get it running. We need to configure that further.</li>
<li>Let's set up logging before we forget. Create the following files in <code>/var/log</code>:
<pre># mkdir /var/log/ldap
# touch /var/log/ldap.log</pre></li>
<li>Set ownership correctly:
<pre># chown openldap:openldap /var/log/ldap</pre></li>
<li>Set up rsyslog to dump the LDAP logs into <code>/var/log/ldap.log</code> by adding the following lines:
<pre># vim /etc/rsyslog.conf
...
# Grab ldap logs, don't duplicate in syslog
local4.* /var/log/ldap.log</pre></li>
<li>Set up log rotation for these by creating the file [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/logrotate.d.ldap <code>/etc/logrotate.d/ldap</code>] with the following contents:
<pre>/var/log/ldap/*log {
weekly
missingok
rotate 1000
compress
delaycompress
notifempty
create 0640 openldap adm
postrotate
if [ -f /var/run/slapd/slapd.pid ]; then
/etc/init.d/slapd restart >/dev/null 2>&1
fi
endscript
}

/var/log/ldap.log {
weekly
missingok
rotate 24
compress
delaycompress
notifempty
}</pre></li>
<li>As of OpenLDAP 2.4, it doesn't actually create a config file for us. Apparently, this is a "feature": LDAP maintainers think we should want to set this up via dynamic queries. We don't, so the first thing we need is our [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/slapd.conf <code>slapd.conf</code>] file.</li></ul>

===== Building <code>slapd.conf</code> from scratch =====

<ul>
<li>Get a copy to work with:
<pre># scp uid@auth1.csclub.uwaterloo.ca:/etc/ldap/slapd.conf /etc/ldap/ ## you need CSC root for this</pre></li>
<li>You'll want to comment out the TLS lines, and anything referring to Kerberos and access for now. You'll also want to comment out lines specifically referring to syscom and office staff.</li>
<li>Make sure you remove the reference to <code>nonMemberTerm</code> as an index, as we're going to remove this field.</li>
<li>You'll also need to generate a root password for the LDAP to bootstrap auth, like so:
<pre># slappasswd
New password:
Re-enter new password:
{SSHA}longhash</pre></li>
<li>Add this line below <code>rootdn</code> in the <code>slapd.conf</code>:
<pre>rootpw {SSHA}longhash</pre></li>
<li>Now we want to edit all instances of "csclub" to be "wics" instead, e.g.:
<pre>suffix "dc=wics,dc=uwaterloo,dc=ca"
rootdn "cn=root,dc=wics,dc=uwaterloo,dc=ca"</pre></li>
<li>Next, we need to grab all the relevant schemas:
<pre>scp -r uid@auth1.csclub.uwaterloo.ca:/etc/ldap/schema/ /tmp/schemas</pre></li>
<li>Use the include directives to help you find the ones you need. I noticed we were missing <code>sudo.schema</code>, <code>csc.schema</code>, and <code>rfc2307bis.schema</code>.</li>
<li>Open up the [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/csc.schema <code>csc.schema</code>] for editing; we're not using it verbatim. Remove the attributes <code>studentid</code> and <code>nonMemberTerm</code> and the objectclass <code>club</code>. Also make sure you change the OID so we don't clash with the CSC. Because we didn't want to go through the process of requesting a [http://pen.iana.org/pen/PenApplication.page PEN number], we chose arbitrarily to use 26338, which belongs to IWICS Inc.</li>
<li>We also need to can the auto-generated config files, so do that:
<pre># rm -rf /etc/openldap/slapd.d/*</pre></li>
<li>Also nuke the auto-generated database:
<pre># rm /var/lib/ldap/__db.*</pre></li>
<li>Configure the database:
<pre># cp /usr/share/slapd/DB_CONFIG /var/lib/ldap/
# chown openldap:openldap /var/lib/ldap/DB_CONFIG </pre></li>
<li>Now we can generate the new configuration files:
<pre># slaptest -f /etc/ldap/slapd.conf -F /etc/ldap/slapd.d/</pre></li>
<li>And ensure that the permissions are all set correctly, lest this break something:
<pre># chown -R openldap:openldap /etc/ldap/slapd.d</pre></li>
<li>If at this point you get a nasty error, such as
<pre>5657d4db hdb_db_open: database "dc=wics,dc=uwaterloo,dc=ca": db_open(/var/lib/ldap/id2entry.bdb) failed: No such file or directory (2).
5657d4db backend_startup_one (type=hdb, suffix="dc=wics,dc=uwaterloo,dc=ca"): bi_db_open failed! (2)
slap_startup failed (test would succeed using the -u switch)</pre>
Just try restarting slapd, and see if that fixes the problem:
<pre># service slapd stop
# service slapd start</pre></li>
<li>Congratulations! Your LDAP service is now configured and running.</li></ul>

==== Getting TLS Up and Running ====

<ul>
<li>Now that we have our LDAP service, we'll want to be able to serve encrypted traffic. This is especially important for any remote access, since binding to LDAP (i.e. sending it a password for auth) occurs over plaintext, and we don't want to leak our admin password.</li>
<li>Our first step is to copy our SSL certificates into the correct places. Public ones go into <code>/etc/ssl/certs/</code> and private ones go into <code>/etc/ssl/private/</code>.</li>
<li>Since the LDAP daemon needs to be able to read our private cert, we need to grant LDAP access to the private folder:
<pre># chgrp openldap /etc/ssl/private
# chmod g+x /etc/ssl/private</pre></li>
<li>Next, uncomment the TLS-related settings in <code>slapd.conf</code>. These are <code>TLSCertificateFile</code> (the public cert), <code>TLSCertificateKeyFile</code> (the private key), <code>TLSCACertificateFile</code> (the intermediate CA cert), and <code>TLSVerifyClient</code> (set to "allow").
<pre># enable TLS connections
TLSCertificateFile /etc/ssl/certs/wics-wildcard.crt
TLSCertificateKeyFile /etc/ssl/private/wics-wildcard.key

# enable TLS client authentication
TLSCACertificateFile /etc/ssl/certs/GlobalSign_Intermediate_Root_SHA256_G2.pem
TLSVerifyClient allow</pre></li>
<li>Update all your LDAP settings:
<pre># rm -rf /etc/openldap/slapd.d/*
# slaptest -f /etc/ldap/slapd.conf -F /etc/ldap/slapd.d/
# chown -R openldap:openldap /etc/ldap/slapd.d</pre></li>
<li>And last, ensure that LDAP will actually serve <code>ldaps://</code> by modifying the init script variables in <code>/etc/default/</code>:
<pre># vim /etc/default/slapd
...
SLAPD_SERVICES="ldap:/// ldapi:/// ldaps:///"
...</pre></li>
<li>Now you can restart the LDAP server:
<pre># service slapd restart</pre></li>
<li>And assuming this is successful, test to ensure LDAP is serving on port 636 for <code>ldaps://</code>:
<pre># netstat -ntaup
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:389 0.0.0.0:* LISTEN 22847/slapd
tcp 0 0 0.0.0.0:636 0.0.0.0:* LISTEN 22847/slapd </pre></li></ul>

==== Populating the Database ====

Now you'll need to start adding objects to the database. While we'll want to mostly do this programmatically, there are a few entries we'll need to bootstrap.

===== Root Entries =====

<ul>
<li>Start by creating a file [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/tree.ldif <code>tree.ldif</code>] to create a few necessary "roots" in our LDAP tree, with the contents:
<pre>dn: dc=wics,dc=uwaterloo,dc=ca
objectClass: dcObject
objectClass: organization
o: Women in Computer Science
dc: wics

dn: ou=People,dc=wics,dc=uwaterloo,dc=ca
objectClass: organizationalUnit
ou: People

dn: ou=Group,dc=wics,dc=uwaterloo,dc=ca
objectClass: organizationalUnit
ou: Group</pre></li>
<li>Now attempt an LDAP add, using the password you set earlier:
<pre># ldapadd -cxWD cn=root,dc=wics,dc=uwaterloo,dc=ca -f tree.ldif
Enter LDAP Password:
adding new entry "dc=wics,dc=uwaterloo,dc=ca"

adding new entry "ou=People,dc=wics,dc=uwaterloo,dc=ca"

adding new entry "ou=Group,dc=wics,dc=uwaterloo,dc=ca"</pre></li>
<li>Test that everything turned out okay, by performing a query of the entire database:
<pre># ldapsearch -x -h localhost
# extended LDIF
#
# LDAPv3
# base <dc=wics,dc=uwaterloo,dc=ca> (default) with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# wics.uwaterloo.ca
dn: dc=wics,dc=uwaterloo,dc=ca
objectClass: dcObject
objectClass: organization
o: Women in Computer Science
dc: wics

# People, wics.uwaterloo.ca
dn: ou=People,dc=wics,dc=uwaterloo,dc=ca
objectClass: organizationalUnit
ou: People

# Group, wics.uwaterloo.ca
dn: ou=Group,dc=wics,dc=uwaterloo,dc=ca
objectClass: organizationalUnit
ou: Group

# search result
search: 2
result: 0 Success

# numResponses: 4
# numEntries: 3</pre></li></ul>

===== Users and Groups =====

<ul>
<li>Next, add users to track the current GID and UID. This will save us from querying the entire database every time we make a new user or group. Create this file, [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/nextxid.ldif <code>nextxid.ldif</code>]:
<pre>dn: uid=nextuid,ou=People,dc=wics,dc=uwaterloo,dc=ca
cn: nextuid
objectClass: account
objectClass: posixAccount
objectClass: top
uidNumber: 20000
gidNumber: 20000
homeDirectory: /dev/null

dn: cn=nextgid,ou=Group,dc=wics,dc=uwaterloo,dc=ca
objectClass: group
objectClass: posixGroup
objectClass: top
gidNumber: 10000</pre>
You'll see here that our first GID is 10000 and our first UID is 20000.</li>
<li>Now add them, like you did with the roots of the tree:
<pre># ldapadd -cxWD cn=root,dc=wics,dc=uwaterloo,dc=ca -f nextxid.ldif
Enter LDAP Password:
adding new entry "uid=nextuid,ou=People,dc=wics,dc=uwaterloo,dc=ca"

adding new entry "cn=nextgid,ou=Group,dc=wics,dc=uwaterloo,dc=ca"</pre></li></ul>

===== Special <code>sudo</code> Entries =====

<ul>
<li>We also need to add a sudoers OU with a defaults object for default sudo settings. We also need entries for syscom, such that members of the syscom group can use sudo on all hosts, and for termcom, whose members can use sudo on only the office terminals. Call this one [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/sudoers.ldif <code>sudoers.ldif</code>]:
<pre>dn: ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca
objectClass: organizationalUnit
ou: SUDOers

dn: cn=defaults,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca
objectClass: top
objectClass: sudoRole
cn: defaults
sudoOption: !lecture
sudoOption: env_reset
sudoOption: listpw=never
sudoOption: mailto="wics-sys@lists.uwaterloo.ca"
sudoOption: shell_noargs

dn: cn=%syscom,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca
objectClass: top
objectClass: sudoRole
cn: %syscom
sudoUser: %syscom
sudoHost: ALL
sudoCommand: ALL
sudoRunAsUser: ALL

dn: cn=%termcom,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca
objectClass: top
objectClass: sudoRole
cn: %termcom
sudoUser: %termcom
sudoHost: honk
sudoHost: hiss
sudoHost: gosling
sudoCommand: ALL
sudoRunAsUser: ALL</pre></li>
<li>Now add them:
<pre># ldapadd -cxWD cn=root,dc=wics,dc=uwaterloo,dc=ca -f sudoers.ldif
Enter LDAP Password:
adding new entry "ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca"

adding new entry "cn=defaults,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca"

adding new entry "cn=%syscom,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca"

adding new entry "cn=%termcom,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca"</pre></li>
<li>Last, add some special local groups via [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/local-groups.ldif <code>local-groups.ldif</code>]:
<pre># ldapadd -cxWD cn=root,dc=wics,dc=uwaterloo,dc=ca -f local-groups.ldif</pre>
The local groups are special because they usually are present on all systems, but we want to be able to add users to them at the LDAP level. For instance, the audio group controls access to sound equipment, and the adm group controls log read access.</li>
<li>That's all the entries we have to add manually! Now we can use software for the rest. See [[weo|<code>ceo</code>]] for more details.</li></ul>

=== Querying LDAP ===

There are many tools available for issuing LDAP queries. Queries should be issued to <tt>ldap1.csclub.uwaterloo.ca</tt>. The search base you almost certainly want is <tt>dc=csclub,dc=uwaterloo,dc=ca</tt>. Read access is available without authentication; [[Kerberos]] is used to authenticate commands which require it.

Example:

ldapsearch -x -H ldap://ldap1.csclub.uwaterloo.ca -b dc=csclub,dc=uwaterloo,dc=ca uid=ctdalek

The <tt>-x</tt> option causes <tt>ldapsearch</tt> to switch to simple authentication rather than trying to authenticate via SASL (which will fail if you do not have a Kerberos ticket).

The University LDAP server (uwldap.uwaterloo.ca) can also be queried like this. Again, use "simple authentication" as read access is available (from on campus) without authentication. SASL authentication will fail without additional parameters.

Example:

ldapsearch -x -H ldap://uwldap.uwaterloo.ca -b dc=uwaterloo,dc=ca "cn=Prabhakar Ragde"

=== Replication ===

While <tt>ldap1.csclub.uwaterloo.ca</tt> ([[Machine_List#auth1|auth1]]) is the LDAP master, an up-to-date replica is available on <tt>ldap2.csclub.uwaterloo.ca</tt> ([[Machine_List#auth2|auth2]]).

In order to replicate changes from the master, the slave maintains an authenticated connection to the master which provides it with full read access to all changes.

Specifically, <tt>/etc/systemd/system/k5start-slapd.service</tt> maintains an active Kerberos ticket for <tt>ldap/auth2.csclub.uwaterloo.ca@CSCLUB.UWATERLOO.CA</tt> in <tt>/var/run/slapd/krb5cc</tt>. This is then used to authenticate the slave to the server, who maps this principal to <tt>cn=ldap-slave,dc=csclub,dc=uwaterloo,dc=ca</tt>, which in turn has full read privileges.

In the event of master failure, all hosts should fail LDAP reads seamlessly over to the slave.

[[Category:Software]]

=== Modifying LDAP entry ===

Editing entries can be easily done with <code>ldapvi</code>. First search for the entry using <code>ldapsearch</code> like above, and change <code>ldapsearch -x</code> to <code>ldapvi -Y GSSAPI</code> to make your edits.

Note that if your <tt>EDITOR</tt> enviroment is set to something not avaliable it will give out errors like

error (misc.c line 180): No such file or directory
editor died
error (ldapvi.c line 83): No such file or directory

This can be fixed by something like

EDITOR=vi ldapvi ******

==== Changing a user's username ====

Only a member of the Systems Committee can change a user's username. '''At all times, a user's username must match the user's username in WatIAM.'''

All changes to an account MUST be done in person so that identity can be confirmed. If a member cannot attend in person, then an alternate method of identity verification may be chosen by the Systems Administrator.

# Edit entries in LDAP (<code>ldapvi -Y GSSAPI</code>)
#* Find and replace the user's old username with the new one (<code>%s/$OLD/$NEW/g</code>)
# Change the user's Kerberos principal (on auth1 using <code>kadmin</code>, <code>renprinc $OLD $NEW</code>)
# Move the user's home directory (on phosphoric-acid, <code>mv /users/$OLD /users/$NEW</code>)
# Modify the user's ~/.forward file if their old username is in it.
# Change the user's csc-general (and csc-industry, if subscribed) email address for <code>$OLD@csclub.uwaterloo.ca</code> to <code>$NEW@csclub.uwaterloo.ca</code>
#* https://mailman.csclub.uwaterloo.ca/admin/csc-general
# If the user has vhosts on caffeine, update them to point to their new username

If the user's account has been around for a while, and they request it, forward email from their old username to their new one.

# Edit <code>/etc/aliases</code> on mail. <code>$OLD: $NEW</code>
# Run <code>newaliases</code>

MediaWiki

2025-09-10T00:38:29Z

K95ma: f

CSC's MediaWiki instance is located at <code>/var/lib/mediawiki</code> on [[Machine List#caffeine|caffeine]].

MediaWiki

2025-09-10T00:37:57Z

K95ma: +

CSC's MediaWiki instance is located at {{code|/var/lib/mediawiki}} on [[Machine List#caffeine|caffeine]].