CSCWiki - User contributions [en]

Ceo

2026-01-22T03:58:49Z

O32patel: fix Uploading Changes to Mirror

[[Image:Pyceo.png|thumb|300px|right|<tt>pyceo</tt>'s main menu screen]]

ceo is the CSC member creation and administration interface. It was originally written in perl by persons of mysterious-ness, was re-written in python by Michael Spang in early 2007, and re-written again (in Python) by Syscom in 2020-2021. The source-code for ceo can be found in git: [https://git.csclub.uwaterloo.ca/public/pyceo https://git.csclub.uwaterloo.ca/public/pyceo].

= Instructions/Usage =
ceo can be accessed by running the "ceo" command from a terminal, or terminal emulator.
By default, a curses-based menu interface is presented. Use the arrow keys to navigate;
on many screens, pressing a letter will select the next menu item beginning with that letter.

=== Command-line Mode ===
Run <tt>ceo --help</tt> to see a list of command-line utilities.

== Adding a New Member ==
After a new member has paid the membership fee and signed the Machine Usage Policy forms, a new member account is added to the CSC system by selecting "New Member" in ceo and following the on-screen instructions. The new member's username is to be identical to their WatIAM username, if applicable. For WatIAM users, the name and program fields will automatically be filled after a username is provided.

== Renewing/Extending a Membership ==
A membership can be renewed or extended by selecting "Renew Membership" in the ceo interface.

== Hosted Clubs ==
Clubs are hosted free of charge. To create a new club account use the "New Club" option in the ceo interface.

=== Club Representatives ===
At this time, there is no limit to the number of representatives a club may have, but representative accounts must be registered with the "New Club Rep" option, and renewed with the "Renew Club Rep" option.

=== Other Club Features ===
For access to features beyond basic hosting (ie, databases), one of the club representatives will need to email the Systems Committee to have this set up.

= raymo's guide on how to fix things after screwing up =

== Changing a member to a nonmember (club rep) and vice-versa ==

ssh hfcs
kinit # if you don't already have [[Kerberos#raymo's guide to keytabs|keytabs]] set up
ldapvi -Y GSSAPI
Use <code>/<username></code> to search for the user in vi and change <code>term</code> to <code>nonMemberTerm</code> (or vice-versa) for the relevant terms. When you're done deleting the file should no longer contain the username. Save and quit (<code>:wq</code>) and press <code>y</code> when prompted.

== Deleting a member ==

* '''RULE: Never do this without good reason. We should NEVER delete accounts or groups that have been used before.'''

If you accidentally created a club rep as a regular member instead, see the [previous section|Ceo#Changing a member to a nonmember (club rep) and vice-versa]. For another reason that doesn't break the '''RULE''' above, first follow the steps in the change membership section above, up to and including <code>ldapvi</code>, then delete both the user and group LDAP records. These are separated by blank lines. When you're done deleting the file should no longer contain the username. Save and quit as if changing membership. Then:
ssh auth1
sudo kadmin.local
delprinc <username>
ssh phosphoric acid
sudo rm -rfI /users/<username>
Unsubscribe the user from [https://mailman.csclub.uwaterloo.ca/postorius/lists/syscom.csclub.uwaterloo.ca/members/member/ csc-general on mailman]

= Feature Requests and Ideas =

* Create a graphical and/or online version of ceo
* Add new members to fuse and plugdev groups

= Contributing to CEO =

== Preliminary Steps ==
=== Generate a GPG Key ===
In order to sign the ceo packages you will need to generate yourself a GPG key if you do not already have one. Assuming you do not run

gpg --gen-key

Choose option (2) DSA (sign only). Choose no expiration when prompted and then your full name and email when asked. It will ask you to confirm the information and then for a passphrase.

=== Add Your Key To Mirror ===
ssh mirror.csclub.uwaterloo.ca
gpg --list-keys

Locate the 8-character id string. For example "16E37635" in
/users/m2ellis/.gnupg/pubring.gpg
---------------------------------
pub 1024D/'''16E37635''' 2010-08-19
uid Michael Ellis <m2ellis@csclub.uwaterloo.ca>

Now you must add this id into the file /srv/debian/conf/uploaders on mirror
sudo vim /srv/debian/conf/uploaders

Now in another terminal run
gpg --export --armor $KEYID

Now on mirror run
sudo -s
GNUPGHOME=/srv/debian/gpg gpg --import

Then paste the output from gpg --export --armor $KEYID and end with CTRL-D. It should give you a confirmation, example
gpg: key 16E37635: public key "Michael Ellis <m2ellis@csclub.uwaterloo.ca>" imported
gpg: Total number processed: 1
gpg: imported: 1

== Making Changes ==
The source-code for ceo can be found in git: [http://git.csclub.uwaterloo.ca/?p=public/pyceo.git;a=summary csclub:/users/git/public/pyceo.git]. To checkout the code run

git clone ~git/public/pyceo.git

When you are done making your change you need to update the changelog with dch. Assuming this is a minor incremental change run

dch -i

Add a description of your change and then save and quit. Once you are sure of your changes commit them to the git repository and push them (test them first!).

Make sure to set a distribution, like distribution UNRELEASED is NOT allowed. So change it to whatever distribution you're deploying to in the <code>debian/changelog</code> file

Then you'll need to make a tar.gz file<syntaxhighlight lang="bash">
tar czf ceo_<major>.<minor>.<patch>.orig.tar.gz pyceo/
</syntaxhighlight>To build the package run debuild

debuild

This will generate the *.deb files in the parent directory.

This may try to auto-sign with whatever your first GPG key is, you can safely exist and debsign yourself.

=== Uploading Changes to Mirror ===
After you make the package, you'll need to sign it, this can be via debsign. (You can find the .changes file in the parent directory)<syntaxhighlight lang="bash">
debsign -k [GPG Key ID] [package].changes

</syntaxhighlight>

In the directory containing the *.deb and *.changes files run<syntaxhighlight lang="bash">
dput -c /path/to/dput.cf ceo_<major>.<minor>.<patch>-<distribution>1_amd64.changes
</syntaxhighlight>Here is a sample dput.cf<syntaxhighlight lang="ini">
[DEFAULT]
default_host_main = debian.csclub

[debian.csclub]
fqdn = potassium-benzoate.csclub.uwaterloo.ca
method = scp
incoming = /srv/debian/incoming
run_install = 1
pre_upload_command = /bin/true
login = o32patel
</syntaxhighlight>Then ssh to mirror and run
reprepro --confdir /srv/debian/conf/ processincoming basic # use --ignore=longkeyid if you uploaded a long GPG id

The package should now be uploaded and you can update in the usual way with apt-get/aptitude.

== How to deploy CEO ==
Firstly, cry

Secondly, check the usual place for all the passwords, install ceo. Install ceod if needed, and edit the configs to make sure the correct servers are contacted for the relevant services. Like install ceod on the management node of cloudstack.

Then setup the kerberos tickets

Then go to `/etc/csc`, and fill in all the ceo.ini, and ceod.ini configs
[[Category:Software]]

SSL

2026-01-22T03:45:07Z

O32patel: add ranch ssl

== GlobalSign ==

The CSC currently has an SSL Certificate from GlobalSign for *.csclub.uwaterloo.ca provided at no cost to us through IST. GlobalSign likes to take a long time to respond to certificate signing requests (CSR) for wildcard certs, so our CSR really needs to be handed off to IST at least 2 weeks in advance. You can do it sooner – the certificate expiry date will be the old expiry date + 1 year (+ a bonus ) Having an invalid cert for any length of time leads to terrible breakage, followed by terrible workarounds and prolonged problems.

When the certificate is due to expire in a month or two, syscom should (but apparently doesn't always) get an email notification. This will include a renewal link. Otherwise, use the [https://uwaterloo.ca/information-systems-technology/about/organizational-structure/information-security-services/certificate-authority/globalsign-signed-x5093-certificates/self-service-globalsign-ssl-certificates IST-CA self service system]. Please keep a copy of the key, CSR and (once issued) certificate in <tt>/users/sysadmin/certs</tt>. The OpenSSL examples linked there are good to generate a 2048-bit RSA key and a corresponding CSR. It's probably a good idea to change the private key (as it's not that much effort anyways). Just sure your CSR is for <tt>*.csclub.uwaterloo.ca</tt>.

At the self-service portal, these options worked in 2013. If you need IST assistance, [mailto:ist-ca@uwaterloo.ca ist-ca@uwaterloo.ca] is the email address you should contact.
Products: OrganizationSSL
SSL Certificate Type: Wildcard SSL Certificate
Validity Period: 1 year
Are you switching from a Competitor? No, I am not switching
Are you renewing this Certificate? Yes (paste current certificate)
30-day bonus: Yes (why not?)
Add specific Subject Alternative Names (SANs): No (*.csclub.uwaterloo.ca automatically adds csclub.uwaterloo.ca as a SAN)
Enter Certificate Signing Request (CSR): Yes (paste CSR)
Contact Information:
First Name: Computer Science Club
Last Name: Systems Committee
Telephone: +1 519 888 4567 x33870
Email Address: syscom@csclub.uwaterloo.ca

=== Helpful links ===
* [https://support.globalsign.com/ssl/ssl-certificates-installation/generate-csr-openssl How to generate a new CSR and private key]
* [https://uwaterloo.atlassian.net/wiki/spaces/ISTKB/pages/262013183/How+to+obtain+a+new+GlobalSign+certificate+or+renew+an+existing+one How to obtain a new GlobalSign certificate or renew an existing one]
* [https://system.globalsign.com/bm/public/certificate/poporder.do?domain=PAR12271n5w6s27pvg8d92v4150t GlobalSign UWaterloo self-service page]
* [https://support.globalsign.com/ca-certificates/intermediate-certificates/organizationssl-intermediate-certificates GlobalSign intermediate certificate] (needed to create a certificate chain; see below)

=== OpenSSL cheat sheet ===
<ul>
<li>
Generate a new CSR and private key (do this in a new directory):
<pre>
openssl req -out csclub.uwaterloo.ca.csr -new -newkey rsa:2048 -keyout csclub.uwaterloo.ca.key -nodes
</pre>
Enter the following information at the prompts:
<pre>
Country Name (2 letter code) [AU]:CA
State or Province Name (full name) [Some-State]:Ontario
Locality Name (eg, city) []:Waterloo
Organization Name (eg, company) [Internet Widgits Pty Ltd]:University of Waterloo
Organizational Unit Name (eg, section) []:Computer Science Club
Common Name (e.g. server FQDN or YOUR name) []:*.csclub.uwaterloo.ca
Email Address []:systems-committee@csclub.uwaterloo.ca

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
</pre>
</li>
<li>
View the information inside a CSR:
<pre>
openssl req -noout -text -in csclub.uwaterloo.ca.csr
</pre>
</li>
<li>
View the information inside a private key:
<pre>
openssl pkey -noout -text -in csclub.uwaterloo.ca.key
</pre>
</li>
<li>
View information inside a certificate:
<pre>
openssl x509 -noout -text -in csclub.uwaterloo.ca.crt
</pre>
</li>
</ul>

=== csclub.cloud ===
Once a year, someone from IST will ask us to create a temporary TXT record for csclub.cloud to prove to GlobalSign that we own it. This must be created at the root of the domain. Since this zone is managed dynamically (via the acme.sh script on biloba, see below), we need to freeze the domain and update /var/lib/bind/db.csclub.cloud directly.

Once you're in the correct server (not Biloba). Here are the steps:
<ol>
<li>Run <code>rndc freeze csclub.cloud</code>.</li>
<li>
Open /var/lib/bind/db.csclub.cloud and add a new TXT record. It'll look something like
<pre>
TXT "_globalsign-domain-verification=blablabla"
</pre>
</li>
<li>
In the same file, make sure to also update the SOA serial number. It should generally be YYYYMMDDNN where NN is a monotonically increasing counter (YYYYMMDD is the current date).
</li>
<li>Run <code>rndc reload</code>.</li>
<li>
Run a DNS query to make sure you can see the TXT record:
<pre>
dig -t txt @dns1 csclub.cloud
dig -t txt @dns2 csclub.cloud
</pre>
</li>
<li>Email back the person from IST and let them know that we created the TXT record.</li>
<li>
Once the certificate has been renewed, delete the TXT record, update the SOA serial number, and run <code>rndc reload</code>.
</li>
<li>Run <code>rndc thaw csclub.cloud</code>.</li>
</ol>

== Certificate Files ==
Let's say you obtain a new certificate for *.csclub.uwaterloo.ca. Here are the files which should be stored in the certs folder:
<ul>
<li>csclub.uwaterloo.ca.key: private key created by openssl</li>
<li>csclub.uwaterloo.ca.csr: certificate signing request created by openssl</li>
<li>order: order number from GlobalSign</li>
<li>csclub.uwaterloo.ca.crt: certificate created by GlobalSign</li>
<li>globalsign-intermediate.crt: intermediate certificate from GlobalSign, obtainable from [https://support.globalsign.com/ca-certificates/intermediate-certificates/organizationssl-intermediate-certificates here]. As of this writing, we use the "OrganizationSSL SHA-256 R3 Intermediate Certificate". Just click the "View in Base64" button and copy the contents.
<ul>
<li>There is an alternative way to get the intermediate certificate: if you run <code>openssl x509 -noout -text -in csclub.uwaterloo.ca.crt</code>, under X509v3 extensions > Authority Information Access, there should be a field called "CA Issuers" which has a URL which looks like http://secure.globalsign.com/cacert/gsrsaovsslca2018.crt. You can download that file and convert it to PEM:
<pre>
wget https://secure.globalsign.com/cacert/gsrsaovsslca2018.crt
openssl x509 -inform der -in gsrsaovsslca2018.crt -out globalsign-intermediate.crt
rm gsrsaovsslca2018.crt
</pre>
</li>
</ul>
</li>
</ul><ul>
<li>csclub.uwaterloo.ca.chain: create this with the following command:
<pre>
cat csclub.uwaterloo.ca.crt globalsign-intermediate.crt > csclub.uwaterloo.ca.chain
</pre>
</li>
<li>csclub.uwaterloo.ca.pem: create this with the following command:
<pre>
cat csclub.uwaterloo.ca.key csclub.uwaterloo.ca.chain > csclub.uwaterloo.ca.pem
chmod 600 csclub.uwaterloo.ca.pem
</pre>
</li>
</ul>

== Certificate Locations ==

Keep a copy of newly generated certificates in /users/sysadmin/certs.

A list of places you'll need to put the new certificate to keep our services running. Private key (if applicable) should be kept next to the certificate with the extension .key.

* auth1:/etc/ssl/private/csclub-wildcard.crt (for slapd, make sure to <code>sudo service slapd restart</code>)
* auth2:/etc/ssl/private/csclub-wildcard.crt (for slapd, make sure to <code>sudo service slapd restart</code>)
* phosphoric-acid:/etc/ssl/private/csclub-wildcard-chain.crt
* ranch:/etc/ssl/private/csclub-wildcard-chain.crt (for ceod)
* coffee:/etc/ssl/private/csclub.uwaterloo.ca (for PostgreSQL and MariaDB)
* caffeine:/etc/ssl/private/csclub-wildcard.crt (for Apache)
* mailman:/etc/ssl/private/csclub-wildcard-chain.crt (for Apache)
* prometheus:/etc/ssl/private/csclub-wildcard-chain.crt (for Apache)
* potassium-benzoate:/etc/ssl/private/csclub-wildcard.crt (for nginx)
* mattermost:/etc/ssl/private/csclub-wildcard.crt (for nginx)
* chat:/etc/ssl/private/csclub-wildcard-chain.crt (for nginx)
* chamomile:/etc/ssl/private/cloud.csclub.uwaterloo.ca.chain.crt, /etc/ssl/private/csclub.cloud.chain, /etc/ssl/private/csclub.uwaterloo.ca.chain (for nginx)
* nextcloud (nspawn container inside guayusa): /etc/ssl/private/csclub.uwaterloo.ca.chain (for nginx)
* keycloak: /opt/keycloak/ssl (you can figure out the rest)
* <s>mail:/etc/ssl/private/csclub-wildcard.crt (for Apache, Postfix and Dovecot)</s> (UPDATE: we use certbot now for these)
* <s>bigbluebutton:/etc/nginx/ssl/csclub-wildcard-chain.crt (podman container on xylitol)</s> (Also certbot)
* <s>load-balancer-0(1|2):/etc/ssl/private/csclub.uwaterloo.ca (for haproxy)</s> [Down since 2020]

Some services (e.g. Dovecot, Postfix) prefer to have the certificate chain in one file. Concatenate the appropriate intermediate root to the end of the certificate and store this as csclub-wildcard-chain.crt.

=== More certificate locations ===
We have some SSL certificates which are not used by web servers, but still need to be renewed eventually.

==== Prometheus node exporter ====
All of our Prometheus node exporters are using mTLS via stunnel (every bare-metal host, as well as caffeine, coffee and mail, is running this exporter). The certificates (both client and server) are set to expire in September 2031; before then, create new keypairs in /opt/prometheus/tls, and deploy the new server.crt, node.crt and node.key to /etc/stunnel/tls on all machines. Restart prometheus and all of the node exporters.

==== ADFS ====
See [[ADFS]]. When the university's IdP certificate expires (October 2025), we can just download a new one and restart Apache; when our own certificate expires (July 2031), we need to submit a new form to IST (please do this before the cert expires).

==== Keycloak ====
See [[Keycloak]]. When the saml-passthrough certificate expires (January 2032), you need to create a new keypair in /srv/saml-passthrough on caffeine, and upload the new certificate into the Keycloak UI (IdP settings). When the Keycloak SP certificate expires (December 2031), make sure to create a new keypair and upload it to the Keycloak UI (Realm Settings).

== letsencrypt ==

We support letsencrypt for our virtual hosts with custom domains. We use the <tt>cerbot</tt> from debian repositories with a configuration file at <tt>/etc/letsencrypt/cli.ini</tt>, and a systemd timer to handle renewals.

The setup for a new domain is:

# Become <tt>certbot</tt> on caffine with <tt>sudo -u certbot bash</tt> or similar.
# Run <tt>certbot certonly -c /etc/letsencrypt/cli.ini -d DOMAIN --logs-dir /tmp</tt>. The logs-dir isn't important and is only needed for troubleshooting.
# Set up the Apache site configuration using the example below. (apache config is in /etc/apache2) Note the permanent redirect to https.
# Make sure to commit your changes when you're done.
# Reloading apache config is <tt>sudo systemctl reload apache2</tt>.

<VirtualHost *:80>
ServerName example.com
ServerAlias *.example.com
ServerAdmin example@csclub.uwaterloo.ca

#DocumentRoot /users/example/www/
Redirect permanent / https://example.com/

ErrorLog /var/log/apache2/example-error.log
CustomLog /var/log/apache2/example-access.log combined
</VirtualHost>

<VirtualHost csclub:443>
SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
SSLStrictSNIVHostCheck on

ServerName example.com
ServerAlias *.example.com
ServerAdmin example@csclub.uwaterloo.ca

DocumentRoot /users/example/www

ErrorLog /var/log/apache2/example-error.log
CustomLog /var/log/apache2/example-access.log combined
</VirtualHost>

== acme.sh ==
We are using [https://github.com/acmesh-official/acme.sh acme.sh] for provisioning SSL certificates for some of our *.csclub.cloud domains. It is currently set up under /root/.acme.sh on biloba.

NOTE: acme.sh has a cron job which automatically renews certificates before they expire and reloads NGINX, so you do not have to do anything after issuing and installing a certificate (i.e. "set-and-forget").

=== How to add a new SSL cert for a custom domain on CSC cloud ===
Note: you do not need to acquire a new cert if the requested domain is directly on csclub.cloud, e.g. app1.csclub.cloud. We can re-use our wildcard cert on csclub.cloud for that. However, if a user requests a multi-level domain on csclub.cloud, or a domain hosted on an external registrar, then you will need to create a new cert.

Let's say user <code>ctdalek</code> wants <code>mydomain.com</code> to point to a VM on CSC cloud.
 
TLDR:
<pre>
# Obtain the cert.
# If a subdomain was also requested, pass the -d option multiple times, e.g.
# `-d mydomain.com -d sub.mydomain.com`. Make sure the "main" domain is specified first.
acme.sh --issue -d mydomain.com -w /var/www

# Install the cert.
# If a subdomain was also requested, only specify the "main" domain.
acme.sh --install-cert -d mydomain.com \
--key-file /etc/nginx/ceod/member-ssl/mydomain.com.key \
--fullchain-file /etc/nginx/ceod/member-ssl/mydomain.com.chain \
--reloadcmd "/root/bin/reload-nginx.sh"

# Create a vhost file.
# Look at the other files in the same directory for inspiration.
# Make sure the file starts with the username and an underscore, e.g. "ctdalek_",
# because this is how ceod keeps track of the vhosts.
# Make sure to set the custom domain name(s) and paths to the SSL key/cert.
vim /etc/nginx/ceod/member-vhosts/ctdalek_mydomain.com

# Finally, reload NGINX on both biloba and chamomile. The /etc/nginx/ceod directory
# is shared between them.
/root/bin/reload-nginx.sh
</pre>

=== Installation ===
<pre>
cd /opt
git clone --depth 1 https://github.com/acmesh-official/acme.sh
cd acme.sh
./acme.sh --install -m syscom@csclub.uwaterloo.ca
. "/root/.acme.sh/acme.sh.env"
</pre>
Important: If invoking acme.sh from another program, it needs the environment variables set in acme.sh.env. Currently, that is just
<pre>
LE_WORKING_DIR="/root/.acme.sh"
</pre>

For testing purposes, make sure to use the Let's Encrypt test server:
<pre>
acme.sh --set-default-ca --server letsencrypt_test
</pre>

=== NGINX setup ===
<pre>
mkdir -p /var/www/.well-known/acme-challenge
</pre>
Add the following snippet to your default NGINX file (e.g. /etc/nginx/sites-enabled/default):
<pre>
# For Let's Encrypt
location /.well-known/acme-challenge/ {
alias /var/www/.well-known/acme-challenge/;
}
</pre>

Now assuming that biloba has the IP address for *.csclub.cloud, you can test that everything is working:
<pre>
acme.sh --issue -d app.merenber.csclub.cloud -w /var/www
</pre>
To install a certificate after it's been issued:
<pre>
acme.sh --install-cert -d app.merenber.csclub.cloud \
--key-file /etc/nginx/ceod/member-ssl/app.merenber.csclub.cloud.key \
--fullchain-file /etc/nginx/ceod/member-ssl/app.merenber.csclub.cloud.chain \
--reloadcmd "/root/bin/reload-nginx.sh"
</pre>
At this point, you should add your NGINX vhost file which uses that SSL certificate.
 
To remove a certificate:
<pre>
acme.sh --remove -d app.merenber.csclub.cloud
rm -r /root/.acme.sh/app.merenber.csclub.cloud
rm /etc/nginx/ceod/member-ssl/app.merenber.csclub.cloud.chain
rm /etc/nginx/ceod/member-ssl/app.merenber.csclub.cloud.key
</pre>
Don't forget to remove the NGINX vhost file too.

Once you think you're ready, use a real ACME provider, e.g.
<pre>
acme.sh --set-default-ca --server letsencrypt
</pre>

Since we have a [https://zerossl.com ZeroSSL] account, and ZeroSSL has no rate limit, we are going to use that instead:
<pre>
acme.sh --register-account --server zerossl \
--eab-kid xxxxxxxxxxxx \
--eab-hmac-key xxxxxxxxx
acme.sh --set-default-ca --server zerossl
</pre>

=== DNS challenge ===
To obtain a wildcard certificate (e.g. *.k8s.csclub.cloud), you will need to perform the DNS-01 challenge. We are going to use nsupdate to interact with our BIND9 server on dns1.

On dns1, run:
<pre>
tsig-keygen csc-cloud
</pre>
Paste the output into the appropriate section in /etc/bind/named.conf.local. Also paste it into a file somewhere on biloba, e.g. /etc/csc/csc-cloud-tsig.key.

Add the following to the csclub.cloud zone block:
<pre>
allow-update {
!{
!127.0.0.1;
!::1;
!129.97.134.0/24;
!2620:101:f000:4901::/64;
any;
};
key csc-cloud;
};
</pre>
(We're basically trying to restrict updates to the given IP ranges. See https://serverfault.com/a/417229.)

The 'bind' user can't write to files under /etc/bind, so we're going to move our zone file to /var/lib/bind instead.
Comment out 'file "/etc/bind/db.csclub.cloud";' from named.conf.local and add this line below it:
<pre>
file "/var/lib/bind/db.csclub.cloud";
</pre>
Then run:
<pre>
cp /etc/bind/db.csclub.cloud /var/lib/bind/db.csclub.cloud
chown bind:bind /var/lib/bind/db.csclub.cloud
rndc reload
</pre>

On biloba, check that everything's working:
<pre>
nsupdate -k /etc/csc/csc-cloud-tsig.key -v <<EOF
update add test.csclub.cloud 300 A 0.0.0.0
send
EOF
</pre>
Use a tool such as <code>dig</code> to make sure that the update was successful.
If it worked, you can delete the record:
<pre>
nsupdate -k /etc/csc/csc-cloud-tsig.key -v <<EOF
delete test.csclub.cloud
send
EOF
</pre>
Now we are ready to actually perform the challenge with acme.sh:
<pre>
export NSUPDATE_SERVER="dns1.csclub.uwaterloo.ca"
export NSUPDATE_KEY="/etc/csc/csc-cloud-tsig.key"
acme.sh --issue --dns dns_nsupdate -d 'k8s.csclub.cloud' -d '*.k8s.csclub.cloud'
</pre>
(If something goes wrong, use the <code>--debug</code> flag.)

If all went well, just install the certificate as usual:
<pre>
acme.sh --install-cert -d k8s.csclub.cloud \
--key-file /etc/nginx/ceod/syscom-ssl/k8s.csclub.cloud.key \
--fullchain-file /etc/nginx/ceod/syscom-ssl/k8s.csclub.cloud.chain \
--reloadcmd 'systemctl reload nginx'
</pre>

Acronyms

2025-12-13T15:33:35Z

O32patel: cat

Over the F22, W23, S23, W24, and S24 terms, many members within the office speculated what CSC stands for. Here are their postulations (in alphabetical order):

* Caffeinated Students Club
* Calculator Slaves Club
* Calcium-deficient Students Club
* Camera-Shy Club
* Canadian Sussy Club
* Can't Sleep Club
* Can't Stop CSC
* Can't Succeed Club
* Card Skills Club
* Cascading Style Club
* Celeste Strawberry Collection
* Celeste Speedrunning Club
* Celeste Streaming Club
* Chair Sleeping Club
* Charge Smartphone Club
* Chess Studies Club
* Chopin Studies Club
* City Skylines Club
* Clown Syndicate Club
* Coffee Supply Club
* Collecting Strawberries Club
* Come Seethe and Cope
* Comic Sans Club
* Committee Sacking Club
* Communal Shoe Club
* Communal Shower Club
* Complimentary Slackness Condition
* Compulsory Sadness Condition
* Compulsory Sleep Club
* Computer Stop-working Club
* Confidential Secrets Club
* Connection Sucks Club
* Constant Sacrificing Club
* Cooking Spaghetti Club
* Cool Shit Club
* Cool Students Club
* Copyright Safeguarding Club
* Cosecant
* Co-op Suckers Club
* Crazy Superglue Club
* Creature Saving Club
* Crying Students Club
* CSC Scribbling Club
* CSC Sussy Committee
* CSC Systems Committee
* Cutting Sticker Club
* Culinary Students Club
* Capital & Securities Club
* CSC Safety Committee
* Committees Sleeping Club
* Consistently Sleeping club
* Cartwheel Stumbling Club
* Camera Success Club
* in-Class Sleeping Club
* Cooked Servers Club
* Crumbling servers club
* Coup SciSoc Club
* Club of Siracha Commandeering
* Cycling Students Club
* Counter Strike club
* Cancerous Spreadsheets Club
* Camera Sawing Club
* Cannabinoid Sales Club
* Cat smuggling Club
* Collecting Sponsorships Club
* concerning security club
* Ctf Team Stealing Club
* Computer Shopping Club
* Consuming Steroids Club
* cuddly shark club
* Communist State Club
* Come Stab Club
* Can't Stream Club
* Cat Supremacy Club

SSL

2025-11-11T17:00:56Z

O32patel: Cert location list update

== GlobalSign ==

The CSC currently has an SSL Certificate from GlobalSign for *.csclub.uwaterloo.ca provided at no cost to us through IST. GlobalSign likes to take a long time to respond to certificate signing requests (CSR) for wildcard certs, so our CSR really needs to be handed off to IST at least 2 weeks in advance. You can do it sooner – the certificate expiry date will be the old expiry date + 1 year (+ a bonus ) Having an invalid cert for any length of time leads to terrible breakage, followed by terrible workarounds and prolonged problems.

When the certificate is due to expire in a month or two, syscom should (but apparently doesn't always) get an email notification. This will include a renewal link. Otherwise, use the [https://uwaterloo.ca/information-systems-technology/about/organizational-structure/information-security-services/certificate-authority/globalsign-signed-x5093-certificates/self-service-globalsign-ssl-certificates IST-CA self service system]. Please keep a copy of the key, CSR and (once issued) certificate in <tt>/users/sysadmin/certs</tt>. The OpenSSL examples linked there are good to generate a 2048-bit RSA key and a corresponding CSR. It's probably a good idea to change the private key (as it's not that much effort anyways). Just sure your CSR is for <tt>*.csclub.uwaterloo.ca</tt>.

At the self-service portal, these options worked in 2013. If you need IST assistance, [mailto:ist-ca@uwaterloo.ca ist-ca@uwaterloo.ca] is the email address you should contact.
Products: OrganizationSSL
SSL Certificate Type: Wildcard SSL Certificate
Validity Period: 1 year
Are you switching from a Competitor? No, I am not switching
Are you renewing this Certificate? Yes (paste current certificate)
30-day bonus: Yes (why not?)
Add specific Subject Alternative Names (SANs): No (*.csclub.uwaterloo.ca automatically adds csclub.uwaterloo.ca as a SAN)
Enter Certificate Signing Request (CSR): Yes (paste CSR)
Contact Information:
First Name: Computer Science Club
Last Name: Systems Committee
Telephone: +1 519 888 4567 x33870
Email Address: syscom@csclub.uwaterloo.ca

=== Helpful links ===
* [https://support.globalsign.com/ssl/ssl-certificates-installation/generate-csr-openssl How to generate a new CSR and private key]
* [https://uwaterloo.atlassian.net/wiki/spaces/ISTKB/pages/262013183/How+to+obtain+a+new+GlobalSign+certificate+or+renew+an+existing+one How to obtain a new GlobalSign certificate or renew an existing one]
* [https://system.globalsign.com/bm/public/certificate/poporder.do?domain=PAR12271n5w6s27pvg8d92v4150t GlobalSign UWaterloo self-service page]
* [https://support.globalsign.com/ca-certificates/intermediate-certificates/organizationssl-intermediate-certificates GlobalSign intermediate certificate] (needed to create a certificate chain; see below)

=== OpenSSL cheat sheet ===
<ul>
<li>
Generate a new CSR and private key (do this in a new directory):
<pre>
openssl req -out csclub.uwaterloo.ca.csr -new -newkey rsa:2048 -keyout csclub.uwaterloo.ca.key -nodes
</pre>
Enter the following information at the prompts:
<pre>
Country Name (2 letter code) [AU]:CA
State or Province Name (full name) [Some-State]:Ontario
Locality Name (eg, city) []:Waterloo
Organization Name (eg, company) [Internet Widgits Pty Ltd]:University of Waterloo
Organizational Unit Name (eg, section) []:Computer Science Club
Common Name (e.g. server FQDN or YOUR name) []:*.csclub.uwaterloo.ca
Email Address []:systems-committee@csclub.uwaterloo.ca

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
</pre>
</li>
<li>
View the information inside a CSR:
<pre>
openssl req -noout -text -in csclub.uwaterloo.ca.csr
</pre>
</li>
<li>
View the information inside a private key:
<pre>
openssl pkey -noout -text -in csclub.uwaterloo.ca.key
</pre>
</li>
<li>
View information inside a certificate:
<pre>
openssl x509 -noout -text -in csclub.uwaterloo.ca.crt
</pre>
</li>
</ul>

=== csclub.cloud ===
Once a year, someone from IST will ask us to create a temporary TXT record for csclub.cloud to prove to GlobalSign that we own it. This must be created at the root of the domain. Since this zone is managed dynamically (via the acme.sh script on biloba, see below), we need to freeze the domain and update /var/lib/bind/db.csclub.cloud directly.

Once you're in the correct server (not Biloba). Here are the steps:
<ol>
<li>Run <code>rndc freeze csclub.cloud</code>.</li>
<li>
Open /var/lib/bind/db.csclub.cloud and add a new TXT record. It'll look something like
<pre>
TXT "_globalsign-domain-verification=blablabla"
</pre>
</li>
<li>
In the same file, make sure to also update the SOA serial number. It should generally be YYYYMMDDNN where NN is a monotonically increasing counter (YYYYMMDD is the current date).
</li>
<li>Run <code>rndc reload</code>.</li>
<li>
Run a DNS query to make sure you can see the TXT record:
<pre>
dig -t txt @dns1 csclub.cloud
dig -t txt @dns2 csclub.cloud
</pre>
</li>
<li>Email back the person from IST and let them know that we created the TXT record.</li>
<li>
Once the certificate has been renewed, delete the TXT record, update the SOA serial number, and run <code>rndc reload</code>.
</li>
<li>Run <code>rndc thaw csclub.cloud</code>.</li>
</ol>

== Certificate Files ==
Let's say you obtain a new certificate for *.csclub.uwaterloo.ca. Here are the files which should be stored in the certs folder:
<ul>
<li>csclub.uwaterloo.ca.key: private key created by openssl</li>
<li>csclub.uwaterloo.ca.csr: certificate signing request created by openssl</li>
<li>order: order number from GlobalSign</li>
<li>csclub.uwaterloo.ca.crt: certificate created by GlobalSign</li>
<li>globalsign-intermediate.crt: intermediate certificate from GlobalSign, obtainable from [https://support.globalsign.com/ca-certificates/intermediate-certificates/organizationssl-intermediate-certificates here]. As of this writing, we use the "OrganizationSSL SHA-256 R3 Intermediate Certificate". Just click the "View in Base64" button and copy the contents.
<ul>
<li>There is an alternative way to get the intermediate certificate: if you run <code>openssl x509 -noout -text -in csclub.uwaterloo.ca.crt</code>, under X509v3 extensions > Authority Information Access, there should be a field called "CA Issuers" which has a URL which looks like http://secure.globalsign.com/cacert/gsrsaovsslca2018.crt. You can download that file and convert it to PEM:
<pre>
wget https://secure.globalsign.com/cacert/gsrsaovsslca2018.crt
openssl x509 -inform der -in gsrsaovsslca2018.crt -out globalsign-intermediate.crt
rm gsrsaovsslca2018.crt
</pre>
</li>
</ul>
</li>
</ul><ul>
<li>csclub.uwaterloo.ca.chain: create this with the following command:
<pre>
cat csclub.uwaterloo.ca.crt globalsign-intermediate.crt > csclub.uwaterloo.ca.chain
</pre>
</li>
<li>csclub.uwaterloo.ca.pem: create this with the following command:
<pre>
cat csclub.uwaterloo.ca.key csclub.uwaterloo.ca.chain > csclub.uwaterloo.ca.pem
chmod 600 csclub.uwaterloo.ca.pem
</pre>
</li>
</ul>

== Certificate Locations ==

Keep a copy of newly generated certificates in /users/sysadmin/certs.

A list of places you'll need to put the new certificate to keep our services running. Private key (if applicable) should be kept next to the certificate with the extension .key.

* auth1:/etc/ssl/private/csclub-wildcard.crt (for slapd, make sure to <code>sudo service slapd restart</code>)
* auth2:/etc/ssl/private/csclub-wildcard.crt (for slapd, make sure to <code>sudo service slapd restart</code>)
* phosphoric-acid:/etc/ssl/private/csclub-wildcard-chain.crt (for ceod)
* coffee:/etc/ssl/private/csclub.uwaterloo.ca (for PostgreSQL and MariaDB)
* caffeine:/etc/ssl/private/csclub-wildcard.crt (for Apache)
* mailman:/etc/ssl/private/csclub-wildcard-chain.crt (for Apache)
* prometheus:/etc/ssl/private/csclub-wildcard-chain.crt (for Apache)
* potassium-benzoate:/etc/ssl/private/csclub-wildcard.crt (for nginx)
* mattermost:/etc/ssl/private/csclub-wildcard.crt (for nginx)
* chat:/etc/ssl/private/csclub-wildcard-chain.crt (for nginx)
* chamomile:/etc/ssl/private/cloud.csclub.uwaterloo.ca.chain.crt, /etc/ssl/private/csclub.cloud.chain, /etc/ssl/private/csclub.uwaterloo.ca.chain (for nginx)
* nextcloud (nspawn container inside guayusa): /etc/ssl/private/csclub.uwaterloo.ca.chain (for nginx)
* <s>mail:/etc/ssl/private/csclub-wildcard.crt (for Apache, Postfix and Dovecot)</s> (UPDATE: we use certbot now for these)
* <s>bigbluebutton:/etc/nginx/ssl/csclub-wildcard-chain.crt (podman container on xylitol)</s> (Also certbot)
* <s>load-balancer-0(1|2):/etc/ssl/private/csclub.uwaterloo.ca (for haproxy)</s> [Down since 2020]

Some services (e.g. Dovecot, Postfix) prefer to have the certificate chain in one file. Concatenate the appropriate intermediate root to the end of the certificate and store this as csclub-wildcard-chain.crt.

=== More certificate locations ===
We have some SSL certificates which are not used by web servers, but still need to be renewed eventually.

==== Prometheus node exporter ====
All of our Prometheus node exporters are using mTLS via stunnel (every bare-metal host, as well as caffeine, coffee and mail, is running this exporter). The certificates (both client and server) are set to expire in September 2031; before then, create new keypairs in /opt/prometheus/tls, and deploy the new server.crt, node.crt and node.key to /etc/stunnel/tls on all machines. Restart prometheus and all of the node exporters.

==== ADFS ====
See [[ADFS]]. When the university's IdP certificate expires (October 2025), we can just download a new one and restart Apache; when our own certificate expires (July 2031), we need to submit a new form to IST (please do this before the cert expires).

==== Keycloak ====
See [[Keycloak]]. When the saml-passthrough certificate expires (January 2032), you need to create a new keypair in /srv/saml-passthrough on caffeine, and upload the new certificate into the Keycloak UI (IdP settings). When the Keycloak SP certificate expires (December 2031), make sure to create a new keypair and upload it to the Keycloak UI (Realm Settings).

== letsencrypt ==

We support letsencrypt for our virtual hosts with custom domains. We use the <tt>cerbot</tt> from debian repositories with a configuration file at <tt>/etc/letsencrypt/cli.ini</tt>, and a systemd timer to handle renewals.

The setup for a new domain is:

# Become <tt>certbot</tt> on caffine with <tt>sudo -u certbot bash</tt> or similar.
# Run <tt>certbot certonly -c /etc/letsencrypt/cli.ini -d DOMAIN --logs-dir /tmp</tt>. The logs-dir isn't important and is only needed for troubleshooting.
# Set up the Apache site configuration using the example below. (apache config is in /etc/apache2) Note the permanent redirect to https.
# Make sure to commit your changes when you're done.
# Reloading apache config is <tt>sudo systemctl reload apache2</tt>.

<VirtualHost *:80>
ServerName example.com
ServerAlias *.example.com
ServerAdmin example@csclub.uwaterloo.ca

#DocumentRoot /users/example/www/
Redirect permanent / https://example.com/

ErrorLog /var/log/apache2/example-error.log
CustomLog /var/log/apache2/example-access.log combined
</VirtualHost>

<VirtualHost csclub:443>
SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
SSLStrictSNIVHostCheck on

ServerName example.com
ServerAlias *.example.com
ServerAdmin example@csclub.uwaterloo.ca

DocumentRoot /users/example/www

ErrorLog /var/log/apache2/example-error.log
CustomLog /var/log/apache2/example-access.log combined
</VirtualHost>

== acme.sh ==
We are using [https://github.com/acmesh-official/acme.sh acme.sh] for provisioning SSL certificates for some of our *.csclub.cloud domains. It is currently set up under /root/.acme.sh on biloba.

NOTE: acme.sh has a cron job which automatically renews certificates before they expire and reloads NGINX, so you do not have to do anything after issuing and installing a certificate (i.e. "set-and-forget").

=== How to add a new SSL cert for a custom domain on CSC cloud ===
Note: you do not need to acquire a new cert if the requested domain is directly on csclub.cloud, e.g. app1.csclub.cloud. We can re-use our wildcard cert on csclub.cloud for that. However, if a user requests a multi-level domain on csclub.cloud, or a domain hosted on an external registrar, then you will need to create a new cert.

Let's say user <code>ctdalek</code> wants <code>mydomain.com</code> to point to a VM on CSC cloud.
 
TLDR:
<pre>
# Obtain the cert.
# If a subdomain was also requested, pass the -d option multiple times, e.g.
# `-d mydomain.com -d sub.mydomain.com`. Make sure the "main" domain is specified first.
acme.sh --issue -d mydomain.com -w /var/www

# Install the cert.
# If a subdomain was also requested, only specify the "main" domain.
acme.sh --install-cert -d mydomain.com \
--key-file /etc/nginx/ceod/member-ssl/mydomain.com.key \
--fullchain-file /etc/nginx/ceod/member-ssl/mydomain.com.chain \
--reloadcmd "/root/bin/reload-nginx.sh"

# Create a vhost file.
# Look at the other files in the same directory for inspiration.
# Make sure the file starts with the username and an underscore, e.g. "ctdalek_",
# because this is how ceod keeps track of the vhosts.
# Make sure to set the custom domain name(s) and paths to the SSL key/cert.
vim /etc/nginx/ceod/member-vhosts/ctdalek_mydomain.com

# Finally, reload NGINX on both biloba and chamomile. The /etc/nginx/ceod directory
# is shared between them.
/root/bin/reload-nginx.sh
</pre>

=== Installation ===
<pre>
cd /opt
git clone --depth 1 https://github.com/acmesh-official/acme.sh
cd acme.sh
./acme.sh --install -m syscom@csclub.uwaterloo.ca
. "/root/.acme.sh/acme.sh.env"
</pre>
Important: If invoking acme.sh from another program, it needs the environment variables set in acme.sh.env. Currently, that is just
<pre>
LE_WORKING_DIR="/root/.acme.sh"
</pre>

For testing purposes, make sure to use the Let's Encrypt test server:
<pre>
acme.sh --set-default-ca --server letsencrypt_test
</pre>

=== NGINX setup ===
<pre>
mkdir -p /var/www/.well-known/acme-challenge
</pre>
Add the following snippet to your default NGINX file (e.g. /etc/nginx/sites-enabled/default):
<pre>
# For Let's Encrypt
location /.well-known/acme-challenge/ {
alias /var/www/.well-known/acme-challenge/;
}
</pre>

Now assuming that biloba has the IP address for *.csclub.cloud, you can test that everything is working:
<pre>
acme.sh --issue -d app.merenber.csclub.cloud -w /var/www
</pre>
To install a certificate after it's been issued:
<pre>
acme.sh --install-cert -d app.merenber.csclub.cloud \
--key-file /etc/nginx/ceod/member-ssl/app.merenber.csclub.cloud.key \
--fullchain-file /etc/nginx/ceod/member-ssl/app.merenber.csclub.cloud.chain \
--reloadcmd "/root/bin/reload-nginx.sh"
</pre>
At this point, you should add your NGINX vhost file which uses that SSL certificate.
 
To remove a certificate:
<pre>
acme.sh --remove -d app.merenber.csclub.cloud
rm -r /root/.acme.sh/app.merenber.csclub.cloud
rm /etc/nginx/ceod/member-ssl/app.merenber.csclub.cloud.chain
rm /etc/nginx/ceod/member-ssl/app.merenber.csclub.cloud.key
</pre>
Don't forget to remove the NGINX vhost file too.

Once you think you're ready, use a real ACME provider, e.g.
<pre>
acme.sh --set-default-ca --server letsencrypt
</pre>

Since we have a [https://zerossl.com ZeroSSL] account, and ZeroSSL has no rate limit, we are going to use that instead:
<pre>
acme.sh --register-account --server zerossl \
--eab-kid xxxxxxxxxxxx \
--eab-hmac-key xxxxxxxxx
acme.sh --set-default-ca --server zerossl
</pre>

=== DNS challenge ===
To obtain a wildcard certificate (e.g. *.k8s.csclub.cloud), you will need to perform the DNS-01 challenge. We are going to use nsupdate to interact with our BIND9 server on dns1.

On dns1, run:
<pre>
tsig-keygen csc-cloud
</pre>
Paste the output into the appropriate section in /etc/bind/named.conf.local. Also paste it into a file somewhere on biloba, e.g. /etc/csc/csc-cloud-tsig.key.

Add the following to the csclub.cloud zone block:
<pre>
allow-update {
!{
!127.0.0.1;
!::1;
!129.97.134.0/24;
!2620:101:f000:4901::/64;
any;
};
key csc-cloud;
};
</pre>
(We're basically trying to restrict updates to the given IP ranges. See https://serverfault.com/a/417229.)

The 'bind' user can't write to files under /etc/bind, so we're going to move our zone file to /var/lib/bind instead.
Comment out 'file "/etc/bind/db.csclub.cloud";' from named.conf.local and add this line below it:
<pre>
file "/var/lib/bind/db.csclub.cloud";
</pre>
Then run:
<pre>
cp /etc/bind/db.csclub.cloud /var/lib/bind/db.csclub.cloud
chown bind:bind /var/lib/bind/db.csclub.cloud
rndc reload
</pre>

On biloba, check that everything's working:
<pre>
nsupdate -k /etc/csc/csc-cloud-tsig.key -v <<EOF
update add test.csclub.cloud 300 A 0.0.0.0
send
EOF
</pre>
Use a tool such as <code>dig</code> to make sure that the update was successful.
If it worked, you can delete the record:
<pre>
nsupdate -k /etc/csc/csc-cloud-tsig.key -v <<EOF
delete test.csclub.cloud
send
EOF
</pre>
Now we are ready to actually perform the challenge with acme.sh:
<pre>
export NSUPDATE_SERVER="dns1.csclub.uwaterloo.ca"
export NSUPDATE_KEY="/etc/csc/csc-cloud-tsig.key"
acme.sh --issue --dns dns_nsupdate -d 'k8s.csclub.cloud' -d '*.k8s.csclub.cloud'
</pre>
(If something goes wrong, use the <code>--debug</code> flag.)

If all went well, just install the certificate as usual:
<pre>
acme.sh --install-cert -d k8s.csclub.cloud \
--key-file /etc/nginx/ceod/syscom-ssl/k8s.csclub.cloud.key \
--fullchain-file /etc/nginx/ceod/syscom-ssl/k8s.csclub.cloud.chain \
--reloadcmd 'systemctl reload nginx'
</pre>

SSL

2025-11-09T16:53:16Z

O32patel: remove old cert locations (rip biloba and citric-acid)

== GlobalSign ==

The CSC currently has an SSL Certificate from GlobalSign for *.csclub.uwaterloo.ca provided at no cost to us through IST. GlobalSign likes to take a long time to respond to certificate signing requests (CSR) for wildcard certs, so our CSR really needs to be handed off to IST at least 2 weeks in advance. You can do it sooner – the certificate expiry date will be the old expiry date + 1 year (+ a bonus ) Having an invalid cert for any length of time leads to terrible breakage, followed by terrible workarounds and prolonged problems.

When the certificate is due to expire in a month or two, syscom should (but apparently doesn't always) get an email notification. This will include a renewal link. Otherwise, use the [https://uwaterloo.ca/information-systems-technology/about/organizational-structure/information-security-services/certificate-authority/globalsign-signed-x5093-certificates/self-service-globalsign-ssl-certificates IST-CA self service system]. Please keep a copy of the key, CSR and (once issued) certificate in <tt>/users/sysadmin/certs</tt>. The OpenSSL examples linked there are good to generate a 2048-bit RSA key and a corresponding CSR. It's probably a good idea to change the private key (as it's not that much effort anyways). Just sure your CSR is for <tt>*.csclub.uwaterloo.ca</tt>.

At the self-service portal, these options worked in 2013. If you need IST assistance, [mailto:ist-ca@uwaterloo.ca ist-ca@uwaterloo.ca] is the email address you should contact.
Products: OrganizationSSL
SSL Certificate Type: Wildcard SSL Certificate
Validity Period: 1 year
Are you switching from a Competitor? No, I am not switching
Are you renewing this Certificate? Yes (paste current certificate)
30-day bonus: Yes (why not?)
Add specific Subject Alternative Names (SANs): No (*.csclub.uwaterloo.ca automatically adds csclub.uwaterloo.ca as a SAN)
Enter Certificate Signing Request (CSR): Yes (paste CSR)
Contact Information:
First Name: Computer Science Club
Last Name: Systems Committee
Telephone: +1 519 888 4567 x33870
Email Address: syscom@csclub.uwaterloo.ca

=== Helpful links ===
* [https://support.globalsign.com/ssl/ssl-certificates-installation/generate-csr-openssl How to generate a new CSR and private key]
* [https://uwaterloo.atlassian.net/wiki/spaces/ISTKB/pages/262013183/How+to+obtain+a+new+GlobalSign+certificate+or+renew+an+existing+one How to obtain a new GlobalSign certificate or renew an existing one]
* [https://system.globalsign.com/bm/public/certificate/poporder.do?domain=PAR12271n5w6s27pvg8d92v4150t GlobalSign UWaterloo self-service page]
* [https://support.globalsign.com/ca-certificates/intermediate-certificates/organizationssl-intermediate-certificates GlobalSign intermediate certificate] (needed to create a certificate chain; see below)

=== OpenSSL cheat sheet ===
<ul>
<li>
Generate a new CSR and private key (do this in a new directory):
<pre>
openssl req -out csclub.uwaterloo.ca.csr -new -newkey rsa:2048 -keyout csclub.uwaterloo.ca.key -nodes
</pre>
Enter the following information at the prompts:
<pre>
Country Name (2 letter code) [AU]:CA
State or Province Name (full name) [Some-State]:Ontario
Locality Name (eg, city) []:Waterloo
Organization Name (eg, company) [Internet Widgits Pty Ltd]:University of Waterloo
Organizational Unit Name (eg, section) []:Computer Science Club
Common Name (e.g. server FQDN or YOUR name) []:*.csclub.uwaterloo.ca
Email Address []:systems-committee@csclub.uwaterloo.ca

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
</pre>
</li>
<li>
View the information inside a CSR:
<pre>
openssl req -noout -text -in csclub.uwaterloo.ca.csr
</pre>
</li>
<li>
View the information inside a private key:
<pre>
openssl pkey -noout -text -in csclub.uwaterloo.ca.key
</pre>
</li>
<li>
View information inside a certificate:
<pre>
openssl x509 -noout -text -in csclub.uwaterloo.ca.crt
</pre>
</li>
</ul>

=== csclub.cloud ===
Once a year, someone from IST will ask us to create a temporary TXT record for csclub.cloud to prove to GlobalSign that we own it. This must be created at the root of the domain. Since this zone is managed dynamically (via the acme.sh script on biloba, see below), we need to freeze the domain and update /var/lib/bind/db.csclub.cloud directly.

Once you're in the correct server (not Biloba). Here are the steps:
<ol>
<li>Run <code>rndc freeze csclub.cloud</code>.</li>
<li>
Open /var/lib/bind/db.csclub.cloud and add a new TXT record. It'll look something like
<pre>
TXT "_globalsign-domain-verification=blablabla"
</pre>
</li>
<li>
In the same file, make sure to also update the SOA serial number. It should generally be YYYYMMDDNN where NN is a monotonically increasing counter (YYYYMMDD is the current date).
</li>
<li>Run <code>rndc reload</code>.</li>
<li>
Run a DNS query to make sure you can see the TXT record:
<pre>
dig -t txt @dns1 csclub.cloud
dig -t txt @dns2 csclub.cloud
</pre>
</li>
<li>Email back the person from IST and let them know that we created the TXT record.</li>
<li>
Once the certificate has been renewed, delete the TXT record, update the SOA serial number, and run <code>rndc reload</code>.
</li>
<li>Run <code>rndc thaw csclub.cloud</code>.</li>
</ol>

== Certificate Files ==
Let's say you obtain a new certificate for *.csclub.uwaterloo.ca. Here are the files which should be stored in the certs folder:
<ul>
<li>csclub.uwaterloo.ca.key: private key created by openssl</li>
<li>csclub.uwaterloo.ca.csr: certificate signing request created by openssl</li>
<li>order: order number from GlobalSign</li>
<li>csclub.uwaterloo.ca.crt: certificate created by GlobalSign</li>
<li>globalsign-intermediate.crt: intermediate certificate from GlobalSign, obtainable from [https://support.globalsign.com/ca-certificates/intermediate-certificates/organizationssl-intermediate-certificates here]. As of this writing, we use the "OrganizationSSL SHA-256 R3 Intermediate Certificate". Just click the "View in Base64" button and copy the contents.
<ul>
<li>There is an alternative way to get the intermediate certificate: if you run <code>openssl x509 -noout -text -in csclub.uwaterloo.ca.crt</code>, under X509v3 extensions > Authority Information Access, there should be a field called "CA Issuers" which has a URL which looks like http://secure.globalsign.com/cacert/gsrsaovsslca2018.crt. You can download that file and convert it to PEM:
<pre>
wget https://secure.globalsign.com/cacert/gsrsaovsslca2018.crt
openssl x509 -inform der -in gsrsaovsslca2018.crt -out globalsign-intermediate.crt
rm gsrsaovsslca2018.crt
</pre>
</li>
</ul>
</li>
</ul><ul>
<li>csclub.uwaterloo.ca.chain: create this with the following command:
<pre>
cat csclub.uwaterloo.ca.crt globalsign-intermediate.crt > csclub.uwaterloo.ca.chain
</pre>
</li>
<li>csclub.uwaterloo.ca.pem: create this with the following command:
<pre>
cat csclub.uwaterloo.ca.key csclub.uwaterloo.ca.chain > csclub.uwaterloo.ca.pem
chmod 600 csclub.uwaterloo.ca.pem
</pre>
</li>
</ul>

== Certificate Locations ==

Keep a copy of newly generated certificates in /users/sysadmin/certs.

A list of places you'll need to put the new certificate to keep our services running. Private key (if applicable) should be kept next to the certificate with the extension .key.

* caffeine:/etc/ssl/private/csclub-wildcard.crt (for Apache)
* coffee:/etc/ssl/private/csclub.uwaterloo.ca (for PostgreSQL and MariaDB)
* <s>mail:/etc/ssl/private/csclub-wildcard.crt (for Apache, Postfix and Dovecot)</s> (UPDATE: we use certbot now for these)
* mailman:/etc/ssl/private/csclub-wildcard-chain.crt (for Apache)
* potassium-benzoate:/etc/ssl/private/csclub-wildcard.crt (for nginx)
* phosphoric-acid:/etc/ssl/private/csclub-wildcard-chain.crt (for ceod)
* auth1:/etc/ssl/private/csclub-wildcard.crt (for slapd, make sure to <code>sudo service slapd restart</code>)
* auth2:/etc/ssl/private/csclub-wildcard.crt (for slapd, make sure to <code>sudo service slapd restart</code>)
* mattermost:/etc/ssl/private/csclub-wildcard.crt (for nginx)
* load-balancer-0(1|2):/etc/ssl/private/csclub.uwaterloo.ca (for haproxy) [temporarily down 2020]
* chat:/etc/ssl/private/csclub-wildcard-chain.crt (for nginx)
* prometheus:/etc/ssl/private/csclub-wildcard-chain.crt (for Apache)
* bigbluebutton:/etc/nginx/ssl/csclub-wildcard-chain.crt (podman container on xylitol)
* chamomile:/etc/ssl/private/cloud.csclub.uwaterloo.ca.chain.crt, /etc/ssl/private/csclub.cloud.chain, /etc/ssl/private/csclub.uwaterloo.ca.chain (for nginx)
* nextcloud (nspawn container inside guayusa): /etc/ssl/private/csclub.uwaterloo.ca.chain (for nginx)

Some services (e.g. Dovecot, Postfix) prefer to have the certificate chain in one file. Concatenate the appropriate intermediate root to the end of the certificate and store this as csclub-wildcard-chain.crt.

=== More certificate locations ===
We have some SSL certificates which are not used by web servers, but still need to be renewed eventually.

==== Prometheus node exporter ====
All of our Prometheus node exporters are using mTLS via stunnel (every bare-metal host, as well as caffeine, coffee and mail, is running this exporter). The certificates (both client and server) are set to expire in September 2031; before then, create new keypairs in /opt/prometheus/tls, and deploy the new server.crt, node.crt and node.key to /etc/stunnel/tls on all machines. Restart prometheus and all of the node exporters.

==== ADFS ====
See [[ADFS]]. When the university's IdP certificate expires (October 2025), we can just download a new one and restart Apache; when our own certificate expires (July 2031), we need to submit a new form to IST (please do this before the cert expires).

==== Keycloak ====
See [[Keycloak]]. When the saml-passthrough certificate expires (January 2032), you need to create a new keypair in /srv/saml-passthrough on caffeine, and upload the new certificate into the Keycloak UI (IdP settings). When the Keycloak SP certificate expires (December 2031), make sure to create a new keypair and upload it to the Keycloak UI (Realm Settings).

== letsencrypt ==

We support letsencrypt for our virtual hosts with custom domains. We use the <tt>cerbot</tt> from debian repositories with a configuration file at <tt>/etc/letsencrypt/cli.ini</tt>, and a systemd timer to handle renewals.

The setup for a new domain is:

# Become <tt>certbot</tt> on caffine with <tt>sudo -u certbot bash</tt> or similar.
# Run <tt>certbot certonly -c /etc/letsencrypt/cli.ini -d DOMAIN --logs-dir /tmp</tt>. The logs-dir isn't important and is only needed for troubleshooting.
# Set up the Apache site configuration using the example below. (apache config is in /etc/apache2) Note the permanent redirect to https.
# Make sure to commit your changes when you're done.
# Reloading apache config is <tt>sudo systemctl reload apache2</tt>.

<VirtualHost *:80>
ServerName example.com
ServerAlias *.example.com
ServerAdmin example@csclub.uwaterloo.ca

#DocumentRoot /users/example/www/
Redirect permanent / https://example.com/

ErrorLog /var/log/apache2/example-error.log
CustomLog /var/log/apache2/example-access.log combined
</VirtualHost>

<VirtualHost csclub:443>
SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
SSLStrictSNIVHostCheck on

ServerName example.com
ServerAlias *.example.com
ServerAdmin example@csclub.uwaterloo.ca

DocumentRoot /users/example/www

ErrorLog /var/log/apache2/example-error.log
CustomLog /var/log/apache2/example-access.log combined
</VirtualHost>

== acme.sh ==
We are using [https://github.com/acmesh-official/acme.sh acme.sh] for provisioning SSL certificates for some of our *.csclub.cloud domains. It is currently set up under /root/.acme.sh on biloba.

NOTE: acme.sh has a cron job which automatically renews certificates before they expire and reloads NGINX, so you do not have to do anything after issuing and installing a certificate (i.e. "set-and-forget").

=== How to add a new SSL cert for a custom domain on CSC cloud ===
Note: you do not need to acquire a new cert if the requested domain is directly on csclub.cloud, e.g. app1.csclub.cloud. We can re-use our wildcard cert on csclub.cloud for that. However, if a user requests a multi-level domain on csclub.cloud, or a domain hosted on an external registrar, then you will need to create a new cert.

Let's say user <code>ctdalek</code> wants <code>mydomain.com</code> to point to a VM on CSC cloud.
 
TLDR:
<pre>
# Obtain the cert.
# If a subdomain was also requested, pass the -d option multiple times, e.g.
# `-d mydomain.com -d sub.mydomain.com`. Make sure the "main" domain is specified first.
acme.sh --issue -d mydomain.com -w /var/www

# Install the cert.
# If a subdomain was also requested, only specify the "main" domain.
acme.sh --install-cert -d mydomain.com \
--key-file /etc/nginx/ceod/member-ssl/mydomain.com.key \
--fullchain-file /etc/nginx/ceod/member-ssl/mydomain.com.chain \
--reloadcmd "/root/bin/reload-nginx.sh"

# Create a vhost file.
# Look at the other files in the same directory for inspiration.
# Make sure the file starts with the username and an underscore, e.g. "ctdalek_",
# because this is how ceod keeps track of the vhosts.
# Make sure to set the custom domain name(s) and paths to the SSL key/cert.
vim /etc/nginx/ceod/member-vhosts/ctdalek_mydomain.com

# Finally, reload NGINX on both biloba and chamomile. The /etc/nginx/ceod directory
# is shared between them.
/root/bin/reload-nginx.sh
</pre>

=== Installation ===
<pre>
cd /opt
git clone --depth 1 https://github.com/acmesh-official/acme.sh
cd acme.sh
./acme.sh --install -m syscom@csclub.uwaterloo.ca
. "/root/.acme.sh/acme.sh.env"
</pre>
Important: If invoking acme.sh from another program, it needs the environment variables set in acme.sh.env. Currently, that is just
<pre>
LE_WORKING_DIR="/root/.acme.sh"
</pre>

For testing purposes, make sure to use the Let's Encrypt test server:
<pre>
acme.sh --set-default-ca --server letsencrypt_test
</pre>

=== NGINX setup ===
<pre>
mkdir -p /var/www/.well-known/acme-challenge
</pre>
Add the following snippet to your default NGINX file (e.g. /etc/nginx/sites-enabled/default):
<pre>
# For Let's Encrypt
location /.well-known/acme-challenge/ {
alias /var/www/.well-known/acme-challenge/;
}
</pre>

Now assuming that biloba has the IP address for *.csclub.cloud, you can test that everything is working:
<pre>
acme.sh --issue -d app.merenber.csclub.cloud -w /var/www
</pre>
To install a certificate after it's been issued:
<pre>
acme.sh --install-cert -d app.merenber.csclub.cloud \
--key-file /etc/nginx/ceod/member-ssl/app.merenber.csclub.cloud.key \
--fullchain-file /etc/nginx/ceod/member-ssl/app.merenber.csclub.cloud.chain \
--reloadcmd "/root/bin/reload-nginx.sh"
</pre>
At this point, you should add your NGINX vhost file which uses that SSL certificate.
 
To remove a certificate:
<pre>
acme.sh --remove -d app.merenber.csclub.cloud
rm -r /root/.acme.sh/app.merenber.csclub.cloud
rm /etc/nginx/ceod/member-ssl/app.merenber.csclub.cloud.chain
rm /etc/nginx/ceod/member-ssl/app.merenber.csclub.cloud.key
</pre>
Don't forget to remove the NGINX vhost file too.

Once you think you're ready, use a real ACME provider, e.g.
<pre>
acme.sh --set-default-ca --server letsencrypt
</pre>

Since we have a [https://zerossl.com ZeroSSL] account, and ZeroSSL has no rate limit, we are going to use that instead:
<pre>
acme.sh --register-account --server zerossl \
--eab-kid xxxxxxxxxxxx \
--eab-hmac-key xxxxxxxxx
acme.sh --set-default-ca --server zerossl
</pre>

=== DNS challenge ===
To obtain a wildcard certificate (e.g. *.k8s.csclub.cloud), you will need to perform the DNS-01 challenge. We are going to use nsupdate to interact with our BIND9 server on dns1.

On dns1, run:
<pre>
tsig-keygen csc-cloud
</pre>
Paste the output into the appropriate section in /etc/bind/named.conf.local. Also paste it into a file somewhere on biloba, e.g. /etc/csc/csc-cloud-tsig.key.

Add the following to the csclub.cloud zone block:
<pre>
allow-update {
!{
!127.0.0.1;
!::1;
!129.97.134.0/24;
!2620:101:f000:4901::/64;
any;
};
key csc-cloud;
};
</pre>
(We're basically trying to restrict updates to the given IP ranges. See https://serverfault.com/a/417229.)

The 'bind' user can't write to files under /etc/bind, so we're going to move our zone file to /var/lib/bind instead.
Comment out 'file "/etc/bind/db.csclub.cloud";' from named.conf.local and add this line below it:
<pre>
file "/var/lib/bind/db.csclub.cloud";
</pre>
Then run:
<pre>
cp /etc/bind/db.csclub.cloud /var/lib/bind/db.csclub.cloud
chown bind:bind /var/lib/bind/db.csclub.cloud
rndc reload
</pre>

On biloba, check that everything's working:
<pre>
nsupdate -k /etc/csc/csc-cloud-tsig.key -v <<EOF
update add test.csclub.cloud 300 A 0.0.0.0
send
EOF
</pre>
Use a tool such as <code>dig</code> to make sure that the update was successful.
If it worked, you can delete the record:
<pre>
nsupdate -k /etc/csc/csc-cloud-tsig.key -v <<EOF
delete test.csclub.cloud
send
EOF
</pre>
Now we are ready to actually perform the challenge with acme.sh:
<pre>
export NSUPDATE_SERVER="dns1.csclub.uwaterloo.ca"
export NSUPDATE_KEY="/etc/csc/csc-cloud-tsig.key"
acme.sh --issue --dns dns_nsupdate -d 'k8s.csclub.cloud' -d '*.k8s.csclub.cloud'
</pre>
(If something goes wrong, use the <code>--debug</code> flag.)

If all went well, just install the certificate as usual:
<pre>
acme.sh --install-cert -d k8s.csclub.cloud \
--key-file /etc/nginx/ceod/syscom-ssl/k8s.csclub.cloud.key \
--fullchain-file /etc/nginx/ceod/syscom-ssl/k8s.csclub.cloud.chain \
--reloadcmd 'systemctl reload nginx'
</pre>

Meeting:Termcom/Saturday 29 June 2025

2025-06-29T19:00:09Z

O32patel:

- Will setup Proxmox on Citric-acid

- Will setup a different method of VHosts:

Kubernetes-backed ingress:

1. Create a self-managed k8s cluster in the virtualized environment (eg. rke2)

2. Configure ingress (eg. Istio)

3. Configure a software-based load balancer (eg. kube-vip, metallb) or an external load balancer (eg. from CSCF/IST)

4. Configure https://gateway-api.sigs.k8s.io/

CEO Integration: Write a custom kubernetes operator, or use a dynamic client for modifying vhost/stream configs.

vhost -> HTTPRoute, GRPCRoute
stream -> TLSRoute

- Taking out yerba-mate, and corn-syrup
- Centralising all our scripts
- Updating ansible scripts
- Standardising our configs

- Rewrite pyceo as Go - Ohm, Nathan, Jenny, Dundee

Debian 12 Transition

2025-04-13T01:21:25Z

O32patel: Add a Kerberos section for potential libk5crypto3 issues

== Upgrade steps ==
1. Create the /etc/apt/keyrings folder.

2. Download the CSC keyring into it:
<pre>
wget -O /etc/apt/keyrings/csclub.gpg http://debian.csclub.uwaterloo.ca/csclub.gpg
</pre>

3. Make sure that the CSC keyring is the only one in /etc/apt/trusted.gpg:
<pre>
gpg --no-options --show-keys /etc/apt/trusted.gpg
</pre>

4. Delete /etc/apt/trusted.gpg and its backup file:
<pre>
rm -f /etc/apt/trusted.gpg /etc/apt/trusted.gpg~
</pre>

5. Replace the old-style /etc/apt/sources.list and /etc/apt/sources.list.d/*.list files with the new Deb822 "sources" style (see /etc/apt/sources.list.d/*.sources on sorbitol; don't copy the one for the Dell repo). Add a helpful note in /etc/apt/sources.list for other syscom members:
<pre>
# See /etc/apt/sources.list.d/*.sources
</pre>

6. apt update && apt dist-upgrade

7. apt autoremove --purge

8. During the upgrade, accept the new configuration files (choose the 'Y' option)
for the following files:
* /etc/fail2ban/fail2ban.conf
* /etc/fail2ban/jail.conf
* /etc/fail2ban/filter.d/sshd.conf
Everything else should keep the old file.

9. Copy the following files from sorbitol:
* /etc/fail2ban/fail2ban.local
* /etc/fail2ban/jail.local
* /etc/fail2ban/filter.d/sshd.local
Then restart fail2ban.

10. If the 'ntp' package is installed, purge it and install systemd-timesyncd instead. Enable the systemd-timesyncd service and copy /etc/systemd/timesyncd.conf.d/csclub.conf from sorbitol. Start the service and make sure it's working.

11. Get rid of python2 if it's still installed:
<pre>
apt purge python2.7-minimal
apt autoremove --purge
</pre>

=== Kerberos ===
If Kerberos and consequently, the NFS mount breaks, see [[New CSC Machine#apt|the new machine apt guide]] and make sure that <code>/etc/apt/preferences.d/99-csclub</code> exists and run <code>apt install --reinstall libk5crypto3</code>.

Here are some places to look and sample errors for the <code>libk5crypto3</code> issue:

* <code>mount.nfs: access denied by server while mounting fs00[...]:/users</code>
* <code>journalctl -u rpc-svcgssd.service</code>: <code>ERROR: GSS-API: [...] GSS_S_FAILURE [...] - No key table entry found matching nfs/[...]</code>
* (Auth1) <code>journalctl -u krb5-kdc.service -r</code>: <code>BAD_ENCRYPTION_TYPE: authtime 0 [...] KDC has no support for encryption type</code>

== Pending machines ==
Machines/containers that have yet to upgrade to Debian 12. Remove entry when upgrade is done.

=== Syscom Only ===

* xylitol: later?
** xylitol runs all sort of critical services
* phosphoric-acid: later?
** phosphoric-acid runs web
* yerba-mate
* cobalamin
* potassium-benzoate: ugh ubuntu and we can't shut down the mirror

=== Cloud ===

Everything. We will need to wait until ceph supports bookworm.

=== Containers ===

* on xylitol
** auth1
** mail
** chat
* on phosphoric-acid
** caffeine
** coffee
** prometheus

LDAP

2025-01-20T02:30:04Z

O32patel:

We use [http://www.openldap.org/ OpenLDAP] for directory services. Our primary LDAP server is [[Machine_List#auth1|auth1]] and our secondary LDAP server is [[Machine_List#auth2|auth2]].

=== ehashman's Guide to Setting up OpenLDAP on Debian ===

Welcome to my nightmare.

==== What is LDAP? ====

<blockquote>'''LDAP:''' Lightweight Directory Access Protocol

An open, vendor-neutral, industry standard application protocol for accessing and maintaining distributed directory information services over an Internet Protocol (IP) network. — [https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol Wikipedia: LDAP]
</blockquote>
In this case, "directory" refers to the user directory, like on an old-school Rolodex. Many groups use LDAP to maintain their user directory, including the University (the "WatIAM" identity management system), the Computer Science Club, and even the UW Amateur Radio Club.

This is a guide documenting how to set up LDAP on a Debian Linux system.

==== First steps ====

<ul>
<li>Ensure that openldap is installed on the machine:
<pre># apt-get install slapd ldap-utils</pre></li>
<li>Debian will do a lot of magic and set up a skeleton LDAP server and get it running. We need to configure that further.</li>
<li>Let's set up logging before we forget. Create the following files in <code>/var/log</code>:
<pre># mkdir /var/log/ldap
# touch /var/log/ldap.log</pre></li>
<li>Set ownership correctly:
<pre># chown openldap:openldap /var/log/ldap</pre></li>
<li>Set up rsyslog to dump the LDAP logs into <code>/var/log/ldap.log</code> by adding the following lines:
<pre># vim /etc/rsyslog.conf
...
# Grab ldap logs, don't duplicate in syslog
local4.* /var/log/ldap.log</pre></li>
<li>Set up log rotation for these by creating the file [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/logrotate.d.ldap <code>/etc/logrotate.d/ldap</code>] with the following contents:
<pre>/var/log/ldap/*log {
weekly
missingok
rotate 1000
compress
delaycompress
notifempty
create 0640 openldap adm
postrotate
if [ -f /var/run/slapd/slapd.pid ]; then
/etc/init.d/slapd restart >/dev/null 2>&1
fi
endscript
}

/var/log/ldap.log {
weekly
missingok
rotate 24
compress
delaycompress
notifempty
}</pre></li>
<li>As of OpenLDAP 2.4, it doesn't actually create a config file for us. Apparently, this is a "feature": LDAP maintainers think we should want to set this up via dynamic queries. We don't, so the first thing we need is our [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/slapd.conf <code>slapd.conf</code>] file.</li></ul>

===== Building <code>slapd.conf</code> from scratch =====

<ul>
<li>Get a copy to work with:
<pre># scp uid@auth1.csclub.uwaterloo.ca:/etc/ldap/slapd.conf /etc/ldap/ ## you need CSC root for this</pre></li>
<li>You'll want to comment out the TLS lines, and anything referring to Kerberos and access for now. You'll also want to comment out lines specifically referring to syscom and office staff.</li>
<li>Make sure you remove the reference to <code>nonMemberTerm</code> as an index, as we're going to remove this field.</li>
<li>You'll also need to generate a root password for the LDAP to bootstrap auth, like so:
<pre># slappasswd
New password:
Re-enter new password:
{SSHA}longhash</pre></li>
<li>Add this line below <code>rootdn</code> in the <code>slapd.conf</code>:
<pre>rootpw {SSHA}longhash</pre></li>
<li>Now we want to edit all instances of "csclub" to be "wics" instead, e.g.:
<pre>suffix "dc=wics,dc=uwaterloo,dc=ca"
rootdn "cn=root,dc=wics,dc=uwaterloo,dc=ca"</pre></li>
<li>Next, we need to grab all the relevant schemas:
<pre>scp -r uid@auth1.csclub.uwaterloo.ca:/etc/ldap/schema/ /tmp/schemas</pre></li>
<li>Use the include directives to help you find the ones you need. I noticed we were missing <code>sudo.schema</code>, <code>csc.schema</code>, and <code>rfc2307bis.schema</code>.</li>
<li>Open up the [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/csc.schema <code>csc.schema</code>] for editing; we're not using it verbatim. Remove the attributes <code>studentid</code> and <code>nonMemberTerm</code> and the objectclass <code>club</code>. Also make sure you change the OID so we don't clash with the CSC. Because we didn't want to go through the process of requesting a [http://pen.iana.org/pen/PenApplication.page PEN number], we chose arbitrarily to use 26338, which belongs to IWICS Inc.</li>
<li>We also need to can the auto-generated config files, so do that:
<pre># rm -rf /etc/openldap/slapd.d/*</pre></li>
<li>Also nuke the auto-generated database:
<pre># rm /var/lib/ldap/__db.*</pre></li>
<li>Configure the database:
<pre># cp /usr/share/slapd/DB_CONFIG /var/lib/ldap/
# chown openldap:openldap /var/lib/ldap/DB_CONFIG </pre></li>
<li>Now we can generate the new configuration files:
<pre># slaptest -f /etc/ldap/slapd.conf -F /etc/ldap/slapd.d/</pre></li>
<li>And ensure that the permissions are all set correctly, lest this break something:
<pre># chown -R openldap:openldap /etc/ldap/slapd.d</pre></li>
<li>If at this point you get a nasty error, such as
<pre>5657d4db hdb_db_open: database "dc=wics,dc=uwaterloo,dc=ca": db_open(/var/lib/ldap/id2entry.bdb) failed: No such file or directory (2).
5657d4db backend_startup_one (type=hdb, suffix="dc=wics,dc=uwaterloo,dc=ca"): bi_db_open failed! (2)
slap_startup failed (test would succeed using the -u switch)</pre>
Just try restarting slapd, and see if that fixes the problem:
<pre># service slapd stop
# service slapd start</pre></li>
<li>Congratulations! Your LDAP service is now configured and running.</li></ul>

==== Getting TLS Up and Running ====

<ul>
<li>Now that we have our LDAP service, we'll want to be able to serve encrypted traffic. This is especially important for any remote access, since binding to LDAP (i.e. sending it a password for auth) occurs over plaintext, and we don't want to leak our admin password.</li>
<li>Our first step is to copy our SSL certificates into the correct places. Public ones go into <code>/etc/ssl/certs/</code> and private ones go into <code>/etc/ssl/private/</code>.</li>
<li>Since the LDAP daemon needs to be able to read our private cert, we need to grant LDAP access to the private folder:
<pre># chgrp openldap /etc/ssl/private
# chmod g+x /etc/ssl/private</pre></li>
<li>Next, uncomment the TLS-related settings in <code>slapd.conf</code>. These are <code>TLSCertificateFile</code> (the public cert), <code>TLSCertificateKeyFile</code> (the private key), <code>TLSCACertificateFile</code> (the intermediate CA cert), and <code>TLSVerifyClient</code> (set to "allow").
<pre># enable TLS connections
TLSCertificateFile /etc/ssl/certs/wics-wildcard.crt
TLSCertificateKeyFile /etc/ssl/private/wics-wildcard.key

# enable TLS client authentication
TLSCACertificateFile /etc/ssl/certs/GlobalSign_Intermediate_Root_SHA256_G2.pem
TLSVerifyClient allow</pre></li>
<li>Update all your LDAP settings:
<pre># rm -rf /etc/openldap/slapd.d/*
# slaptest -f /etc/ldap/slapd.conf -F /etc/ldap/slapd.d/
# chown -R openldap:openldap /etc/ldap/slapd.d</pre></li>
<li>And last, ensure that LDAP will actually serve <code>ldaps://</code> by modifying the init script variables in <code>/etc/default/</code>:
<pre># vim /etc/default/slapd
...
SLAPD_SERVICES="ldap:/// ldapi:/// ldaps:///"
...</pre></li>
<li>Now you can restart the LDAP server:
<pre># service slapd restart</pre></li>
<li>And assuming this is successful, test to ensure LDAP is serving on port 636 for <code>ldaps://</code>:
<pre># netstat -ntaup
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:389 0.0.0.0:* LISTEN 22847/slapd
tcp 0 0 0.0.0.0:636 0.0.0.0:* LISTEN 22847/slapd </pre></li></ul>

==== Populating the Database ====

Now you'll need to start adding objects to the database. While we'll want to mostly do this programmatically, there are a few entries we'll need to bootstrap.

===== Root Entries =====

<ul>
<li>Start by creating a file [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/tree.ldif <code>tree.ldif</code>] to create a few necessary "roots" in our LDAP tree, with the contents:
<pre>dn: dc=wics,dc=uwaterloo,dc=ca
objectClass: dcObject
objectClass: organization
o: Women in Computer Science
dc: wics

dn: ou=People,dc=wics,dc=uwaterloo,dc=ca
objectClass: organizationalUnit
ou: People

dn: ou=Group,dc=wics,dc=uwaterloo,dc=ca
objectClass: organizationalUnit
ou: Group</pre></li>
<li>Now attempt an LDAP add, using the password you set earlier:
<pre># ldapadd -cxWD cn=root,dc=wics,dc=uwaterloo,dc=ca -f tree.ldif
Enter LDAP Password:
adding new entry "dc=wics,dc=uwaterloo,dc=ca"

adding new entry "ou=People,dc=wics,dc=uwaterloo,dc=ca"

adding new entry "ou=Group,dc=wics,dc=uwaterloo,dc=ca"</pre></li>
<li>Test that everything turned out okay, by performing a query of the entire database:
<pre># ldapsearch -x -h localhost
# extended LDIF
#
# LDAPv3
# base <dc=wics,dc=uwaterloo,dc=ca> (default) with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# wics.uwaterloo.ca
dn: dc=wics,dc=uwaterloo,dc=ca
objectClass: dcObject
objectClass: organization
o: Women in Computer Science
dc: wics

# People, wics.uwaterloo.ca
dn: ou=People,dc=wics,dc=uwaterloo,dc=ca
objectClass: organizationalUnit
ou: People

# Group, wics.uwaterloo.ca
dn: ou=Group,dc=wics,dc=uwaterloo,dc=ca
objectClass: organizationalUnit
ou: Group

# search result
search: 2
result: 0 Success

# numResponses: 4
# numEntries: 3</pre></li></ul>

===== Users and Groups =====

<ul>
<li>Next, add users to track the current GID and UID. This will save us from querying the entire database every time we make a new user or group. Create this file, [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/nextxid.ldif <code>nextxid.ldif</code>]:
<pre>dn: uid=nextuid,ou=People,dc=wics,dc=uwaterloo,dc=ca
cn: nextuid
objectClass: account
objectClass: posixAccount
objectClass: top
uidNumber: 20000
gidNumber: 20000
homeDirectory: /dev/null

dn: cn=nextgid,ou=Group,dc=wics,dc=uwaterloo,dc=ca
objectClass: group
objectClass: posixGroup
objectClass: top
gidNumber: 10000</pre>
You'll see here that our first GID is 10000 and our first UID is 20000.</li>
<li>Now add them, like you did with the roots of the tree:
<pre># ldapadd -cxWD cn=root,dc=wics,dc=uwaterloo,dc=ca -f nextxid.ldif
Enter LDAP Password:
adding new entry "uid=nextuid,ou=People,dc=wics,dc=uwaterloo,dc=ca"

adding new entry "cn=nextgid,ou=Group,dc=wics,dc=uwaterloo,dc=ca"</pre></li></ul>

===== Special <code>sudo</code> Entries =====

<ul>
<li>We also need to add a sudoers OU with a defaults object for default sudo settings. We also need entries for syscom, such that members of the syscom group can use sudo on all hosts, and for termcom, whose members can use sudo on only the office terminals. Call this one [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/sudoers.ldif <code>sudoers.ldif</code>]:
<pre>dn: ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca
objectClass: organizationalUnit
ou: SUDOers

dn: cn=defaults,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca
objectClass: top
objectClass: sudoRole
cn: defaults
sudoOption: !lecture
sudoOption: env_reset
sudoOption: listpw=never
sudoOption: mailto="wics-sys@lists.uwaterloo.ca"
sudoOption: shell_noargs

dn: cn=%syscom,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca
objectClass: top
objectClass: sudoRole
cn: %syscom
sudoUser: %syscom
sudoHost: ALL
sudoCommand: ALL
sudoRunAsUser: ALL

dn: cn=%termcom,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca
objectClass: top
objectClass: sudoRole
cn: %termcom
sudoUser: %termcom
sudoHost: honk
sudoHost: hiss
sudoHost: gosling
sudoCommand: ALL
sudoRunAsUser: ALL</pre></li>
<li>Now add them:
<pre># ldapadd -cxWD cn=root,dc=wics,dc=uwaterloo,dc=ca -f sudoers.ldif
Enter LDAP Password:
adding new entry "ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca"

adding new entry "cn=defaults,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca"

adding new entry "cn=%syscom,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca"

adding new entry "cn=%termcom,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca"</pre></li>
<li>Last, add some special local groups via [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/local-groups.ldif <code>local-groups.ldif</code>]:
<pre># ldapadd -cxWD cn=root,dc=wics,dc=uwaterloo,dc=ca -f local-groups.ldif</pre>
The local groups are special because they usually are present on all systems, but we want to be able to add users to them at the LDAP level. For instance, the audio group controls access to sound equipment, and the adm group controls log read access.</li>
<li>That's all the entries we have to add manually! Now we can use software for the rest. See [[weo|<code>ceo</code>]] for more details.</li></ul>

=== Querying LDAP ===

There are many tools available for issuing LDAP queries. Queries should be issued to <tt>ldap1.csclub.uwaterloo.ca</tt>. The search base you almost certainly want is <tt>dc=csclub,dc=uwaterloo,dc=ca</tt>. Read access is available without authentication; [[Kerberos]] is used to authenticate commands which require it.

Example:

ldapsearch -x -h ldap1.csclub.uwaterloo.ca -b dc=csclub,dc=uwaterloo,dc=ca uid=ctdalek

The <tt>-x</tt> option causes <tt>ldapsearch</tt> to switch to simple authentication rather than trying to authenticate via SASL (which will fail if you do not have a Kerberos ticket).

The University LDAP server (uwldap.uwaterloo.ca) can also be queried like this. Again, use "simple authentication" as read access is available (from on campus) without authentication. SASL authentication will fail without additional parameters.

Example:

ldapsearch -x -h uwldap.uwaterloo.ca -b dc=uwaterloo,dc=ca "cn=Prabhakar Ragde"

=== Replication ===

While <tt>ldap1.csclub.uwaterloo.ca</tt> ([[Machine_List#auth1|auth1]]) is the LDAP master, an up-to-date replica is available on <tt>ldap2.csclub.uwaterloo.ca</tt> ([[Machine_List#auth2|auth2]]).

In order to replicate changes from the master, the slave maintains an authenticated connection to the master which provides it with full read access to all changes.

Specifically, <tt>/etc/systemd/system/k5start-slapd.service</tt> maintains an active Kerberos ticket for <tt>ldap/auth2.csclub.uwaterloo.ca@CSCLUB.UWATERLOO.CA</tt> in <tt>/var/run/slapd/krb5cc</tt>. This is then used to authenticate the slave to the server, who maps this principal to <tt>cn=ldap-slave,dc=csclub,dc=uwaterloo,dc=ca</tt>, which in turn has full read privileges.

In the event of master failure, all hosts should fail LDAP reads seamlessly over to the slave.

[[Category:Software]]

=== Modifying LDAP entry ===

Editing entries can be easily done with <code>ldapvi</code>. First search for the entry using <code>ldapsearch</code> like above, and change <code>ldapsearch -x</code> to <code>ldapvi -Y GSSAPI</code> to make your edits.

Note that if your <tt>EDITOR</tt> enviroment is set to something not avaliable it will give out errors like

error (misc.c line 180): No such file or directory
editor died
error (ldapvi.c line 83): No such file or directory

This can be fixed by something like

EDITOR=vi ldapvi ******

==== Changing a user's username ====

Only a member of the Systems Committee can change a user's username. '''At all times, a user's username must match the user's username in WatIAM.'''

All changes to an account MUST be done in person so that identity can be confirmed. If a member cannot attend in person, then an alternate method of identity verification may be chosen by the Systems Administrator.

# Edit entries in LDAP (<code>ldapvi -Y GSSAPI</code>)
#* Find and replace the user's old username with the new one (<code>%s/$OLD/$NEW/g</code>)
# Change the user's Kerberos principal (on auth1 using <code>kadmin</code>, <code>renprinc $OLD $NEW</code>)
# Move the user's home directory (on phosphoric-acid, <code>mv /users/$OLD /users/$NEW</code>)
# Modify the user's ~/.forward file if their old username is in it.
# Change the user's csc-general (and csc-industry, if subscribed) email address for <code>$OLD@csclub.uwaterloo.ca</code> to <code>$NEW@csclub.uwaterloo.ca</code>
#* https://mailman.csclub.uwaterloo.ca/admin/csc-general
# If the user has vhosts on caffeine, update them to point to their new username

If the user's account has been around for a while, and they request it, forward email from their old username to their new one.

# Edit <code>/etc/aliases</code> on mail. <code>$OLD: $NEW</code>
# Run <code>newaliases</code>

Office Policies

2024-11-17T22:56:02Z

O32patel: Link criminal code

The following are the day-to-day policies for running the CSC Office. They are set down by the Office Manager or, should there not be a current Office Manager, by the AVP.

[[Category:Policy]]
[[Category:Office]]

;Respect Other People
:#Don't be rude towards other members, even if someone is causing issues. Office Staff are expected to handle any such situations maturely.
:#Respect people's boundaries and try to avoid making people uncomfortable. If someone asks you to stop, be respectful. Office Staff have the final say.
:##Be considerate when discussing or making jokes about sensitive topics such as sex, suicide /self-harm, or politics.
:#Respect others' privacy. Don't touch others' devices or other property without permission.
:#Keep the noise level in the office at a reasonable volume, especially during business hours.
;Respect the Space
:#The space is for people, not bags. If you leave your bag on a chair or table and someone needs the space, they take priority over your bag.
:#CSC/Office Staff are not responsible for any loss or theft of property. If you choose to leave things here, you do so at your own risk.
:#Do not leave food in the office and clean up your garbage when you leave.
:#The stapler is only to be used inside the office or outside of the office for club business.
:#Disconnect from the music queue before you leave the office (otherwise people in the office will hear audio distortion from your device attempting to stay connected). Share the queue and keep music at a reasonable volume.
;Strictly Banned Behavior
:#No sleeping in the office.
:#No sex in the office. [https://laws-lois.justice.gc.ca/eng/acts/c-46/section-173.html (R.S.C., 1985, c. C-46, s. 173 (1))]
:#No illegal activities in the office.
;Office Moderation and Operations
:#There must be an Office Staff present at all times while the office is open.
:#Office Staff are expected to moderate the office environment. Office Staff should deal with issues professionally and not make situations worse.
:#Any significant incidents, such as harassment and repeated infractions of more minor rules, must be reported to the Office Manager, even if resolved.
:#If there are people in the office, the doors should be open. Office doors must remain unblocked for safety reasons.

Office Policies

2024-11-17T22:52:16Z

O32patel: F24 policy update

The following are the day-to-day policies for running the CSC Office. They are set down by the Office Manager or, should there not be a current Office Manager, by the AVP.

[[Category:Policy]]
[[Category:Office]]

;Respect Other People
:#Don't be rude towards other members, even if someone is causing issues. Office Staff are expected to handle any such situations maturely.
:#Respect people's boundaries and try to avoid making people uncomfortable. If someone asks you to stop, be respectful. Office Staff have the final say.
:##Be considerate when discussing or making jokes about sensitive topics such as sex, suicide /self-harm, or politics.
:#Respect others' privacy. Don't touch others' devices or other property without permission.
:#Keep the noise level in the office at a reasonable volume, especially during business hours.
;Respect the Space
:#The space is for people, not bags. If you leave your bag on a chair or table and someone needs the space, they take priority over your bag.
:#CSC/Office Staff are not responsible for any loss or theft of property. If you choose to leave things here, you do so at your own risk.
:#Do not leave food in the office and clean up your garbage when you leave.
:#The stapler is only to be used inside the office or outside of the office for club business.
:#Disconnect from the music queue before you leave the office (otherwise people in the office will hear audio distortion from your device attempting to stay connected). Share the queue and keep music at a reasonable volume.
;Strictly Banned Behavior
:#No sleeping in the office.
:#No sex in the office. (R.S.C., 1985, c. C-46, s. 173 (1))
:#No illegal activities in the office.
;Office Moderation and Operations
:#There must be an Office Staff present at all times while the office is open.
:#Office Staff are expected to moderate the office environment. Office Staff should deal with issues professionally and not make situations worse.
:#Any significant incidents, such as harassment and repeated infractions of more minor rules, must be reported to the Office Manager, even if resolved.
:#If there are people in the office, the doors should be open. Office doors must remain unblocked for safety reasons.

Shlink

2024-11-16T04:13:36Z

O32patel: change shlink group

[https://shlink.io/ Shlink] is a self-hosted URL shortener. We run an instance on [[Machine List#caffeine|caffeine]], currently for exec-use only.

== Installation Details ==
Shlink was installed manually (as of F2022 there is currently no Debian package for it). Its files are in <code>/usr/local/lib/shlink</code> on caffeine, with a symlink of <code>bin/cli</code> (the Shlink CLI) to <code>/usr/local/bin/shlink</code> as recommended by Shlink's docs. Due to Shlink only supporting PHP 8.2, which is in debian-unstable at the moment, the installation and CLI scripts were modified to use the <code>php8.2</code> binary installed from <code>packages.sury.org</code>.

The web client (used to manage shortlinks graphically, the other option being the CLI) sources are located in <code>/usr/local/bin/shlink/web-client/</code>. Apache is configured (through <code>sites-available/csc-links.conf</code> and <code>sites-real/csc-links</code>) to serve actual shortlinks through [https://csclub.ca csclub.ca] (which itself redirects to the CSC homepage) and the web client through [https://links.csclub.uwaterloo.ca links.csclub.uwaterloo.ca].

== Usage ==
Links can be managed at [https://links.csclub.uwaterloo.ca links.csclub.uwaterloo.ca], or through the CLI (currently restricted to the <code>exec</code> group).
[[File:Shlink1.png|none|thumb|When you open it for the first time, you will see an interface that looks like this. Click '''Add a server'''.]]
[[File:Shlink2.png|none|thumb|Enter the details as follows. Name: <code>csclub.ca</code> URL: <code><nowiki>https://csclub.ca</nowiki></code> API key: The exec API key (in the usual exec passwords location, or ask an exec for it if you aren't one)]]
On the next interface, '''custom slug''' is the part of the URL that can be customized. If it is left blank, a random 4-character slug will be generated.

Use the <code>csctest</code> parameter to disable tracking on test requests.

Machine List

2024-10-22T00:00:12Z

O32patel: Add SSDs for PA

Most of our machines are in the E7, F7, G7 and H7 racks (as of Jan. 2022) in the MC 3015 server room. There is an additional rack in the DC 3558 machine room on the third floor. Our office terminals are in the CSC office, in MC 3036/3037.

= Web Server =
You are highly encouraged to avoid running anything that's not directly related to your CSC webspace on our web server. We have plenty of general-use machines; please use those instead. You can even edit web pages from any other machine--usually the only reason you'd *need* to be on caffeine is for database access.

== ''caffeine'' ==

Caffeine is the Computer Science Club's web server. It serves websites, databases for websites, and a large amount of other services.

''(Redundant active backup coming soon...)''

==== Specs ====

* LXC virtual machine hosted on [[Machine List#phosphoric-acid|phosphoric-acid]]
** 12 vCPUs
** 32GB of RAM

==== Services ====

* Club and member web sites with [https://www.apache.org/ Apache]
* [[MySQL]] databases
* [[PostgreSQL]] databases
* [[ceo]] daemon

==''mathnews''==

[[#xylitol|xylitol]] hosts a systemd-nspawn container which serves as the mathNEWS webserver. It is administered by mathNEWS, as a pilot for providing containers to select groups who have more specialized demands than the general-use infrastructure can meet.

= General-Use Servers =

These machines can be used for (nearly) anything you like (though be polite and remember that these are shared machines). Recall that when you signed the Machine Usage Agreement, you promised not to use these machines to generate profit (so no cryptocurrency mining).

For computationally-intensive jobs (CPU/memory bound) we recommend running on high-fructose-corn-syrup, carbonated-water, sorbitol, mannitol, or corn-syrup, listed in roughly decreasing order of available resources. For low-intensity interactive jobs, such as IRC clients, we recommend running on neotame. '''If you have a long-running computationally intensive job, it's good to nice[https://en.wikipedia.org/wiki/Nice_(Unix)] your process, and possibly let syscom know too.'''

== ''corn-syrup'' ==

Dell PowerEdge 2950

==== Specs ====

* 2 × Intel Xeon E5405 (2.00 GHz, 4 cores each)
* 32 GB RAM
* eth0 ("Gb0") mac addr 00:24:e8:52:41:27
* eth1 ("Gb1") mac addr 00:24:e8:52:41:29
* IPMI mac addr 00:24:e8:52:41:2b
* 3 × Western-Digital 160GB SATA hard drive (445 GB software RAID0 array)

==== Notes ====

* Use eth0/Gb0 for the mathstudentorgsnet connection

==== Services ====

* Hosts 1 TB <tt>[[scratch|/scratch]]</tt> and exports via NFS (sec=krb5)

== ''high-fructose-corn-syrup'' ==

High-fructose-corn-syrup (or hfcs) is a large SuperMicro server. It's been in CSC service since April 2012.

==== Specs ====

* 4x AMD Opteron 6272 (2.4 GHz, 16 cores each)
* 192 GB RAM
* Supermicro H8QGi+-F Motherboard Quad 1944-pin Socket [http://csclub.uwaterloo.ca/misc/manuals/motherboard-H8QGI+-F.pdf (Manual)]
* 500 GB Seagate Barracuda
* Supermicro Case Rackmount CSE-748TQ-R1400B 4U [http://csclub.uwaterloo.ca/misc/manuals/SC748.pdf (Manual)]
'''Notes'''

* Missing moba IO shield (as of January 2024)

== ''carbonated-water'' ==

carbonated-water is a Dell R815 provided by CSCF.

==== Specs ====

* 4x AMD Opteron 6176 processors (2.3 GHz, 12 cores each)
* 128GB RAM

== ''neotame'' ==

neotame is a SuperMicro server funded by MEF. It is the successor to taurine.

'''We strongly discourage running computationally-intensive jobs''' on neotame as many users run interactive applications such as IRC clients on it and any significant service degradation will be more likely to affect other users (who will probably notice right away).

==== Specs ====

* 2x Intel Xeon E5-2630 v4 processors (2.2 GHz, 10 cores/20 threads each)
* 64GB RAM
'''Notes'''
* SSH server also listens on ports 21, 22, 53, 80, 81, 443, 8000, 8080 for your convenience.

== ''sorbitol'' ==

sorbitol is a SuperMicro server funded by MEF.

==== Specs ====

* 2x Intel Xeon E5-2630 v4 processors (2.2 GHz, 10 cores/20 threads each)
* 64GB RAM
== ''mannitol'' ==

mannitol is a SuperMicro server funded by MEF.

==== Specs ====

* 2x Intel Xeon E5-2630 v4 processors (2.2 GHz, 10 cores/20 threads each)
* 64GB RAM

= Office Terminals =

It's possible to SSH into these machines, but we discourage you from trying to use these machines when you're not sitting in front of them. They are bounced at least every time our login manager, lightdm, throws a tantrum (which is several times a day). These are for use inside our physical office.

== ''cyanide'' ==
cyanide is a [https://support.apple.com/kb/sp710 Mac Mini (Late 2014)], identical in specification to powernap

=== Spec ===

* Intel i7-4578U (4) @ 3.500GHz
* 16GB RAM
* Intel Iris Graphics 5100
* 256GB On-board SSD

== ''suika'' ==

Suika is an office terminal built from various components donated by our members.

==== Specs ====

* AMD Ryzen 7 2700X
* 2x 8GB DDR4
* 1x Samsung 256GB SSD
* AMD Radeon RX 550 4GB

==''powernap''==
powernap is a [https://support.apple.com/kb/sp710 Mac Mini (Late 2014)].

=== Spec ===

* Intel i7-4578U (4) @ 3.500GHz
* 16GB RAM
* Intel Iris Graphics 5100
* 256GB On-board SSD

=== Speaker ===
powernap has the office speakers (a pair of nice studio monitors) currently connected to it.

=== Services ===
* MPD for playing music. Only office/termcom/syscom can log into powernap. Use `ncmpcpp` to control MPD.
** TODO: this is not the case anymore
* Bluetooth audio receiver. Only syscom can control bluetooth pairing. Use `bluetoothctl` to control bluetooth.

Music is located in `/music` on the office terminals.

= Progcom Only =
The Programme Committee has access to a VM on corn-syrup called 'progcom'. They have sudo rights in this VM so they may install and run their own software inside it. This VM should only be accessible by members of progcom or syscom.

= Codey Bot Only =
Ran on CSC Cloud in a separate Cloudstack project. codey-staging, codey-dev, codey-prod.

TODO: migrating from cloudstack

= Syscom Only =

The following systems are only be accessible to members of the [[Systems Committee]] for a variety of reasons; the most common of which being that some of these machines host [[Kerberos]] authentication services for the CSC.

==''xylitol''==

xylitol is a Dell PowerEdge R815 donated by CSCF. It is primarily a container host for services previously hosted on aspartame and dextrose, including munin, rt, mathnews, auth1, and dns1. It was provisioned with the intent to replace both of those hosts.

==== Specs ====

* Dual AMD Opteron 6176 (2.3 GHz, 48 cores total)
* 128GB RAM
* 500GB volume group on RAID1 SSD (xylitol-mirrored)
* 500ish-GB volume group on RAID10 HDD (xylitol-raidten)

==''auth1''==

Container on [[#xylitol|xylitol]].

==== Services ====

*[[LDAP]] primary
*[[Kerberos]] primary

==''chat''==

Container on [[#xylitol|xylitol]].

==== Services ====

* The Lounge web IRC client (https://chat.csclub.uwaterloo.ca)

==''phosphoric-acid''==

phosphoric-acid is a Dell PowerEdge R815 donated by CSCF and is a clone of xylitol. It may be used to provide redundant cloud services in the future.

==== Specs ====

* (clone of Xylitol)
* 4x 2TB Kingston KC3000 (ZFS Z2 [Sustain 2-failures]) (KIN-SKC3000D2048G)
** Mounted on 2x Startech Dual M.2 PCIE SSD Adapter Cards (STA-PEX8M2E2)

==== Services ====

*[[#caffeine|caffeine]]
*[[#coffee|coffee]]

==''coffee''==

Virtual machine running on phosphoric-acid.

==== Services ====

*[[Database#MySQL|MySQL]]
*[[Database#Postgres|Postgres]]

==''cobalamin''==

Dell PowerEdge 2950 donated to us by FEDS. Located in the Science machine room on the first floor of Physics, on Science Computing Rack 2. NICs are plugged into A1 and A2 on the adjacent rack. Acts as a backup server for many things.

TODO: should replace with another Syscom server when Science Computing clears out the rack (ETA before 09/2024)

==== Specs ====

* 1 × Intel Xeon E5420 (2.50 GHz, 4 cores)
* 16GB RAM
* Broadcom NetworkXtreme II
* 2x73GB Hard Drives, hardware RAID1
** Soon to be 2x1TB in MegaRAID1
*http://www.dell.com/support/home/ca/en/cabsdt1/product-support/servicetag/51TYRG1/configuration

==== Services ====

* Containers: [[#auth2|auth2]] (kerberos)

==== Notes ====

* '''TODO: Mega unreliable.''' (Goes down once every few weeks... due to power outages in the PHYS server room)
** It is plugged into a UPS but the UPS has dead batteries.
* The network card requires non-free drivers. Be sure to use an installation disc with non-free.

* We have separate IP ranges for cobalamin and its containers because the machine is located in a different building. They are:
** VLAN ID 506 (csc-data1): 129.97.18.16/29; gateway 129.97.18.17; mask 255.255.255.240
** VLAN ID 504 (csc-ipmi): 172.19.5.24/29; gateway 172.19.5.25; mask 255.255.255.248
* Physical access to the PHYS server rooms can be acquired by visiting Science Computing in PHYS 2006.

==''auth2''==

Container on [[#cobalamin|cobalamin]].

==== Services ====

*[[LDAP]] secondary
*[[Kerberos]] secondary

MAC Address: c2:c0:00:00:00:a2

==''mail''==

mail is the CSC's mail server. It hosts mail delivery, imap(s), smtp(s), and mailman. It is also syscom-only. It is a [[Virtualization#Linux_Containers|Linux container]] at present.

TODO: "HA"-ish configuration

==== Specs ====

* currently hosted on [[#xylitol|xylitol]]

==== Services ====

*[[Mail]] services
* mailman (web interface at [http://mailman.csclub.uwaterloo.ca/])
*[[Webmail]]
*[[ceo]] daemon

==''sodium-benzoate''==

Sodium-benzoate is our previous mirror server, funded by MEF.

It is currently sitting in the office pending repurposing. Will likely become a machine for backups in DC.

==== Specs ====

* Intel Xeon Quad Core E5405 @ 2.00 GHz
* 16GB RAM
* vg0: 228 GB block device behind DELL PERC 6/i (contains root partition)

Space disks are currently in the office underneath maltodextrin.

TODO: gone??

==''potassium-benzoate''==

potassium-benzoate is our mirror server, funded by MEF.

==== Specs ====

* 36 drive Supermicro chassis (SSG-6048R-E1CR36L)
* 1 x Intel Xeon E5-2630 v3 (8 cores, 2.40 GHz)
* 64 GB (4 x 16GB) of DDR4 (2133Mhz) ECC RAM
* 2 x 1 TB Samsung Evo 850 SSD drives
* 17 x 4 TB Western Digital Gold drives (separate funding from MEF)
* 9 x 18TB Seagate Exos X18 (8 ZFS, Z2,1 hot-spare)
* 10 Gbps SFP+ card (loaned from CSCF)
* 50 Gbps Mellanox QSFP card (from ginkgo; currently unconnected)

==== Network Connections ====

potassium-benzoate has two connections to our network:

* 1 Gbps to our switch (used for management)
* 2 x 10 Gbps (LACP bond) to mc-rt-3015-mso-a (for mirror)

Mirror's bandwidth is limited to 1 Gbps on each of the 4 campus internet links. Mirror's bandwidth is not limited on campus.

==== Services ====

*[[Mirror]]
*[[Talks]] mirror
*[[Debian_Repository|CSClub packages repository]]

==''munin''==

munin is a syscom-only monitoring and accounting machine. It is a [[Virtualization#Linux_Containers|Linux container]] at present.

==== Specs ====

* currently hosted on [[#xylitol|xylitol]]

==== Services ====

*[http://munin.csclub.uwaterloo.ca munin] systems monitoring daemon
TODO: Debian 9?

==''yerba-mate''==

A Dell PowerEdge 2950 donated by a CSC member.

==== Specs ====

* 2x 3.00 GHz quad core Intel Xeon 5160
* 32GB RAM
* 2x75GB 15k drives (RAID 1)

==== Services ====

* test-ipv6 (test-ipv6.csclub.uwaterloo.ca; a test-ipv6.com mirror)
'''Notes'''

* Also used for experimenting new CSC services.

* TODO: use as backup server

==''citric-acid''==
A Dell PowerEdge R815 (TODO: check model) provided by CSCF to replace [[Machine List#aspartame|aspartame]].

'''Specs'''

* 2 x AMD Opteron 6174 (12 cores, 2.20 GHz)
* 128 GB RAM

'''Services'''

* Configured for [https://pass.uwaterloo.ca pass.uwaterloo.ca], a university-wide password manager hosted by CSC as a demo service for all Nexus (ADFS) user.
* [[Plane]], an internal (CSC) project management tool.
* Minio
'''Notes'''

* Being repurposed for Termcom training and development.
* TODO: migrate Vaultwarden (https://pass.csclub.uwaterloo.ca/)??
* UFW opened-ports: SSH, HTTP/HTTPS
* Upgraded to Podman 4.x

= Cloud =

These machines are used by [https://cloud.csclub.uwaterloo.ca cloud.csclub.uwaterloo.ca]. The machines themselves are restricted to Syscom only access.

==''chamomile''==

A Dell PowerEdge R815 provided by CSCF.

==== Specs ====

* 4x 2.20GHz 12-core processors (AMD Opteron(tm) Processor 6174)
* 128GB RAM
* 10GbE connection to core router

==== Services ====

* Cloudstack host
* TODO: cloudstack migration

==''riboflavin''==

A Dell PowerEdge R515 provided by CSCF.

==== Specs ====

* 2x 2.6 GHz 8-core processors (AMD Opteron(tm) Processor 4376 HE)
* 64GB RAM
* 10GbE connection to core router
* 2x 500GB internal SSD
* 12x Seagate 4TB SSHD
==== Services ====

* OpenStack block and object storage for csclub.cloud
* ????
'''Notes'''

* TODO: cloudstack migration

==''guayusa''==

A Dell PowerEdge 2950 donated by a CSC member.

==== Specs ====

* 2x 3.00 GHz quad core Intel Xeon 5160
* 32GB RAM
* 2TB PCI-Express Flash SSD
* 2x75GB 15k drives (RAID 1)

==== Services ====

* load-balancer-01

Was used to experiment the following then-new CSC services:

* cifs (for booting ginkgo from CD)
* caffeine-01 (testing of multi-node caffeine)
* TODO: ???
** block1.cloud
** object1.cloud
'''Notes'''

* TODO: cloudstack migration
* TODO: ditch... Currently being used to set up NextCloud.

==''ginkgo''==

Supermicro server funded by MEF for CSC web hosting. Locate in MC 3015.

==== Specs ====

* 2x Intel Xeon E5-2697 v4 @ 2.30GHz [18 cores each]
* 256GB RAM
* 2 x 1.2 TB SSD (400GB of each for RAID 1)
* 10GbE onboard, 25GbE SFP+ card (also included 50GbE SFP+ card which will probably go in mirror)

==== Services ====

* OpenStack Compute machine

No longer in use:

* controller1.cloud
* db1.cloud
* router1.cloud (NAT for cloud tenant network)
* network1.cloud

'''Notes'''
* TODO: cloudstack migration

==''biloba''==

Supermicro server funded by SLEF for CSC web hosting. Located in DC 3558. TODO: rack??

==== Specs ====

* 2x Intel Xeon Gold 6140 @ 2.30GHz [18 cores each]
* 384GB RAM
* 12 3.5" Hot Swap Drive Bays
** 2 x 480 GB SSD
* 10GbE onboard, 10GbE SFP+ card (on loan from CSCF)

==== Services ====

* OpenStack Compute machine
'''Notes'''
* TODO: cloudstack migration

No longer in use:

* caffeine
* mail
* mattermost

= Storage =

==''fs00''==

fs00 is a '''NetApp FAS3040''' series fileserver donated by CSCF.

It is currently being used for testing of a HA NetApp nodes and serving home directories directly from the NetApp filer.

==== Specs ====

* dual SFP connection to core switch

... TODO

==''fs01''==

fs01 is a '''NetApp FAS3040''' series fileserver donated by CSCF.

It is currently being used for testing of a HA NetApp nodes and serving home directories directly from the NetApp filer.

==== Specs ====
... TODO

TODO: disconnected??

==''fs10''==

fs10 is a '''NetApp FAS8040''' series fileserver donated by CSCF.

It is currently being used for testing of a HA NetApp nodes and serving home directories directly from the NetApp filer.

==== Specs ====

* FAS8040 (dual heads)
** ... TODO
* 6 DS4324 HDD shelves (24-disks each)
** 24 x 2TB HDDs (assorted brands/models)
** Dual IOM3 controllers.
** Loop 1: bottom 4 shelves
** Loop 2: top 2 shelves + SSD shelf
* 1 DS2246 SSD shelf (TODO: right model?)
** 24 Samsung SM1625 SSDs (MZ-6ER2000/0G3), 200GB (SAS 2, 2.5")

= Other =

==''mathnews''==

[[#xylitol|xylitol]] hosts a systemd-nspawn container which serves as the mathNEWS webserver. It is administered by mathNEWS, as a pilot for providing containers to select groups who have more specialized demands than the general-use infrastructure can meet.

== ps3 ==
This is just a very wide PS3, the model that supported running Linux natively before it was removed. Firmware was updated to remove this feature, however it can still be done via. homebrew.

'''Specs'''

* It's a PS3.

'''2022-10-24''' - Thermal paste replaced + firmware updated to latest supported version, also modded.

==''binaerpilot''==

This is a Gumstix Overo Tide CPU on a Tobi expansion board. It is currently attached to corn-syrup in the machine room and even more currently turned off until someone can figure out what is wrong with it.

==== Specs ====

* TI OMAP 3530 750Mhz (ARM Cortex-A8)
* 512MB RAM

==''anamanaguchi''==

This is a Gumstix Overo Tide CPU on a Chestnut43 expansion board. It is currently in the hardware drawer in the CSC.

==== Specs ====

* TI OMAP 3530 750Mhz (ARM Cortex-A8)
* 512MB RAM

==''digital cutter''==

See [[Digital Cutter|here]].

= Decommissioned =

==''aspartame''==

aspartame was a taurine clone donated by CSCF. It was once our primary file server, serving as the gateway interface to space on phlogiston. It also used to host the [[#auth1|auth1]] container, which has been temporarily moved to [[#dextrose|dextrose]]. Decomissioned in March 2021 after refusing to boot following a power outage.

==''psilodump''==

psilodump is a NetApp FAS3000 series fileserver donated by CSCF. It, along with its sibling phlogiston, hosted disk shelves exported as iSCSI block devices.

psilodump was plugged into aspartame. It's still installed but inaccessible.

==''phlogiston''==

phlogiston is a NetApp FAS3000 series fileserver donated by CSCF. It, along with its sibling psilodump, hosted disk shelves exported as iSCSI block devices.

phlogiston is turned off and should remain that way. It is misconfigured to have its drives overlap with those owned by psilodump, and if it is turned on, it will likely cause irreparable data loss.

==== Specs ====

* 2 AMD Opteron 2218 CPUs
* 10GB RAM

==== Notes from before decommissioning ====

* The lxc files are still present and should not be started up, or else the two copies of auth1 will collide.
* It currently cannot route the 10.0.0.0/8 block to a misconfiguration on the NetApp. This should be fixed at some point.

==''glomag''==

Glomag hosted [[#caffeine|caffeine]]. Decommissioned April 6, 2018.

==== Specs ====

* Intel Xeon X3450 @ 2.67 GHz
* 6 GB RAM
* vg0: 465 GB software RAID1 (contains root partition):
** 750 GB Seagate Barracuda SATA hard drive
** 500 GB Western-Digital Caviar Blue SATA hard drive
* vg1: 596 GB software RAID1 (contains caffeine):
** 2 × 640 GB Western-Digital Caviar Blue SATA hard drive

==== Services ====

* Before its decommissioning, glomag hosted [[#caffeine|caffeine]], [[#mail|mail]], and [[#munin|munin]] as [[Virtualization#Linux_Container|Linux containers]]

==''Lisp machine''==

A Symbolics XL1200 Lisp machine. Donated to a new home when we couldn't get it working.

http://www.globalnerdy.com/2008/12/03/symbolics-xl1200-lisp-machine-free-to-a-good-home/ for some history on this hardware.

==== Specs ====

Currently inoperable due to (at least) a missing console cable.

==''ginseng''==

Ginseng used to be our fileserver, before aspartame and the netapp took over.

==== Specs ====

* Intel Pentium Dual Core E2180
* 8GB RAM
*[http://csclub.uwaterloo.ca/misc/manuals/s3000ah_tps_1_1.pdf Intel S3000AHV Motherboard]
* 4 × 640 GB Western-Digital Caviar Blue in [[wikipedia:Nested_RAID_levels#RAID_10_.28RAID_1.2B0.29|RAID 10]] behind a [http://www.3ware.com/products/serial_ata2-9650.asp 3ware 9650SE RAID card].
[[Category:Hardware]]

==''calum''==

Calum used to be our main server and was named after Calum T Dalek. Purchased new by the club in 1994.

==== Specs ====

* SPARCserver 10 (headless SPARCstation 10)

==''paza''==

An iMac G3 that was used as a dumb terminal.

==== Specs ====

* 233Mhz PowerPC 740/750
* 96 MB RAM

==''romana''==

Romana was a BeBox that has been in the CSC's possession since long before BeOS became defunct.

Confirmed on March 19th, 2016 to be fully functional. An SSHv1 compatible client was installed from http://www.abstrakt.ch/be/ and a compatible firewalled daemon was started on Sucrose (living in /root, prefix is /root/ssh-romana). The insecure daemon is to be used a bastion host to jump to hosts only supporting >=SSHv2. The mail daemon on the BeBox has also been configured to send mail through mail.csclub.uwaterloo.ca.

==== Specs ====

* 2 PowerPC based processors
* Stylish Blinken processor-load lights

==''sodium-citrate''==

Sodium-citrate was an SGI O2 machine.

In order to net boot you need to set /proc/sys/net/ipv4/ip_no_pmtu_disc to 1. When the O2 boots, hit F5 at the boot menu and type bootp():.

==== Specs ====
* SGI O2 MIPS processor
* 423 MB (?) RAM
* 2 × 2 GB hard drive

==''acesulfame-potassium''==

An old office terminal.

==== Specs ====
* Intel Pentium 4 2.67GHz
* 1GB RAM
*[http://csclub.uwaterloo.ca/misc/manuals/ABIT_VT7.pdf ABIT VT7] Motherboard
* ATI Radeon 7000

==''skynet''==

skynet was a Sun E6500 machine donated by Sanjay Singh. It was never fully set up.

==== Specs ====

* 15 full CPU/memory boards
** 2x UltraSPARC II 464MHz / 8MB Cache Processors
** ??? RAM?
* 1 I/O board (type=???)
** ???x disks?
* 1 CD-ROM drive

*[http://mirror.csclub.uwaterloo.ca/csclub/sun_e6500/ent6k.srvr/ e6500 documentation (hosted on mirror, currently dead link)]
*[http://docs.oracle.com/cd/E19095-01/ent6k.srvr/ e6500 documentation (backup link)]
*[http://www.e6500.com/ e6500]

==''freebsd''==

FreeBSD was a virtual machine with FreeBSD installed.

==== Services ====

* Newer software

==''rainbowdragoneyes''==

Rainbowdragoneyes was our Lemote Fuloong MIPS machine. This machine is aliased to rde.csclub.uwaterloo.ca.

==== Specs ====

* 800MHz MIPS Loongson 2f CPU

==''denardo''==

Due to some instability, general uselessness, and the acquisition of a more powerful SPARC machine from MFCF, denardo was decommissioned in February 2015.

==== Specs ====

* Sun Fire V210
* TI UltraSparc IIIi (Jalapeño)
* 2 GB RAM
* 160 GB RAID array
* ALOM on denardo-alom.csclub can be used to power machine on/off
==''artificial-flavours''==

Artificial-flavours was our secondary (backup services) server. It used to be an office terminal. It was decommissioned in February 2015 and transferred to the ownership of Women in Computer Science (WiCS).

==== Specs ====

* Intel Celeron 3.2GHz
* 2GB RAM
*[http://csclub.uwaterloo.ca/misc/manuals/Biostar_P4M80-M4.pdf Biostar P4M80-M4] Motherboard
* Western-Digital 80 GB ATA hard drive

==''potassium-citrate''==

Potassium-citrate is a dual-processor Alpha machine. It is on extended loan from pbarfuss.

It is temporarily decommissioned pending the reinstallation of a supported operating system (such as OpenBSD).

==== Specs ====
* Alphaserver CS20 (2 833MHz EV68al CPUs)
* 512MB RAM
* 36 GB Seagate SCSI hard drive

==''potassium-nitrate''==

This was a Sun Fire E2900 from a decommissioned MFCF compute cluster. It had a SPARC architecture and ran OpenBSD, unlike many of our other systems which are x86/x86-64 and Linux/Debian. After multiple unsuccessful attempts to boot a modern Linux kernel and possible hardware instability, it was determined to be non-cost-effective and non-effort-effective to put more work into running this machine. The system was reclaimed by MFCF where someone from CS had better luck running a suitable operating system (probably Solaris).

The name is from saltpetre, because sparks.

==== Specs ====

* 24 CPUs
* 90GB main memory
* 400GB scratch disk local storage in /scratch-potassium-nitrate

There is a [[Sun 2900 Strategy Guide|setup guide]] available for this machine.

See also [[Sun 2900]].

==''taurine''==

'''Note: On August 21, 2019, just before 2:30PM EDT, we were informed that taurine caught fire'''. As a result, taurine has been decommissioned as of Fall 2019.

==== Specs ====

* 2 AMD Opteron 2218 CPUs
* 8GB RAM
* 136 GB LVM volume group

==== Services ====

* Virtual machines
* BitlBee IRC instant messaging gateway (localhost only)
*[[ident]] server to maintain high connection cap to freenode
* Runs ssh on ports 21,22,53,80,81,443,8000,8080 for user's convenience.

==''dextrose''==

dextrose was a [[#taurine|taurine]] clone donated by CSCF and was decommissioned in Fall 2019 after being replaced with a more powerful server.

==''sucrose''==

sucrose was a [[#taurine|taurine]] clone donated by CSCF. It was decommissioned in Fall 2019 following multiple hardware failures.

==''goto80''==
'''Note (2022-10-25): This seems to have gone missing or otherwise left our hands.'''

This was small ARM machine we picked up in order to have similar hardware to the Real Time Operating Systems (CS 452) course. It has a [[TS-7800_JTAG|JTAG]] interface. Located was the office on the top shelf above strombola.

==== Specs ====

* 500 MHz Feroceon (ARM926ej-s compatible) processor
* ARMv5TEJ architecture

Use -march=armv5te -mtune=arm926ej-s options to GCC.

For information on the TS-7800's hardware see here:
http://www.embeddedarm.com/products/board-detail.php?product=ts-7800

==''nullsleep''==

nullsleep is an [http://csclub.uwaterloo.ca/misc/manuals/ASRock_ION_330.pdf ASRock ION 330] machine given to us by CSCF and funded by MEF.

It's decommissioned on 2023-03-20 due to repeated unexpected shutdown. Replaced by [[#powernap|powernap]].

==== Specs ====

* Intel® Dual Core Atom™ 330
* 2GB RAM
* NVIDIA® ION™ graphics
* 1x 64GB SanDisk SDSSDP064G SSD
* DVD Burner

==== Speakers ====
Nullsleep has the office speakers (a pair of nice studio monitors) currently connected to it.

==== Services ====
Nullsleep runs MPD for playing music. Control of MPD is available only to users in the "audio" group.
Music is located in /music on the office terminal

== ''bit-shifter'' ==
bit-shifter was an office terminal, decommissioned April 2023 due to extended age. It was upgraded to the same specs as Strombola at an unknown point in time.

==== Specs ====

* Intel Core 2 Quad CPU Q8300
* 4GB RAM
* Nvidia GeForce GT 440
*[http://csclub.uwaterloo.ca/misc/manuals/motherboard_manual_ga-ep45-ud3l.pdf Gigabyte GA-EP45-UD3L] Motherboard
* 1x 64GB SanDisk SDSSDP064G SSD
* Jacob Parker's Firewire Card

==== Services ====

*[http://csclub.uwaterloo.ca/office/webcam Office webcam]

== ''strombola''==
Strombola was an office terminal named after Gordon Strombola. It was retired in April 2023.

==== Specs ====
* Intel Pentium G4600 2 cores @ 3.6Ghz
* 8 GB RAM
* 1x 64GB SanDisk SDSSDP064G SSD

==== Speakers ====
Strombola used to have integrated 5.1 channel sound before we got new speakers and moved audio stuff to nullsleep.

== ''gwem'' ==
gwem was an office terminal that was created because AMD donated a graphics card. It entered CSC service in February 2012.

=== Specs ===

* AMD FX-8150 3.6GHz 8-Core CPU
* 16 GB RAM
* AMD Radeon 6870 HD 1GB GPU
*[http://csclub.uwaterloo.ca/misc/manuals/ga-990fxa-ud7_e.pdf Gigabyte GA-990FXA-UD7] Motherboard
* 1x 64GB SanDisk SDSSDP064G SSD

== ''maltodextrin'' ==
(*specs are outdated at least as of 2023-05-27*)
*[http://csclub.uwaterloo.ca/misc/manuals/motherboard_manual_ga-ep45-ud3l.pdf Gigabyte GA-EP45-UD3L] Motherboard
Maltodextrin was an office terminal. It was upgraded in Spring 2014 after an unidentified failure. Not operational (no video output) as of July 2022.

==== Specs ====

* Intel Core i3-4130 @ 3.40 GHz
* 8GB RAM
* 1x 64GB SanDisk SDSSDP064G SSD
*[http://csclub.uwaterloo.ca/misc/manuals/E8425_H81I_PLUS.pdf ASUS H81-PLUS] Motherboard

==== Services ====

*[http://csclub.uwaterloo.ca/office/webcam Office webcam]

== ''natural-flavours'' ==

Natural-flavours is an office terminal; it used to be our mirror.

In Fall 2016, it received a major upgrade thanks the MathSoc's Capital Improvement Fund.

==== Specs ====

* Intel Core i7-6700k
* 2x8GB RAM
* 1x 64GB SanDisk SDSSDP064G SSD
* Cup Holder (DVD drive has power, but not connected to mother board)
= UPS =

All of the machines in the MC 3015 machine room are connected to one of our UPSs.

All of our UPSs can be monitored via CSCF:

* MC3015-UPS-B2
* mc-3015-e7-ups-1.cs.uwaterloo.ca (rbc55, batteries replaced July 2014) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-e7-ups-1&var-Interval=30m)
* mc-3015-f7-ups-1.cs.uwaterloo.ca (rbc55, batteries replaced Feb 2017) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-f7-ups-1&var-Interval=30m)
* mc-3015-g7-ups-1.cs.uwaterloo.ca (su5000t, batteries replaced 2010) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-g7-ups-1&var-Interval=30m)
* mc-3015-g7-ups-2.cs.uwaterloo.ca (unknown) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-g7-ups-2&var-Interval=30m)
* mc-3015-h7-ups-1.cs.uwaterloo.ca (su5000t, batteries replaced 2004) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-h7-ups-1&var-Interval=30m)
* mc-3015-h7-ups-2.cs.uwaterloo.ca (unknown) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-h7-ups-2&var-Interval=30m)

We will receive email alerts for any issues with the UPS. Their status can be monitored via [[SNMP]].

TODO: Fix labels & verify info is correct & figure out why we can't talk to cacti.

IPMI101

2024-10-16T09:16:17Z

O32patel: riboflavin ipmi config detail

= Guide to IPMI (IPMI 101) =

IPMI is a necessary evil. Let’s learn to make the best of it.

== Setting up IPMI ==

# Install ipmitool

<pre># apt-get install ipmitool</pre>
<ol start="2" style="list-style-type: decimal;">
<li>Load IPMI modules (they are included in most upstream kernels)</li></ol>

You may also need a kernel module specific to your motherboard’s manufacture as some BMC/LOMs do not conform to IPMI spec and thus need a translation layer.

<pre># modprobe ipmi_*</pre>
<ol start="3" style="list-style-type: decimal;">
<li>Locally connect to the <code>/dev/ipmi</code> interface</li></ol>

<pre># ipmitool shell
> help
> mc info</pre>
== Securing IPMI ==

Note that root on the machine is root on the BMC and vice versa.

# User administration

(re)set the password, rename the admin account to root and delete any extra users as they can have surprising privilege. You may have to use the BMC’s web interface delete accounts.

<pre># ipmitool shell
> user list 1
ID Name ...
2 ADMIN ...
> user set password 2
User id 2: *******
User id 2: *******
> user set username 2 root
> user disable $other_user_ids</pre>
<ol start="2" style="list-style-type: decimal;">
<li>Disable NULL password and cipher suite 0</li></ol>

Note that the $channel is usually 0 but can range from 0-10 and there can be multiple NICs and so multiple channels to fix.

<pre># ipmitool shell
> lan print $channel
> lan set $channel auth ADMIN MD5
> lan set $channel auth CALLBACK MD5
> lan set $channel auth USER MD5
> lan set $channel auth OPERATOR MD5
> lan set $channel cipher_privs XXXaXXXXXXXXXXX
> lan print $channel</pre>
== Configuring networking ==

Note once again that there are sometimes multiple channels, to find the correct channel it is helpful to use either trial and error and/or an ARP scanner to find the correct MAC address. Usually the channel is 0 but I have seen 1, 8 and 17. Especially when there are multiple NICs.

<pre># ipmitool shell
> lan print $channel
> lan set $channel ipsrc static
> lan set $channel ipaddr 10.15.134.?
> lan set $channel defgw ipaddr 10.15.134.1
> lan set $channel netmask 255.255.255.0
// if you have vlan tagging enabled on the switch port, useful for a shared NIC
> lan set $channel vlan id 520</pre>
== Configuring Serial over LAN ==

To enable serial over LAN you need to ensure that it is enabled in your BIOS or EFI setup utility and further note the baud rate. 115200 is used as an example below. Note that GRUB is the only boot loader that takes input via serial properly, in my experience. Syslinux failed horribly on corn-syrup.

Paste the following into /etc/default/grub.d/99-csclub.cfg:

<pre>
GRUB_CMDLINE_LINUX="console=tty1 console=ttyS1,115200n8"
GRUB_TERMINAL_INPUT="console serial"
GRUB_TERMINAL_OUTPUT="console serial"
GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=1 --word=8 --parity=no --stop=1"
</pre>
and then run:

<pre>// on debian based distros
// Yay, Debian magic :\
# update-grub
// on upstream packages (Arch, Fedora, etc.)
# grub-mkconfig -o /boot/grub/grub.cfg

# reboot</pre>

= iDRAC =
== riboflavin ==
riboflavin is using iDRAC 6. The web console can be viewed from https://riboflavin-ipmi.csclub.uwaterloo.ca; if you are not on campus, you can use a [[How_to_SSH#SOCKS_proxy|SOCKS proxy]]. Unfortunately, the virtual console uses Java Web Start, which is now deprecated. Here's a workaround which you can use instead.

From the web UI, go to the "Console/Media" tab and click the "Launch virtual console" button. This will download a file whose name starts with "viewer.jnlp". Now go to https://www.java.com and download JRE 8; any later version will not have support for JWS (note that OpenJDK will not work; JWS was a proprietary framework from Sun/Oracle). Unpack the tarball, open jre1.8.0_391/lib/security/java.security in a text editor, and comment out the following properties (note that each property spans multiple lines):
<ul>
<li>jdk.certpath.disabledAlgorithms</li>
<li>jdk.jar.disabledAlgorithms</li>
<li>jdk.tls.disabledAlgorithms</li>
</ul>

If you are off-campus, you will need to setup some proxying so that the Java application can access ports 443 and 5900 on riboflavin-ipmi. In the example below, I am using caffeine as a jump host, but any machine on campus should do:
<pre>
ssh -L 5443:localhost:5443 -L 5900:localhost:5900 caffeine.csclub.uwaterloo.ca
</pre>

Now on caffeine, open a tmux/screen session, and run the following commands in two different panes:
<ul>
<li><code>socat TCP-LISTEN:5443,fork TCP:riboflavin-ipmi:443</code></li>
<li><code>socat TCP-LISTEN:5900,fork TCP:riboflavin-ipmi:5900</code></li>
</ul>

Back on your personal machine, open the viewer.jnlp file in a text editor and perform the following:
<ol>
<li>Replace all instances of <code>riboflavin-ipmi.csclub.uwaterloo.ca:443</code> with <code>localhost:5443</code></li>
<li>Under the <code>application-desc</code> element, the first <code>argument</code> child element should say <code>ip=riboflavin-ipmi.csclub.uwaterloo.ca</code>. Replace this with <code>ip=localhost</code></li>.
<li>Under the <code>application-desc</code> element, there are child <code>argument</code>elements for <code>user</code> and <code>passwd</code>. For some reason these are set to numbers; set these to the username and password for IPMI (username should be <code>root</code>).</li>
</ol>

Now run:
<pre>
jre1.8.0_391/bin/javaws viewer.jnlp
</pre>

If all goes well, the virtual console should eventually appear:
[[File:Riboflavin-idrac-virtual-console.png|1000px]]

== carbonated-water ==
carbonated-water is also using iDRAC 6, but seems to have some kind of TLS certificate configuration which prevents modern browsers from loading its web UI. So we're going to run an old version of Firefox inside a Podman container instead:
<pre>
podman run --name firefox -it -e DISPLAY --net=host -v $XAUTHORITY:/root/.Xauthority -v /tmp/.X11-unix:/tmp/.X11-unix debian:9-slim bash
sed -i 's/deb\.debian\.org/archive.debian.org/' /etc/apt/sources.list
sed -i 's/security\.debian\.org/archive.debian.org/' /etc/apt/sources.list
sed -i '/stretch-updates/d' /etc/apt/sources.list
apt update
apt install firefox-esr
firefox
</pre>
Next, follow the instructions here to set up a SOCKS proxy: [[How to SSH#SOCKS proxy]]

Now visit https://carbonated-water-ipmi.csclub.uwaterloo.ca from Firefox, login using the IPMI credentials, and download the JNLP file. Copy it from the Podman container to your computer (replace "viewer.jnlp" with the full file name):
<pre>
podman cp firefox:/root/Downloads/viewer.jnlp launch.jnlp
</pre>
Follow the same steps as done for riboflavin to edit the JDK settings and JNLP file. In addition, there are a few more settings which we need to tweak:
<ul>
<li>Run <code>jre1.8.0_391/bin/ControlPanel</code>, go to the Advanced tab, scroll down and check "TLS 1.0" and "TLS 1.1"</li>
<li>We also need to disable OCSP. In the same window, set "Check for signed code certificate revocation using" to "Certificate Revocation Lists (CRLs)" and set "Check for TLS certificate revocation using" to "Certificate Revocation Lists (CRLs)" (see [https://www.kunxi.org/2015/01/bypass-the-certpathvalidatorexception-caused-by-malformed-ocsp-response/ here] for the reference).</li>
</ul>
[[File:java-control-panel-advanced.png]]

Now you can launch the JNLP file as usual.

= Supermicro =
== ginkgo ==
To access the virtual console on ginkgo, the steps are the same as those for riboflavin, with the following changes:
<ul>
<li>In the launch.jnlp file, in the root <code><jnlp></code> tag, change the value of the <code>codebase</code> attribute from <code>https<nowiki/>://ginkgo-ipmi.csclub.uwaterloo.ca:443</code> to <code>https<nowiki/>://localhost:5443</code>. Next, in the first <code><argument></code> element under <code><application-desc></code>, replace <code>ginkgo-ipmi.csclub.uwaterloo.ca</code> with <code>localhost</code>. These are the only changes which you should make to this file (unless you are already on the campus network, in which case you do not need to modify this file at all).</li>
<li>Run <code>jre1.8.0_391/bin/ControlPanel</code>, go to the Security tab, click "Edit Site List", and add <code>https<nowiki/>://ginkgo-ipmi.csclub.uwaterloo.ca</code> as an exception.
</ul>

Machine List

2024-04-04T06:26:24Z

O32patel: Sorbitol is back online

Most of our machines are in the E7, F7, G7 and H7 racks (as of Jan. 2022) in the MC 3015 server room. There is an additional rack in the DC 3558 machine room on the third floor. Our office terminals are in the CSC office, in MC 3036/3037.

= Web Server =
You are highly encouraged to avoid running anything that's not directly related to your CSC webspace on our web server. We have plenty of general-use machines; please use those instead. You can even edit web pages from any other machine--usually the only reason you'd *need* to be on caffeine is for database access.

== ''caffeine'' ==

Caffeine is the Computer Science Club's web server. It serves websites, databases for websites, and a large amount of other services.

''(Redundant active backup coming soon...)''

==== Specs ====

* LXC virtual machine hosted on [[Machine List#phosphoric-acid|phosphoric-acid]]
** 12 vCPUs
** 32GB of RAM

==== Services ====

* Club and member web sites with [https://www.apache.org/ Apache]
* [[MySQL]] databases
* [[PostgreSQL]] databases
* [[ceo]] daemon

==''mathnews''==

[[#xylitol|xylitol]] hosts a systemd-nspawn container which serves as the mathNEWS webserver. It is administered by mathNEWS, as a pilot for providing containers to select groups who have more specialized demands than the general-use infrastructure can meet.

= General-Use Servers =

These machines can be used for (nearly) anything you like (though be polite and remember that these are shared machines). Recall that when you signed the Machine Usage Agreement, you promised not to use these machines to generate profit (so no cryptocurrency mining).

For computationally-intensive jobs (CPU/memory bound) we recommend running on high-fructose-corn-syrup, carbonated-water, sorbitol, mannitol, or corn-syrup, listed in roughly decreasing order of available resources. For low-intensity interactive jobs, such as IRC clients, we recommend running on neotame. '''If you have a long-running computationally intensive job, it's good to nice[https://en.wikipedia.org/wiki/Nice_(Unix)] your process, and possibly let syscom know too.'''

== ''corn-syrup'' ==

Dell PowerEdge 2950

==== Specs ====

* 2 × Intel Xeon E5405 (2.00 GHz, 4 cores each)
* 32 GB RAM
* eth0 ("Gb0") mac addr 00:24:e8:52:41:27
* eth1 ("Gb1") mac addr 00:24:e8:52:41:29
* IPMI mac addr 00:24:e8:52:41:2b
* 3 × Western-Digital 160GB SATA hard drive (445 GB software RAID0 array)

==== Notes ====

* Use eth0/Gb0 for the mathstudentorgsnet connection

==== Services ====

* Hosts 1 TB <tt>[[scratch|/scratch]]</tt> and exports via NFS (sec=krb5)

== ''high-fructose-corn-syrup'' ==

High-fructose-corn-syrup (or hfcs) is a large SuperMicro server. It's been in CSC service since April 2012.

==== Specs ====

* 4x AMD Opteron 6272 (2.4 GHz, 16 cores each)
* 192 GB RAM
* Supermicro H8QGi+-F Motherboard Quad 1944-pin Socket [http://csclub.uwaterloo.ca/misc/manuals/motherboard-H8QGI+-F.pdf (Manual)]
* 500 GB Seagate Barracuda
* Supermicro Case Rackmount CSE-748TQ-R1400B 4U [http://csclub.uwaterloo.ca/misc/manuals/SC748.pdf (Manual)]
'''Notes'''

* Missing moba IO shield (as of January 2024)

== ''carbonated-water'' ==

carbonated-water is a Dell R815 provided by CSCF.

'''01/19/23: IPMI (temporarily) disconnected. (Require new patch cable)'''

==== Specs ====

* 4x AMD Opteron 6176 processors (2.3 GHz, 12 cores each)
* 128GB RAM

== ''neotame'' ==

neotame is a SuperMicro server funded by MEF. It is the successor to taurine.

'''We strongly discourage running computationally-intensive jobs''' on neotame as many users run interactive applications such as IRC clients on it and any significant service degradation will be more likely to affect other users (who will probably notice right away).

==== Specs ====

* 2x Intel Xeon E5-2630 v4 processors (2.2 GHz, 10 cores/20 threads each)
* 64GB RAM
'''Notes'''
* SSH server also listens on ports 21, 22, 53, 80, 81, 443, 8000, 8080 for your convenience.

== ''sorbitol'' ==

sorbitol is a SuperMicro server funded by MEF.

==== Specs ====

* 2x Intel Xeon E5-2630 v4 processors (2.2 GHz, 10 cores/20 threads each)
* 64GB RAM
== ''mannitol'' ==

mannitol is a SuperMicro server funded by MEF.

==== Specs ====

* 2x Intel Xeon E5-2630 v4 processors (2.2 GHz, 10 cores/20 threads each)
* 64GB RAM

= Office Terminals =

It's possible to SSH into these machines, but we discourage you from trying to use these machines when you're not sitting in front of them. They are bounced at least every time our login manager, lightdm, throws a tantrum (which is several times a day). These are for use inside our physical office.

== ''cyanide'' ==
cyanide is a [https://support.apple.com/kb/sp710 Mac Mini (Late 2014)], identical in specification to powernap

=== Spec ===

* Intel i7-4578U (4) @ 3.500GHz
* 16GB RAM
* Intel Iris Graphics 5100
* 256GB On-board SSD

== ''suika'' ==

Suika is an office terminal built from various components donated by our members.

==== Specs ====

* AMD Ryzen 7 2700X
* 2x 8GB DDR4
* 1x Samsung 256GB SSD
* AMD Radeon RX 550 4GB

==''powernap''==
powernap is a [https://support.apple.com/kb/sp710 Mac Mini (Late 2014)].

=== Spec ===

* Intel i7-4578U (4) @ 3.500GHz
* 16GB RAM
* Intel Iris Graphics 5100
* 256GB On-board SSD

=== Speaker ===
powernap has the office speakers (a pair of nice studio monitors) currently connected to it.

=== Services ===
* MPD for playing music. Only office/termcom/syscom can log into powernap. Use `ncmpcpp` to control MPD.
** TODO: this is not the case anymore
* Bluetooth audio receiver. Only syscom can control bluetooth pairing. Use `bluetoothctl` to control bluetooth.

Music is located in `/music` on the office terminals.

= Progcom Only =
The Programme Committee has access to a VM on corn-syrup called 'progcom'. They have sudo rights in this VM so they may install and run their own software inside it. This VM should only be accessible by members of progcom or syscom.

= Codey Bot Only =
Ran on CSC Cloud in a separate Cloudstack project. codey-staging, codey-dev, codey-prod.

TODO: migrating from cloudstack

= Syscom Only =

The following systems are only be accessible to members of the [[Systems Committee]] for a variety of reasons; the most common of which being that some of these machines host [[Kerberos]] authentication services for the CSC.

==''xylitol''==

xylitol is a Dell PowerEdge R815 donated by CSCF. It is primarily a container host for services previously hosted on aspartame and dextrose, including munin, rt, mathnews, auth1, and dns1. It was provisioned with the intent to replace both of those hosts.

==== Specs ====

* Dual AMD Opteron 6176 (2.3 GHz, 48 cores total)
* 128GB RAM
* 500GB volume group on RAID1 SSD (xylitol-mirrored)
* 500ish-GB volume group on RAID10 HDD (xylitol-raidten)

==''auth1''==

Container on [[#xylitol|xylitol]].

==== Services ====

*[[LDAP]] primary
*[[Kerberos]] primary

==''chat''==

Container on [[#xylitol|xylitol]].

==== Services ====

* The Lounge web IRC client (https://chat.csclub.uwaterloo.ca)

==''phosphoric-acid''==

phosphoric-acid is a Dell PowerEdge R815 donated by CSCF and is a clone of xylitol. It may be used to provide redundant cloud services in the future.

'''01/19/23: IPMI (temporarily) disconnected. (Require new patch cable)'''

==== Specs ====

* (clone of Xylitol)

==== Services ====

*[[#caffeine|caffeine]]
*[[#coffee|coffee]]

==''coffee''==

Virtual machine running on phosphoric-acid.

==== Services ====

*[[Database#MySQL|MySQL]]
*[[Database#Postgres|Postgres]]

==''cobalamin''==

Dell PowerEdge 2950 donated to us by FEDS. Located in the Science machine room on the first floor of Physics. Will act as a backup server for many things. TODO: should replace with another Syscom server...

==== Specs ====

* 1 × Intel Xeon E5420 (2.50 GHz, 4 cores)
* 16GB RAM
* Broadcom NetworkXtreme II
* 2x73GB Hard Drives, hardware RAID1
** Soon to be 2x1TB in MegaRAID1
*http://www.dell.com/support/home/ca/en/cabsdt1/product-support/servicetag/51TYRG1/configuration

==== Services ====

* Containers: [[#auth2|auth2]] (kerberos)

==== Notes ====

* '''TODO: Mega unreliable.''' (Goes down once every few weeks...)
* The network card requires non-free drivers. Be sure to use an installation disc with non-free.

* We have separate IP ranges for cobalamin and its containers because the machine is located in a different building. They are:

** VLAN ID 506 (csc-data1): 129.97.18.16/29; gateway 129.97.18.17; mask 255.255.255.240
** VLAN ID 504 (csc-ipmi): 172.19.5.24/29; gateway 172.19.5.25; mask 255.255.255.248

* For some reason, the keyboard is shit. Try to avoid having to use it. It's doable, but painful. IPMI works now, and then we don't need to bug about physical access so it's better anyway.
** We don't have "physical" access to the PHYS server rooms.

==''auth2''==

Container on [[#cobalamin|cobalamin]].

==== Services ====

*[[LDAP]] secondary
*[[Kerberos]] secondary

MAC Address: c2:c0:00:00:00:a2

==''mail''==

mail is the CSC's mail server. It hosts mail delivery, imap(s), smtp(s), and mailman. It is also syscom-only. It is a [[Virtualization#Linux_Containers|Linux container]] at present.

TODO: "HA"-ish configuration

==== Specs ====

* currently hosted on [[#xylitol|xylitol]]

==== Services ====

*[[Mail]] services
* mailman (web interface at [http://mailman.csclub.uwaterloo.ca/])
*[[Webmail]]
*[[ceo]] daemon

==''sodium-benzoate''==

Sodium-benzoate is our previous mirror server, funded by MEF.

It is currently sitting in the office pending repurposing. Will likely become a machine for backups in DC.

==== Specs ====

* Intel Xeon Quad Core E5405 @ 2.00 GHz
* 16GB RAM
* vg0: 228 GB block device behind DELL PERC 6/i (contains root partition)

Space disks are currently in the office underneath maltodextrin.

TODO: gone??

==''potassium-benzoate''==

potassium-benzoate is our mirror server, funded by MEF.

==== Specs ====

* 36 drive Supermicro chassis (SSG-6048R-E1CR36L)
* 1 x Intel Xeon E5-2630 v3 (8 cores, 2.40 GHz)
* 64 GB (4 x 16GB) of DDR4 (2133Mhz) ECC RAM
* 2 x 1 TB Samsung Evo 850 SSD drives
* 17 x 4 TB Western Digital Gold drives (separate funding from MEF)
* 9 x 18TB Seagate Exos X18 (8 ZFS, Z2,1 hot-spare)
* 10 Gbps SFP+ card (loaned from CSCF)
* 50 Gbps Mellanox QSFP card (from ginkgo; currently unconnected)

==== Network Connections ====

potassium-benzoate has two connections to our network:

* 1 Gbps to our switch (used for management)
* 2 x 10 Gbps (LACP bond) to mc-rt-3015-mso-a (for mirror)

Mirror's bandwidth is limited to 1 Gbps on each of the 4 campus internet links. Mirror's bandwidth is not limited on campus.

==== Services ====

*[[Mirror]]
*[[Talks]] mirror
*[[Debian_Repository|CSClub packages repository]]

==''munin''==

munin is a syscom-only monitoring and accounting machine. It is a [[Virtualization#Linux_Containers|Linux container]] at present.

==== Specs ====

* currently hosted on [[#xylitol|xylitol]]

==== Services ====

*[http://munin.csclub.uwaterloo.ca munin] systems monitoring daemon
TODO: Debian 9?

==''yerba-mate''==

A Dell PowerEdge 2950 donated by a CSC member.

==== Specs ====

* 2x 3.00 GHz quad core Intel Xeon 5160
* 32GB RAM
* 2x75GB 15k drives (RAID 1)

==== Services ====

* test-ipv6 (test-ipv6.csclub.uwaterloo.ca; a test-ipv6.com mirror)
'''Notes'''

* Also used for experimenting new CSC services.

* TODO: use as backup server

==''citric-acid''==
A Dell PowerEdge R815 (TODO: check model) provided by CSCF to replace [[Machine List#aspartame|aspartame]].

'''Specs'''

* 2 x AMD Opteron 6174 (12 cores, 2.20 GHz)
* 128 GB RAM

'''Services'''

* Configured for [https://pass.uwaterloo.ca pass.uwaterloo.ca], a university-wide password manager hosted by CSC as a demo service for all Nexus (ADFS) user.
* [[Plane]], an internal (CSC) project management tool.
* Minio
'''Notes'''

* Being repurposed for Termcom training and development.
* TODO: migrate Vaultwarden (https://pass.csclub.uwaterloo.ca/)??
* UFW opened-ports: SSH, HTTP/HTTPS
* Upgraded to Podman 4.x

= Cloud =

These machines are used by [https://cloud.csclub.uwaterloo.ca cloud.csclub.uwaterloo.ca]. The machines themselves are restricted to Syscom only access.

==''chamomile''==

A Dell PowerEdge R815 provided by CSCF.

==== Specs ====

* 4x 2.20GHz 12-core processors (AMD Opteron(tm) Processor 6174)
* 128GB RAM
* 10GbE connection to core router

==== Services ====

* Cloudstack host
* TODO: cloudstack migration

==''riboflavin''==

A Dell PowerEdge R515 provided by CSCF.

==== Specs ====

* 2x 2.6 GHz 8-core processors (AMD Opteron(tm) Processor 4376 HE)
* 64GB RAM
* 10GbE connection to core router
* 2x 500GB internal SSD
* 12x Seagate 4TB SSHD
==== Services ====

* OpenStack block and object storage for csclub.cloud
* ????
'''Notes'''

* TODO: cloudstack migration

==''guayusa''==

A Dell PowerEdge 2950 donated by a CSC member.

==== Specs ====

* 2x 3.00 GHz quad core Intel Xeon 5160
* 32GB RAM
* 2TB PCI-Express Flash SSD
* 2x75GB 15k drives (RAID 1)

==== Services ====

* load-balancer-01

Was used to experiment the following then-new CSC services:

* cifs (for booting ginkgo from CD)
* caffeine-01 (testing of multi-node caffeine)
* TODO: ???
** block1.cloud
** object1.cloud
'''Notes'''

* TODO: cloudstack migration
* TODO: ditch... Currently being used to set up NextCloud.

==''ginkgo''==

Supermicro server funded by MEF for CSC web hosting. Locate in MC 3015.

'''01/19/23: IPMI (temporarily) disconnected. (Require new patch cable)'''

==== Specs ====

* 2x Intel Xeon E5-2697 v4 @ 2.30GHz [18 cores each]
* 256GB RAM
* 2 x 1.2 TB SSD (400GB of each for RAID 1)
* 10GbE onboard, 25GbE SFP+ card (also included 50GbE SFP+ card which will probably go in mirror)

==== Services ====

* OpenStack Compute machine

No longer in use:

* controller1.cloud
* db1.cloud
* router1.cloud (NAT for cloud tenant network)
* network1.cloud

'''Notes'''
* TODO: cloudstack migration

==''biloba''==

Supermicro server funded by SLEF for CSC web hosting. Located in DC 3558. TODO: rack??

==== Specs ====

* 2x Intel Xeon Gold 6140 @ 2.30GHz [18 cores each]
* 384GB RAM
* 12 3.5" Hot Swap Drive Bays
** 2 x 480 GB SSD
* 10GbE onboard, 10GbE SFP+ card (on loan from CSCF)

==== Services ====

* OpenStack Compute machine
'''Notes'''
* TODO: cloudstack migration

No longer in use:

* caffeine
* mail
* mattermost

= Storage =

==''fs00''==

fs00 is a '''NetApp FAS3040''' series fileserver donated by CSCF.

It is currently being used for testing of a HA NetApp nodes and serving home directories directly from the NetApp filer.

==== Specs ====

* dual SFP connection to core switch

... TODO

==''fs01''==

fs01 is a '''NetApp FAS3040''' series fileserver donated by CSCF.

It is currently being used for testing of a HA NetApp nodes and serving home directories directly from the NetApp filer.

==== Specs ====
... TODO

TODO: disconnected??

==''fs10''==

fs10 is a '''NetApp FAS8040''' series fileserver donated by CSCF.

It is currently being used for testing of a HA NetApp nodes and serving home directories directly from the NetApp filer.

==== Specs ====

* FAS8040 (dual heads)
** ... TODO
* 6 DS4324 HDD shelves (24-disks each)
** 24 x 2TB HDDs (assorted brands/models)
** Dual IOM3 controllers.
** Loop 1: bottom 4 shelves
** Loop 2: top 2 shelves + SSD shelf
* 1 DS2246 SSD shelf (TODO: right model?)
** 24 Samsung SM1625 SSDs (MZ-6ER2000/0G3), 200GB (SAS 2, 2.5")

= Other =

==''mathnews''==

[[#xylitol|xylitol]] hosts a systemd-nspawn container which serves as the mathNEWS webserver. It is administered by mathNEWS, as a pilot for providing containers to select groups who have more specialized demands than the general-use infrastructure can meet.

== ps3 ==
This is just a very wide PS3, the model that supported running Linux natively before it was removed. Firmware was updated to remove this feature, however it can still be done via. homebrew.

'''Specs'''

* It's a PS3.

'''2022-10-24''' - Thermal paste replaced + firmware updated to latest supported version, also modded.

==''binaerpilot''==

This is a Gumstix Overo Tide CPU on a Tobi expansion board. It is currently attached to corn-syrup in the machine room and even more currently turned off until someone can figure out what is wrong with it.

==== Specs ====

* TI OMAP 3530 750Mhz (ARM Cortex-A8)
* 512MB RAM

==''anamanaguchi''==

This is a Gumstix Overo Tide CPU on a Chestnut43 expansion board. It is currently in the hardware drawer in the CSC.

==== Specs ====

* TI OMAP 3530 750Mhz (ARM Cortex-A8)
* 512MB RAM

==''digital cutter''==

See [[Digital Cutter|here]].

= Decommissioned =

==''aspartame''==

aspartame was a taurine clone donated by CSCF. It was once our primary file server, serving as the gateway interface to space on phlogiston. It also used to host the [[#auth1|auth1]] container, which has been temporarily moved to [[#dextrose|dextrose]]. Decomissioned in March 2021 after refusing to boot following a power outage.

==''psilodump''==

psilodump is a NetApp FAS3000 series fileserver donated by CSCF. It, along with its sibling phlogiston, hosted disk shelves exported as iSCSI block devices.

psilodump was plugged into aspartame. It's still installed but inaccessible.

==''phlogiston''==

phlogiston is a NetApp FAS3000 series fileserver donated by CSCF. It, along with its sibling psilodump, hosted disk shelves exported as iSCSI block devices.

phlogiston is turned off and should remain that way. It is misconfigured to have its drives overlap with those owned by psilodump, and if it is turned on, it will likely cause irreparable data loss.

==== Specs ====

* 2 AMD Opteron 2218 CPUs
* 10GB RAM

==== Notes from before decommissioning ====

* The lxc files are still present and should not be started up, or else the two copies of auth1 will collide.
* It currently cannot route the 10.0.0.0/8 block to a misconfiguration on the NetApp. This should be fixed at some point.

==''glomag''==

Glomag hosted [[#caffeine|caffeine]]. Decommissioned April 6, 2018.

==== Specs ====

* Intel Xeon X3450 @ 2.67 GHz
* 6 GB RAM
* vg0: 465 GB software RAID1 (contains root partition):
** 750 GB Seagate Barracuda SATA hard drive
** 500 GB Western-Digital Caviar Blue SATA hard drive
* vg1: 596 GB software RAID1 (contains caffeine):
** 2 × 640 GB Western-Digital Caviar Blue SATA hard drive

==== Services ====

* Before its decommissioning, glomag hosted [[#caffeine|caffeine]], [[#mail|mail]], and [[#munin|munin]] as [[Virtualization#Linux_Container|Linux containers]]

==''Lisp machine''==

A Symbolics XL1200 Lisp machine. Donated to a new home when we couldn't get it working.

http://www.globalnerdy.com/2008/12/03/symbolics-xl1200-lisp-machine-free-to-a-good-home/ for some history on this hardware.

==== Specs ====

Currently inoperable due to (at least) a missing console cable.

==''ginseng''==

Ginseng used to be our fileserver, before aspartame and the netapp took over.

==== Specs ====

* Intel Pentium Dual Core E2180
* 8GB RAM
*[http://csclub.uwaterloo.ca/misc/manuals/s3000ah_tps_1_1.pdf Intel S3000AHV Motherboard]
* 4 × 640 GB Western-Digital Caviar Blue in [[wikipedia:Nested_RAID_levels#RAID_10_.28RAID_1.2B0.29|RAID 10]] behind a [http://www.3ware.com/products/serial_ata2-9650.asp 3ware 9650SE RAID card].
[[Category:Hardware]]

==''calum''==

Calum used to be our main server and was named after Calum T Dalek. Purchased new by the club in 1994.

==== Specs ====

* SPARCserver 10 (headless SPARCstation 10)

==''paza''==

An iMac G3 that was used as a dumb terminal.

==== Specs ====

* 233Mhz PowerPC 740/750
* 96 MB RAM

==''romana''==

Romana was a BeBox that has been in the CSC's possession since long before BeOS became defunct.

Confirmed on March 19th, 2016 to be fully functional. An SSHv1 compatible client was installed from http://www.abstrakt.ch/be/ and a compatible firewalled daemon was started on Sucrose (living in /root, prefix is /root/ssh-romana). The insecure daemon is to be used a bastion host to jump to hosts only supporting >=SSHv2. The mail daemon on the BeBox has also been configured to send mail through mail.csclub.uwaterloo.ca.

==== Specs ====

* 2 PowerPC based processors
* Stylish Blinken processor-load lights

==''sodium-citrate''==

Sodium-citrate was an SGI O2 machine.

In order to net boot you need to set /proc/sys/net/ipv4/ip_no_pmtu_disc to 1. When the O2 boots, hit F5 at the boot menu and type bootp():.

==== Specs ====
* SGI O2 MIPS processor
* 423 MB (?) RAM
* 2 × 2 GB hard drive

==''acesulfame-potassium''==

An old office terminal.

==== Specs ====
* Intel Pentium 4 2.67GHz
* 1GB RAM
*[http://csclub.uwaterloo.ca/misc/manuals/ABIT_VT7.pdf ABIT VT7] Motherboard
* ATI Radeon 7000

==''skynet''==

skynet was a Sun E6500 machine donated by Sanjay Singh. It was never fully set up.

==== Specs ====

* 15 full CPU/memory boards
** 2x UltraSPARC II 464MHz / 8MB Cache Processors
** ??? RAM?
* 1 I/O board (type=???)
** ???x disks?
* 1 CD-ROM drive

*[http://mirror.csclub.uwaterloo.ca/csclub/sun_e6500/ent6k.srvr/ e6500 documentation (hosted on mirror, currently dead link)]
*[http://docs.oracle.com/cd/E19095-01/ent6k.srvr/ e6500 documentation (backup link)]
*[http://www.e6500.com/ e6500]

==''freebsd''==

FreeBSD was a virtual machine with FreeBSD installed.

==== Services ====

* Newer software

==''rainbowdragoneyes''==

Rainbowdragoneyes was our Lemote Fuloong MIPS machine. This machine is aliased to rde.csclub.uwaterloo.ca.

==== Specs ====

* 800MHz MIPS Loongson 2f CPU

==''denardo''==

Due to some instability, general uselessness, and the acquisition of a more powerful SPARC machine from MFCF, denardo was decommissioned in February 2015.

==== Specs ====

* Sun Fire V210
* TI UltraSparc IIIi (Jalapeño)
* 2 GB RAM
* 160 GB RAID array
* ALOM on denardo-alom.csclub can be used to power machine on/off
==''artificial-flavours''==

Artificial-flavours was our secondary (backup services) server. It used to be an office terminal. It was decommissioned in February 2015 and transferred to the ownership of Women in Computer Science (WiCS).

==== Specs ====

* Intel Celeron 3.2GHz
* 2GB RAM
*[http://csclub.uwaterloo.ca/misc/manuals/Biostar_P4M80-M4.pdf Biostar P4M80-M4] Motherboard
* Western-Digital 80 GB ATA hard drive

==''potassium-citrate''==

Potassium-citrate is a dual-processor Alpha machine. It is on extended loan from pbarfuss.

It is temporarily decommissioned pending the reinstallation of a supported operating system (such as OpenBSD).

==== Specs ====
* Alphaserver CS20 (2 833MHz EV68al CPUs)
* 512MB RAM
* 36 GB Seagate SCSI hard drive

==''potassium-nitrate''==

This was a Sun Fire E2900 from a decommissioned MFCF compute cluster. It had a SPARC architecture and ran OpenBSD, unlike many of our other systems which are x86/x86-64 and Linux/Debian. After multiple unsuccessful attempts to boot a modern Linux kernel and possible hardware instability, it was determined to be non-cost-effective and non-effort-effective to put more work into running this machine. The system was reclaimed by MFCF where someone from CS had better luck running a suitable operating system (probably Solaris).

The name is from saltpetre, because sparks.

==== Specs ====

* 24 CPUs
* 90GB main memory
* 400GB scratch disk local storage in /scratch-potassium-nitrate

There is a [[Sun 2900 Strategy Guide|setup guide]] available for this machine.

See also [[Sun 2900]].

==''taurine''==

'''Note: On August 21, 2019, just before 2:30PM EDT, we were informed that taurine caught fire'''. As a result, taurine has been decommissioned as of Fall 2019.

==== Specs ====

* 2 AMD Opteron 2218 CPUs
* 8GB RAM
* 136 GB LVM volume group

==== Services ====

* Virtual machines
* BitlBee IRC instant messaging gateway (localhost only)
*[[ident]] server to maintain high connection cap to freenode
* Runs ssh on ports 21,22,53,80,81,443,8000,8080 for user's convenience.

==''dextrose''==

dextrose was a [[#taurine|taurine]] clone donated by CSCF and was decommissioned in Fall 2019 after being replaced with a more powerful server.

==''sucrose''==

sucrose was a [[#taurine|taurine]] clone donated by CSCF. It was decommissioned in Fall 2019 following multiple hardware failures.

==''goto80''==
'''Note (2022-10-25): This seems to have gone missing or otherwise left our hands.'''

This was small ARM machine we picked up in order to have similar hardware to the Real Time Operating Systems (CS 452) course. It has a [[TS-7800_JTAG|JTAG]] interface. Located was the office on the top shelf above strombola.

==== Specs ====

* 500 MHz Feroceon (ARM926ej-s compatible) processor
* ARMv5TEJ architecture

Use -march=armv5te -mtune=arm926ej-s options to GCC.

For information on the TS-7800's hardware see here:
http://www.embeddedarm.com/products/board-detail.php?product=ts-7800

==''nullsleep''==

nullsleep is an [http://csclub.uwaterloo.ca/misc/manuals/ASRock_ION_330.pdf ASRock ION 330] machine given to us by CSCF and funded by MEF.

It's decommissioned on 2023-03-20 due to repeated unexpected shutdown. Replaced by [[#powernap|powernap]].

==== Specs ====

* Intel® Dual Core Atom™ 330
* 2GB RAM
* NVIDIA® ION™ graphics
* 1x 64GB SanDisk SDSSDP064G SSD
* DVD Burner

==== Speakers ====
Nullsleep has the office speakers (a pair of nice studio monitors) currently connected to it.

==== Services ====
Nullsleep runs MPD for playing music. Control of MPD is available only to users in the "audio" group.
Music is located in /music on the office terminal

== ''bit-shifter'' ==
bit-shifter was an office terminal, decommissioned April 2023 due to extended age. It was upgraded to the same specs as Strombola at an unknown point in time.

==== Specs ====

* Intel Core 2 Quad CPU Q8300
* 4GB RAM
* Nvidia GeForce GT 440
*[http://csclub.uwaterloo.ca/misc/manuals/motherboard_manual_ga-ep45-ud3l.pdf Gigabyte GA-EP45-UD3L] Motherboard
* 1x 64GB SanDisk SDSSDP064G SSD
* Jacob Parker's Firewire Card

==== Services ====

*[http://csclub.uwaterloo.ca/office/webcam Office webcam]

== ''strombola''==
Strombola was an office terminal named after Gordon Strombola. It was retired in April 2023.

==== Specs ====
* Intel Pentium G4600 2 cores @ 3.6Ghz
* 8 GB RAM
* 1x 64GB SanDisk SDSSDP064G SSD

==== Speakers ====
Strombola used to have integrated 5.1 channel sound before we got new speakers and moved audio stuff to nullsleep.

== ''gwem'' ==
gwem was an office terminal that was created because AMD donated a graphics card. It entered CSC service in February 2012.

=== Specs ===

* AMD FX-8150 3.6GHz 8-Core CPU
* 16 GB RAM
* AMD Radeon 6870 HD 1GB GPU
*[http://csclub.uwaterloo.ca/misc/manuals/ga-990fxa-ud7_e.pdf Gigabyte GA-990FXA-UD7] Motherboard
* 1x 64GB SanDisk SDSSDP064G SSD

== ''maltodextrin'' ==
(*specs are outdated at least as of 2023-05-27*)
*[http://csclub.uwaterloo.ca/misc/manuals/motherboard_manual_ga-ep45-ud3l.pdf Gigabyte GA-EP45-UD3L] Motherboard
Maltodextrin was an office terminal. It was upgraded in Spring 2014 after an unidentified failure. Not operational (no video output) as of July 2022.

==== Specs ====

* Intel Core i3-4130 @ 3.40 GHz
* 8GB RAM
* 1x 64GB SanDisk SDSSDP064G SSD
*[http://csclub.uwaterloo.ca/misc/manuals/E8425_H81I_PLUS.pdf ASUS H81-PLUS] Motherboard

==== Services ====

*[http://csclub.uwaterloo.ca/office/webcam Office webcam]

== ''natural-flavours'' ==

Natural-flavours is an office terminal; it used to be our mirror.

In Fall 2016, it received a major upgrade thanks the MathSoc's Capital Improvement Fund.

==== Specs ====

* Intel Core i7-6700k
* 2x8GB RAM
* 1x 64GB SanDisk SDSSDP064G SSD
* Cup Holder (DVD drive has power, but not connected to mother board)
= UPS =

All of the machines in the MC 3015 machine room are connected to one of our UPSs.

All of our UPSs can be monitored via CSCF:

* MC3015-UPS-B2
* mc-3015-e7-ups-1.cs.uwaterloo.ca (rbc55, batteries replaced July 2014) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-e7-ups-1&var-Interval=30m)
* mc-3015-f7-ups-1.cs.uwaterloo.ca (rbc55, batteries replaced Feb 2017) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-f7-ups-1&var-Interval=30m)
* mc-3015-g7-ups-1.cs.uwaterloo.ca (su5000t, batteries replaced 2010) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-g7-ups-1&var-Interval=30m)
* mc-3015-g7-ups-2.cs.uwaterloo.ca (unknown) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-g7-ups-2&var-Interval=30m)
* mc-3015-h7-ups-1.cs.uwaterloo.ca (su5000t, batteries replaced 2004) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-h7-ups-1&var-Interval=30m)
* mc-3015-h7-ups-2.cs.uwaterloo.ca (unknown) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-h7-ups-2&var-Interval=30m)

We will receive email alerts for any issues with the UPS. Their status can be monitored via [[SNMP]].

TODO: Fix labels & verify info is correct & figure out why we can't talk to cacti.

Meeting:Termcom/Sunday 11 February 2024

2024-02-16T07:18:18Z

O32patel: Task assignments

'''Updates'''

- CEO updates

- Netapp migration

- Hardware acquisition

'''Tasks'''

- repurposing citric-acid (Ohm)

- mirror checker deployment (Jonathan)

- hardware acquisition (Nathan)

- CSC Cloud migration (Frank)

- hardware inventory (Nathan, Frank)

- Netapp migration (Leon)

- librarian api (postponed/Ohm)

Machine List

2024-02-12T03:00:46Z

O32patel: citric-acid has 2 sockets

Most of our machines are in the E7, F7, G7 and H7 racks (as of Jan. 2022) in the MC 3015 server room. There is an additional rack in the DC 3558 machine room on the third floor. Our office terminals are in the CSC office, in MC 3036/3037.

= Web Server =
You are highly encouraged to avoid running anything that's not directly related to your CSC webspace on our web server. We have plenty of general-use machines; please use those instead. You can even edit web pages from any other machine--usually the only reason you'd *need* to be on caffeine is for database access.

== ''caffeine'' ==

Caffeine is the Computer Science Club's web server. It serves websites, databases for websites, and a large amount of other services.

==== Specs ====

* currently a virtual machine hosted on phosphoric-acid
** 12 vCPUs
** 32GB of RAM

==== Services ====

* Club and member web sites with [[Apache]]
* [[MySQL]] databases
* [[PostgreSQL]] databases
* [[ceo]] daemon
* mail was migrated to [[#mail|mail]]

= General-Use Servers =

These machines can be used for (nearly) anything you like (though be polite and remember that these are shared machines). Recall that when you signed the Machine Usage Agreement, you promised not to use these machines to generate profit (so no cryptocurrency mining).

For computationally-intensive jobs (CPU/memory bound) we recommend running on high-fructose-corn-syrup, carbonated-water, sorbitol, mannitol, or corn-syrup, listed in roughly decreasing order of available resources. For low-intensity interactive jobs, such as IRC clients, we recommend running on neotame. If you have a long-running computationally intensive job, it's good to nice[https://en.wikipedia.org/wiki/Nice_(Unix)] your process, and possibly let syscom know too.

== ''corn-syrup'' ==

PowerEdge 2950

==== Specs ====

* 2 × Intel Xeon E5405 (2.00 GHz, 4 cores each)
* 32 GB RAM
* eth0 ("Gb0") mac addr 00:24:e8:52:41:27
* eth1 ("Gb1") mac addr 00:24:e8:52:41:29
* IPMI mac addr 00:24:e8:52:41:2b
* 3 × Western-Digital 160GB SATA hard drive (445 GB software RAID0 array)

==== Notes ====

* Use eth0/Gb0 for the mathstudentorgsnet connection
* has ipmi on corn-syrup-ipmi.csclub.uwaterloo.ca.

==== Services ====

* Hosts 1 TB <tt>[[scratch|/scratch]]</tt> and exports via NFS (sec=krb5)

== ''high-fructose-corn-syrup'' ==

High-fructose-corn-syrup (or hfcs) is a large SuperMicro server. It's been in CSC service since April 2012.

==== Specs ====

* 4x AMD Opteron 6272 (2.4 GHz, 16 cores each)
* 192 GB RAM
* Supermicro H8QGi+-F Motherboard Quad 1944-pin Socket [http://csclub.uwaterloo.ca/misc/manuals/motherboard-H8QGI+-F.pdf (Manual)]
* 500 GB Seagate Barracuda
* Supermicro Case Rackmount CSE-748TQ-R1400B 4U [http://csclub.uwaterloo.ca/misc/manuals/SC748.pdf (Manual)]

== ''carbonated-water'' ==

carbonated-water is a Dell R815 provided by CSCF.

'''01/19/23: IPMI (temporarily) disconnected.'''

==== Specs ====

* 4x AMD Opteron 6176 processors (2.3 GHz, 12 cores each)
* 128GB RAM

== ''neotame'' ==

neotame is a SuperMicro server funded by MEF. It is the successor to taurine.

We discourage running computationally-intensive jobs on neotame as many users run interactive applications such as IRC clients on it and any significant service degradation will be more likely to affect other users (who will probably notice right away).

==== Specs ====

* 2x Intel Xeon E5-2630 v4 processors (2.2 GHz, 10 cores/20 threads each)
* 64GB RAM
* SSH server also listens on ports 21, 22, 53, 80, 81, 443, 8000, 8080 for your convenience.

== ''sorbitol'' ==

sorbitol is a SuperMicro server funded by MEF.

==== Specs ====

* 2x Intel Xeon E5-2630 v4 processors (2.2 GHz, 10 cores/20 threads each)
* 64GB RAM

== ''mannitol'' ==

mannitol is a SuperMicro server funded by MEF.

==== Specs ====

* 2x Intel Xeon E5-2630 v4 processors (2.2 GHz, 10 cores/20 threads each)
* 64GB RAM

= Office Terminals =

It's possible to SSH into these machines, but we discourage you from trying to use these machines when you're not sitting in front of them. They are bounced at least every time our login manager, lightdm, throws a tantrum (which is several times a day). These are for use inside our physical office.

== ''cyanide'' ==
(Work in progress)

cyanide is a Mac Mini

== ''natural-flavours'' ==

Natural-flavours is an office terminal; it used to be our mirror.

In Fall 2016, it received a major upgrade thanks the MathSoc's Capital Improvement Fund.

==== Specs ====

* Intel Core i7-6700k
* 2x8GB RAM
* 1x 64GB SanDisk SDSSDP064G SSD
* Cup Holder (DVD drive has power, but not connected to mother board)

==''powernap''==
powernap is a [https://support.apple.com/kb/sp710 Mac Mini (Late 2014)].

=== Spec ===

* Intel i7-4578U (4) @ 3.500GHz
* 16GB RAM
* Intel Iris Graphics 5100
* 256GB On-board SSD

=== Speaker ===
powernap has the office speakers (a pair of nice studio monitors) currently connected to it.

=== Services ===
* MPD for playing music. Only office/termcom/syscom can log into powernap. Use `ncmpcpp` to control MPD.
* Bluetooth audio receiver. Only syscom can control bluetooth pairing. Use `bluetoothctl` to control bluetooth.

Music is located in /music on the office terminals.

= Progcom Only =
The Programme Committee has access to a VM on corn-syrup called 'progcom'. They have sudo rights in this VM so they may install and run their own software inside it. This VM should only be accessible by members of progcom or syscom.

= Syscom Only =

The following systems may only be accessible to members of the [[Systems Committee]] for a variety of reasons; the most common of which being that some of these machines host [[Kerberos]] authentication services for the CSC.

==''xylitol''==

xylitol is a Dell PowerEdge R815 donated by CSCF. It is primarily a container host for services previously hosted on aspartame and dextrose, including munin, rt, mathnews, auth1, and dns1. It was provisioned with the intent to replace both of those hosts.

'''01/19/23: IPMI (temporarily) disconnected.'''

==== Specs ====

* Dual AMD Opteron 6176 (2.3 GHz, 48 cores total)
* 128GB RAM
* 500GB volume group on RAID1 SSD (xylitol-mirrored)
* 500ish-GB volume group on RAID10 HDD (xylitol-raidten)

==''auth1''==

Container on [[#xylitol|xylitol]].

==== Services ====

*[[LDAP]] primary
*[[Kerberos]] primary

==''chat''==

Container on [[#xylitol|xylitol]].

==== Services ====

* The Lounge web IRC client (https://chat.csclub.uwaterloo.ca)

==''phosphoric-acid''==

phosphoric-acid is a Dell PowerEdge R815 donated by CSCF and is a clone of xylitol. It may be used to provide redundant cloud services in the future.

'''01/19/23: IPMI (temporarily) disconnected.'''

==== Specs ====

* (clone of Xylitol)

==== Services ====

*[[#caffeine|caffeine]]
*[[#coffee|coffee]]

==''coffee''==

Virtual machine running on phosphoric-acid.

==== Services ====

*[[Database#MySQL|MySQL]]
*[[Database#Postgres|Postgres]]

==''cobalamin''==

Dell PowerEdge 2950 donated to us by FEDS. Located in the Science machine room on the first floor of Physics. Will act as a backup server for many things.

==== Specs ====

* 1 × Intel Xeon E5420 (2.50 GHz, 4 cores)
* 16GB RAM
* Broadcom NetworkXtreme II
* 2x73GB Hard Drives, hardware RAID1
** Soon to be 2x1TB in MegaRAID1
*http://www.dell.com/support/home/ca/en/cabsdt1/product-support/servicetag/51TYRG1/configuration

==== Services ====

* Containers: [[#auth2|auth2]]

==== Notes ====

* The network card requires non-free drivers. Be sure to use an installation disc with non-free.

* We have separate IP ranges for cobalamin and its containers because the machine is located in a different building. They are:

** VLAN ID 506 (csc-data1): 129.97.18.16/29; gateway 129.97.18.17; mask 255.255.255.240
** VLAN ID 504 (csc-ipmi): 172.19.5.24/29; gateway 172.19.5.25; mask 255.255.255.248

* For some reason, the keyboard is shit. Try to avoid having to use it. It's doable, but painful. IPMI works now, and then we don't need to bug about physical access so it's better anyway.

==''auth2''==

Container on [[#cobalamin|cobalamin]].

==== Services ====

*[[LDAP]] secondary
*[[Kerberos]] secondary

MAC Address: c2:c0:00:00:00:a2

==''mail''==

mail is the CSC's mail server. It hosts mail delivery, imap(s), smtp(s), and mailman. It is also syscom-only. It is a [[Virtualization#Linux_Containers|Linux container]] at present.

==== Specs ====

* currently hosted on [[#xylitol|xylitol]]

==== Services ====

*[[Mail]] services
* mailman (web interface at [http://mailman.csclub.uwaterloo.ca/])
*[[Webmail]]
*[[ceo]] daemon

==''sodium-benzoate''==

Sodium-benzoate is our previous mirror server, funded by MEF.

It is currently sitting in the office pending repurposing. Will likely become a machine for backups in DC.

==== Specs ====

* Intel Xeon Quad Core E5405 @ 2.00 GHz
* 16GB RAM
* vg0: 228 GB block device behind DELL PERC 6/i (contains root partition)

Space disks are currently in the office underneath maltodextrin.

==''potassium-benzoate''==

potassium-benzoate is our mirror server, funded by MEF.

==== Specs ====

* 36 drive Supermicro chassis (SSG-6048R-E1CR36L)
* 1 x Intel Xeon E5-2630 v3 (8 cores, 2.40 GHz)
* 64 GB (4 x 16GB) of DDR4 (2133Mhz) ECC RAM
* 2 x 1 TB Samsung Evo 850 SSD drives
* 17 x 4 TB Western Digital Gold drives (separate funding from MEF)
* 9 x 18TB Seagate Exos X18 (8 ZFS, Z2,1 hot-spare)
* 10 Gbps SFP+ card (loaned from CSCF)
* 50 Gbps Mellanox QSFP card (from ginkgo; currently unconnected)

==== Network Connections ====

potassium-benzoate has two connections to our network:

* 1 Gbps to our switch (used for management)
* 2 x 10 Gbps (LACP bond) to mc-rt-3015-mso-a (for mirror)

Mirror's bandwidth is limited to 1 Gbps on each of the 4 campus internet links. Mirror's bandwidth is not limited on campus.

==== Services ====

*[[Mirror]]
*[[Talks]] mirror
*[[Debian_Repository|CSClub packages repository]]

==''munin''==

munin is a syscom-only monitoring and accounting machine. It is a [[Virtualization#Linux_Containers|Linux container]] at present.

==== Specs ====

* currently hosted on [[#xylitol|xylitol]]

==== Services ====

*[http://munin.csclub.uwaterloo.ca munin] systems monitoring daemon
==''yerba-mate''==

A Dell PowerEdge 2950 donated by a CSC member.

==== Specs ====

* 2x 3.00 GHz quad core Intel Xeon 5160
* 32GB RAM
* 2x75GB 15k drives (RAID 1)

==== Services ====

* test-ipv6 (test-ipv6.csclub.uwaterloo.ca; a test-ipv6.com mirror)
* shibboleth (under development)

Also used for experimenting new CSC services.

==''citric-acid''==
A Dell PowerEdge provided by CSCF to replace [[Machine List#aspartame|aspartame]].

'''Specs'''

* 2 x AMD Opteron 6174 (12 cores, 2.20 GHz)
* 128 GB RAM

'''Services'''

* Being configured for [https://pass.uwaterloo.ca pass.uwaterloo.ca], a university-wide password manager hosted by CSC as a demo service for all Nexus (ADFS) users

= Cloud =

These machines are used by [https://cloud.csclub.uwaterloo.ca cloud.csclub.uwaterloo.ca]. The machines themselves are restricted to Syscom only access.

==''chamomile''==

A Dell PowerEdge R815 provided by CSCF.

==== Specs ====

* 4x 2.20GHz 12-core processors (AMD Opteron(tm) Processor 6174)
* 128GB RAM
* 10GbE connection to core router

==== Services ====

* OpenStack primary controller services for csclub.cloud

==''riboflavin''==

A Dell PowerEdge R515 provided by CSCF.

==== Specs ====

* 2x 2.6 GHz 8-core processors (AMD Opteron(tm) Processor 4376 HE)
* 64GB RAM
* 10GbE connection to core router
* 2x 500GB internal SSD
* 12x Seagate 4TB SSHD

==== Services ====

* OpenStack block and object storage for csclub.cloud

==''guayusa''==

A Dell PowerEdge 2950 donated by a CSC member.

==== Specs ====

* 2x 3.00 GHz quad core Intel Xeon 5160
* 32GB RAM
* 2TB PCI-Express Flash SSD
* 2x75GB 15k drives (RAID 1)

==== Services ====

Currently being used to set up NextCloud.

Was used to experiment the following then-new CSC services:

* logstash (testing of logstash)
* load-balancer-01
* cifs (for booting ginkgo from CD)
* caffeine-01 (testing of multi-node caffeine)
* block1.cloud
* object1.cloud

==''ginkgo''==

Supermicro server funded by MEF for CSC web hosting. Locate in MC 3015.

'''01/19/23: IPMI (temporarily) disconnected.'''

==== Specs ====

* 2x Intel Xeon E5-2697 v4 @ 2.30GHz [18 cores each]
* 256GB RAM
* 2 x 1.2 TB SSD (400GB of each for RAID 1)
* 10GbE onboard, 25GbE SFP+ card (also included 50GbE SFP+ card which will probably go in mirror)

==== Services ====

* OpenStack Compute machine

No longer in use:

* controller1.cloud
* db1.cloud
* router1.cloud (NAT for cloud tenant network)
* network1.cloud

==''biloba''==

Supermicro server funded by SLEF for CSC web hosting. Located in DC 3558.

==== Specs ====

* 2x Intel Xeon Gold 6140 @ 2.30GHz [18 cores each]
* 384GB RAM
* 12 3.5" Hot Swap Drive Bays
** 2 x 480 GB SSD
* 10GbE onboard, 10GbE SFP+ card (on loan from CSCF)

==== Services ====

* OpenStack Compute machine

No longer in use:

* caffeine
* mail
* mattermost

= Storage =

==''fs00''==

fs00 is a NetApp FAS3040 series fileserver donated by CSCF.

It is currently being used for testing of a HA NetApp nodes and serving home directories directly from the NetApp filer.

==== Specs ====

==''fs01''==

fs01 is a NetApp FAS3040 series fileserver donated by CSCF.

It is currently being used for testing of a HA NetApp nodes and serving home directories directly from the NetApp filer.

==== Specs ====

= Other =

== ps3 ==
This is just a very wide PS3, the model that supported running Linux natively before it was removed. Firmware was updated to remove this feature, however it can still be done via. homebrew.

'''Specs'''

* It's a PS3.

'''2022-10-24''' - Thermal paste replaced + firmware updated to latest supported version, also modded.

==''binaerpilot''==

This is a Gumstix Overo Tide CPU on a Tobi expansion board. It is currently attached to corn-syrup in the machine room and even more currently turned off until someone can figure out what is wrong with it.

==== Specs ====

* TI OMAP 3530 750Mhz (ARM Cortex-A8)
* 512MB RAM

==''anamanaguchi''==

This is a Gumstix Overo Tide CPU on a Chestnut43 expansion board. It is currently in the hardware drawer in the CSC.

==== Specs ====

* TI OMAP 3530 750Mhz (ARM Cortex-A8)
* 512MB RAM

==''digital cutter''==

See [[Digital Cutter|here]].

==''mathnews''==

[[#xylitol|xylitol]] hosts a systemd-nspawn container which serves as the mathNEWS webserver. It is administered by mathNEWS, as a pilot for providing containers to select groups who have more specialized demands than the general-use infrastructure can meet.

= Decommissioned =

==''aspartame''==

aspartame was a taurine clone donated by CSCF. It was once our primary file server, serving as the gateway interface to space on phlogiston. It also used to host the [[#auth1|auth1]] container, which has been temporarily moved to [[#dextrose|dextrose]]. Decomissioned in March 2021 after refusing to boot following a power outage.

==''psilodump''==

psilodump is a NetApp FAS3000 series fileserver donated by CSCF. It, along with its sibling phlogiston, hosted disk shelves exported as iSCSI block devices.

psilodump was plugged into aspartame. It's still installed but inaccessible.

==''phlogiston''==

phlogiston is a NetApp FAS3000 series fileserver donated by CSCF. It, along with its sibling psilodump, hosted disk shelves exported as iSCSI block devices.

phlogiston is turned off and should remain that way. It is misconfigured to have its drives overlap with those owned by psilodump, and if it is turned on, it will likely cause irreparable data loss.

==== Specs ====

* 2 AMD Opteron 2218 CPUs
* 10GB RAM

==== Notes from before decommissioning ====

* The lxc files are still present and should not be started up, or else the two copies of auth1 will collide.
* It currently cannot route the 10.0.0.0/8 block to a misconfiguration on the NetApp. This should be fixed at some point.

==''glomag''==

Glomag hosted [[#caffeine|caffeine]]. Decommissioned April 6, 2018.

==== Specs ====

* Intel Xeon X3450 @ 2.67 GHz
* 6 GB RAM
* vg0: 465 GB software RAID1 (contains root partition):
** 750 GB Seagate Barracuda SATA hard drive
** 500 GB Western-Digital Caviar Blue SATA hard drive
* vg1: 596 GB software RAID1 (contains caffeine):
** 2 × 640 GB Western-Digital Caviar Blue SATA hard drive

==== Services ====

* Before its decommissioning, glomag hosted [[#caffeine|caffeine]], [[#mail|mail]], and [[#munin|munin]] as [[Virtualization#Linux_Container|Linux containers]]

==''Lisp machine''==

A Symbolics XL1200 Lisp machine. Donated to a new home when we couldn't get it working.

http://www.globalnerdy.com/2008/12/03/symbolics-xl1200-lisp-machine-free-to-a-good-home/ for some history on this hardware.

==== Specs ====

Currently inoperable due to (at least) a missing console cable.

==''ginseng''==

Ginseng used to be our fileserver, before aspartame and the netapp took over.

==== Specs ====

* Intel Pentium Dual Core E2180
* 8GB RAM
*[http://csclub.uwaterloo.ca/misc/manuals/s3000ah_tps_1_1.pdf Intel S3000AHV Motherboard]
* 4 × 640 GB Western-Digital Caviar Blue in [[wikipedia:Nested_RAID_levels#RAID_10_.28RAID_1.2B0.29|RAID 10]] behind a [http://www.3ware.com/products/serial_ata2-9650.asp 3ware 9650SE RAID card].
[[Category:Hardware]]

==''calum''==

Calum used to be our main server and was named after Calum T Dalek. Purchased new by the club in 1994.

==== Specs ====

* SPARCserver 10 (headless SPARCstation 10)

==''paza''==

An iMac G3 that was used as a dumb terminal.

==== Specs ====

* 233Mhz PowerPC 740/750
* 96 MB RAM

==''romana''==

Romana was a BeBox that has been in the CSC's possession since long before BeOS became defunct.

Confirmed on March 19th, 2016 to be fully functional. An SSHv1 compatible client was installed from http://www.abstrakt.ch/be/ and a compatible firewalled daemon was started on Sucrose (living in /root, prefix is /root/ssh-romana). The insecure daemon is to be used a bastion host to jump to hosts only supporting >=SSHv2. The mail daemon on the BeBox has also been configured to send mail through mail.csclub.uwaterloo.ca.

==== Specs ====

* 2 PowerPC based processors
* Stylish Blinken processor-load lights

==''sodium-citrate''==

Sodium-citrate was an SGI O2 machine.

In order to net boot you need to set /proc/sys/net/ipv4/ip_no_pmtu_disc to 1. When the O2 boots, hit F5 at the boot menu and type bootp():.

==== Specs ====
* SGI O2 MIPS processor
* 423 MB (?) RAM
* 2 × 2 GB hard drive

==''acesulfame-potassium''==

An old office terminal.

==== Specs ====
* Intel Pentium 4 2.67GHz
* 1GB RAM
*[http://csclub.uwaterloo.ca/misc/manuals/ABIT_VT7.pdf ABIT VT7] Motherboard
* ATI Radeon 7000

==''skynet''==

skynet was a Sun E6500 machine donated by Sanjay Singh. It was never fully set up.

==== Specs ====

* 15 full CPU/memory boards
** 2x UltraSPARC II 464MHz / 8MB Cache Processors
** ??? RAM?
* 1 I/O board (type=???)
** ???x disks?
* 1 CD-ROM drive

*[http://mirror.csclub.uwaterloo.ca/csclub/sun_e6500/ent6k.srvr/ e6500 documentation (hosted on mirror, currently dead link)]
*[http://docs.oracle.com/cd/E19095-01/ent6k.srvr/ e6500 documentation (backup link)]
*[http://www.e6500.com/ e6500]

==''freebsd''==

FreeBSD was a virtual machine with FreeBSD installed.

==== Services ====

* Newer software

==''rainbowdragoneyes''==

Rainbowdragoneyes was our Lemote Fuloong MIPS machine. This machine is aliased to rde.csclub.uwaterloo.ca.

==== Specs ====

* 800MHz MIPS Loongson 2f CPU

==''denardo''==

Due to some instability, general uselessness, and the acquisition of a more powerful SPARC machine from MFCF, denardo was decommissioned in February 2015.

==== Specs ====

* Sun Fire V210
* TI UltraSparc IIIi (Jalapeño)
* 2 GB RAM
* 160 GB RAID array
* ALOM on denardo-alom.csclub can be used to power machine on/off
==''artificial-flavours''==

Artificial-flavours was our secondary (backup services) server. It used to be an office terminal. It was decommissioned in February 2015 and transferred to the ownership of Women in Computer Science (WiCS).

==== Specs ====

* Intel Celeron 3.2GHz
* 2GB RAM
*[http://csclub.uwaterloo.ca/misc/manuals/Biostar_P4M80-M4.pdf Biostar P4M80-M4] Motherboard
* Western-Digital 80 GB ATA hard drive

==''potassium-citrate''==

Potassium-citrate is a dual-processor Alpha machine. It is on extended loan from pbarfuss.

It is temporarily decommissioned pending the reinstallation of a supported operating system (such as OpenBSD).

==== Specs ====
* Alphaserver CS20 (2 833MHz EV68al CPUs)
* 512MB RAM
* 36 GB Seagate SCSI hard drive

==''potassium-nitrate''==

This was a Sun Fire E2900 from a decommissioned MFCF compute cluster. It had a SPARC architecture and ran OpenBSD, unlike many of our other systems which are x86/x86-64 and Linux/Debian. After multiple unsuccessful attempts to boot a modern Linux kernel and possible hardware instability, it was determined to be non-cost-effective and non-effort-effective to put more work into running this machine. The system was reclaimed by MFCF where someone from CS had better luck running a suitable operating system (probably Solaris).

The name is from saltpetre, because sparks.

==== Specs ====

* 24 CPUs
* 90GB main memory
* 400GB scratch disk local storage in /scratch-potassium-nitrate

There is a [[Sun 2900 Strategy Guide|setup guide]] available for this machine.

See also [[Sun 2900]].

==''taurine''==

'''Note: On August 21, 2019, just before 2:30PM EDT, we were informed that taurine caught fire'''. As a result, taurine has been decommissioned as of Fall 2019.

==== Specs ====

* 2 AMD Opteron 2218 CPUs
* 8GB RAM
* 136 GB LVM volume group

==== Services ====

* Virtual machines
* BitlBee IRC instant messaging gateway (localhost only)
*[[ident]] server to maintain high connection cap to freenode
* Runs ssh on ports 21,22,53,80,81,443,8000,8080 for user's convenience.

==''dextrose''==

dextrose was a [[#taurine|taurine]] clone donated by CSCF and was decommissioned in Fall 2019 after being replaced with a more powerful server.

==''sucrose''==

sucrose was a [[#taurine|taurine]] clone donated by CSCF. It was decommissioned in Fall 2019 following multiple hardware failures.

==''goto80''==
'''Note (2022-10-25): This seems to have gone missing or otherwise left our hands.'''

This was small ARM machine we picked up in order to have similar hardware to the Real Time Operating Systems (CS 452) course. It has a [[TS-7800_JTAG|JTAG]] interface. Located was the office on the top shelf above strombola.

==== Specs ====

* 500 MHz Feroceon (ARM926ej-s compatible) processor
* ARMv5TEJ architecture

Use -march=armv5te -mtune=arm926ej-s options to GCC.

For information on the TS-7800's hardware see here:
http://www.embeddedarm.com/products/board-detail.php?product=ts-7800

==''nullsleep''==

nullsleep is an [http://csclub.uwaterloo.ca/misc/manuals/ASRock_ION_330.pdf ASRock ION 330] machine given to us by CSCF and funded by MEF.

It's decommissioned on 2023-03-20 due to repeated unexpected shutdown. Replaced by [[#powernap|powernap]].

==== Specs ====

* Intel® Dual Core Atom™ 330
* 2GB RAM
* NVIDIA® ION™ graphics
* 1x 64GB SanDisk SDSSDP064G SSD
* DVD Burner

==== Speakers ====
Nullsleep has the office speakers (a pair of nice studio monitors) currently connected to it.

==== Services ====
Nullsleep runs MPD for playing music. Control of MPD is available only to users in the "audio" group.
Music is located in /music on the office terminal

== ''bit-shifter'' ==
bit-shifter was an office terminal, decommissioned April 2023 due to extended age. It was upgraded to the same specs as Strombola at an unknown point in time.

==== Specs ====

* Intel Core 2 Quad CPU Q8300
* 4GB RAM
* Nvidia GeForce GT 440
*[http://csclub.uwaterloo.ca/misc/manuals/motherboard_manual_ga-ep45-ud3l.pdf Gigabyte GA-EP45-UD3L] Motherboard
* 1x 64GB SanDisk SDSSDP064G SSD
* Jacob Parker's Firewire Card

==== Services ====

*[http://csclub.uwaterloo.ca/office/webcam Office webcam]

== ''strombola''==
Strombola was an office terminal named after Gordon Strombola. It was retired in April 2023.

==== Specs ====
* Intel Pentium G4600 2 cores @ 3.6Ghz
* 8 GB RAM
* 1x 64GB SanDisk SDSSDP064G SSD

==== Speakers ====
Strombola used to have integrated 5.1 channel sound before we got new speakers and moved audio stuff to nullsleep.

== ''gwem'' ==
gwem was an office terminal that was created because AMD donated a graphics card. It entered CSC service in February 2012.

=== Specs ===

* AMD FX-8150 3.6GHz 8-Core CPU
* 16 GB RAM
* AMD Radeon 6870 HD 1GB GPU
*[http://csclub.uwaterloo.ca/misc/manuals/ga-990fxa-ud7_e.pdf Gigabyte GA-990FXA-UD7] Motherboard
* 1x 64GB SanDisk SDSSDP064G SSD

== ''maltodextrin'' ==
(*specs are outdated at least as of 2023-05-27*)
*[http://csclub.uwaterloo.ca/misc/manuals/motherboard_manual_ga-ep45-ud3l.pdf Gigabyte GA-EP45-UD3L] Motherboard
Maltodextrin was an office terminal. It was upgraded in Spring 2014 after an unidentified failure. Not operational (no video output) as of July 2022.

==== Specs ====

* Intel Core i3-4130 @ 3.40 GHz
* 8GB RAM
* 1x 64GB SanDisk SDSSDP064G SSD
*[http://csclub.uwaterloo.ca/misc/manuals/E8425_H81I_PLUS.pdf ASUS H81-PLUS] Motherboard

==== Services ====

*[http://csclub.uwaterloo.ca/office/webcam Office webcam]

= UPS =

All of the machines in the MC 3015 machine room are connected to one of our UPSs.

All of our UPSs can be monitored via CSCF:

* MC3015-UPS-B2
* mc-3015-e7-ups-1.cs.uwaterloo.ca (rbc55, batteries replaced July 2014) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-e7-ups-1&var-Interval=30m)
* mc-3015-f7-ups-1.cs.uwaterloo.ca (rbc55, batteries replaced Feb 2017) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-f7-ups-1&var-Interval=30m)
* mc-3015-g7-ups-1.cs.uwaterloo.ca (su5000t, batteries replaced 2010) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-g7-ups-1&var-Interval=30m)
* mc-3015-g7-ups-2.cs.uwaterloo.ca (unknown) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-g7-ups-2&var-Interval=30m)
* mc-3015-h7-ups-1.cs.uwaterloo.ca (su5000t, batteries replaced 2004) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-h7-ups-1&var-Interval=30m)
* mc-3015-h7-ups-2.cs.uwaterloo.ca (unknown) (https://metrics.cscf.uwaterloo.ca/grafana/dashboard/db/ups-statistics?orgId=1&var-UPS=mc-3015-h7-ups-2&var-Interval=30m)

We will receive email alerts for any issues with the UPS. Their status can be monitored via [[SNMP]].

TODO: Fix labels & verify info is correct & figure out why we can't talk to cacti.