SSL

GlobalSign

The CSC currently has an SSL Certificate from GlobalSign for *.csclub.uwaterloo.ca provided at no cost to us through IST. GlobalSign likes to take a long time to respond to certificate signing requests (CSR) for wildcard certs, so our CSR really needs to be handed off to IST at least 2 weeks in advance. You can do it sooner – the certificate expiry date will be the old expiry date + 1 year (+ a bonus ) Having an invalid cert for any length of time leads to terrible breakage, followed by terrible workarounds and prolonged problems.

When the certificate is due to expire in a month or two, syscom should (but apparently doesn't always) get an email notification. This will include a renewal link. Otherwise, use the IST-CA self service system. Please keep a copy of the key, CSR and (once issued) certificate in /users/sysadmin/certs. The OpenSSL examples linked there are good to generate a 2048-bit RSA key and a corresponding CSR. It's probably a good idea to change the private key (as it's not that much effort anyways). Just sure your CSR is for *.csclub.uwaterloo.ca.

At the self-service portal, these options worked in 2013. If you need IST assistance, ist-ca@uwaterloo.ca is the email address you should contact.

 Products: OrganizationSSL
 SSL Certificate Type: Wildcard SSL Certificate
 Validity Period: 1 year
 Are you switching from a Competitor? No, I am not switching
 Are you renewing this Certificate? Yes (paste current certificate)
 30-day bonus: Yes (why not?)
 Add specific Subject Alternative Names (SANs): No (*.csclub.uwaterloo.ca automatically adds csclub.uwaterloo.ca as a SAN)
 Enter Certificate Signing Request (CSR): Yes (paste CSR)
 Contact Information:
   First Name: Computer Science Club
   Last Name: Systems Committee
   Telephone: +1 519 888 4567 x33870
   Email Address: syscom@csclub.uwaterloo.ca

Helpful links

• How to generate a new CSR and private key
• How to obtain a new GlobalSign certificate or renew an existing one
• GlobalSign UWaterloo self-service page
• GlobalSign intermediate certificate (needed to create a certificate chain; see below)

OpenSSL cheat sheet

• Generate a new CSR and private key (do this in a new directory):

  openssl req -out csclub.uwaterloo.ca.csr -new -newkey rsa:2048 -keyout csclub.uwaterloo.ca.key -nodes
  

  Enter the following information at the prompts:

  Country Name (2 letter code) [AU]:CA
  State or Province Name (full name) [Some-State]:Ontario
  Locality Name (eg, city) []:Waterloo
  Organization Name (eg, company) [Internet Widgits Pty Ltd]:University of Waterloo
  Organizational Unit Name (eg, section) []:Computer Science Club
  Common Name (e.g. server FQDN or YOUR name) []:*.csclub.uwaterloo.ca
  Email Address []:systems-committee@csclub.uwaterloo.ca
  
  Please enter the following 'extra' attributes
  to be sent with your certificate request
  A challenge password []:
  An optional company name []:
  

• View the information inside a CSR:

  openssl req -noout -text -in csclub.uwaterloo.ca.csr
  

• View the information inside a private key:

  openssl pkey -noout -text -in csclub.uwaterloo.ca.key
  

• View information inside a certificate:

  openssl x509 -noout -text -in csclub.uwaterloo.ca.crt
  

csclub.cloud

Once a year, someone from IST will ask us to create a temporary TXT record for csclub.cloud to prove to GlobalSign that we own it. This must be created at the root of the domain. Since this zone is managed dynamically (via the acme.sh script on biloba, see below), we need to freeze the domain and update /var/lib/bind/db.csclub.cloud directly.

Once you're in the correct server (not Biloba). Here are the steps:

1. Run rndc freeze csclub.cloud.
2. Open /var/lib/bind/db.csclub.cloud and add a new TXT record. It'll look something like

   TXT "_globalsign-domain-verification=blablabla"
   

3. In the same file, make sure to also update the SOA serial number. It should generally be YYYYMMDDNN where NN is a monotonically increasing counter (YYYYMMDD is the current date).
4. Run rndc reload.
5. Run a DNS query to make sure you can see the TXT record:

   dig -t txt @dns1 csclub.cloud
   dig -t txt @dns2 csclub.cloud
   

6. Email back the person from IST and let them know that we created the TXT record.
7. Once the certificate has been renewed, delete the TXT record, update the SOA serial number, and run rndc reload.
8. Run rndc thaw csclub.cloud.

Certificate Files

Let's say you obtain a new certificate for *.csclub.uwaterloo.ca. Here are the files which should be stored in the certs folder:

• csclub.uwaterloo.ca.key: private key created by openssl
• csclub.uwaterloo.ca.csr: certificate signing request created by openssl
• order: order number from GlobalSign
• csclub.uwaterloo.ca.crt: certificate created by GlobalSign
• globalsign-intermediate.crt: intermediate certificate from GlobalSign, obtainable from here. As of this writing, we use the "OrganizationSSL SHA-256 R3 Intermediate Certificate". Just click the "View in Base64" button and copy the contents.
  • There is an alternative way to get the intermediate certificate: if you run openssl x509 -noout -text -in csclub.uwaterloo.ca.crt, under X509v3 extensions > Authority Information Access, there should be a field called "CA Issuers" which has a URL which looks like http://secure.globalsign.com/cacert/gsrsaovsslca2018.crt. You can download that file and convert it to PEM:

    wget https://secure.globalsign.com/cacert/gsrsaovsslca2018.crt
    openssl x509 -inform der -in gsrsaovsslca2018.crt -out globalsign-intermediate.crt
    rm gsrsaovsslca2018.crt
    

• csclub.uwaterloo.ca.chain: create this with the following command:

  cat csclub.uwaterloo.ca.crt globalsign-intermediate.crt > csclub.uwaterloo.ca.chain
  

• csclub.uwaterloo.ca.pem: create this with the following command:

  cat csclub.uwaterloo.ca.key csclub.uwaterloo.ca.chain > csclub.uwaterloo.ca.pem
  chmod 600 csclub.uwaterloo.ca.pem
  

Certificate Locations

Keep a copy of newly generated certificates in /users/sysadmin/certs.

A list of places you'll need to put the new certificate to keep our services running. Private key (if applicable) should be kept next to the certificate with the extension .key.

• caffeine:/etc/ssl/private/csclub-wildcard.crt (for Apache)
• coffee:/etc/ssl/private/csclub.uwaterloo.ca (for PostgreSQL and MariaDB)
• mail:/etc/ssl/private/csclub-wildcard.crt (for Apache, Postfix and Dovecot) (UPDATE: we use certbot now for these)
• mailman:/etc/ssl/private/csclub-wildcard-chain.crt (for Apache)
• rt:/etc/ssl/private/csclub-wildcard.crt (for Apache)
• potassium-benzoate:/etc/ssl/private/csclub-wildcard.crt (for nginx)
• phosphoric-acid:/etc/ssl/private/csclub-wildcard-chain.crt (for ceod)
• auth1:/etc/ssl/private/csclub-wildcard.crt (for slapd, make sure to sudo service slapd restart)
• auth2:/etc/ssl/private/csclub-wildcard.crt (for slapd, make sure to sudo service slapd restart)
• mattermost:/etc/ssl/private/csclub-wildcard.crt (for nginx)
• load-balancer-0(1|2):/etc/ssl/private/csclub.uwaterloo.ca (for haproxy) [temporarily down 2020]
• chat:/etc/ssl/private/csclub-wildcard-chain.crt (for nginx)
• prometheus:/etc/ssl/private/csclub-wildcard-chain.crt (for Apache)
• bigbluebutton:/etc/nginx/ssl/csclub-wildcard-chain.crt (podman container on xylitol)
• icy:/etc/ssl/private/csclub-wildcard.pem (for Icecast)
• chamomile:/etc/ssl/private/cloud.csclub.uwaterloo.ca.chain.crt, /etc/ssl/private/csclub.cloud.chain, /etc/ssl/private/csclub.uwaterloo.ca.chain (for nginx)
• biloba:/etc/ssl/private/cloud.csclub.uwaterloo.ca.chain.crt, /etc/ssl/private/csclub.cloud.chain, /etc/ssl/private/csclub.uwaterloo.ca.chain (for nginx)
• nextcloud (nspawn container inside guayusa): /etc/ssl/private/csclub.uwaterloo.ca.chain (for nginx)
• citric-acid (runs vaultwarden): /etc/ssl/private/csclub.uwaterloo.ca.{chain,key} (for nginx)

Some services (e.g. Dovecot, Postfix) prefer to have the certificate chain in one file. Concatenate the appropriate intermediate root to the end of the certificate and store this as csclub-wildcard-chain.crt.

More certificate locations

We have some SSL certificates which are not used by web servers, but still need to be renewed eventually.

Prometheus node exporter

All of our Prometheus node exporters are using mTLS via stunnel (every bare-metal host, as well as caffeine, coffee and mail, is running this exporter). The certificates (both client and server) are set to expire in September 2031; before then, create new keypairs in /opt/prometheus/tls, and deploy the new server.crt, node.crt and node.key to /etc/stunnel/tls on all machines. Restart prometheus and all of the node exporters.

ADFS

See ADFS. When the university's IdP certificate expires (October 2025), we can just download a new one and restart Apache; when our own certificate expires (July 2031), we need to submit a new form to IST (please do this before the cert expires).

Keycloak

See Keycloak. When the saml-passthrough certificate expires (January 2032), you need to create a new keypair in /srv/saml-passthrough on caffeine, and upload the new certificate into the Keycloak UI (IdP settings). When the Keycloak SP certificate expires (December 2031), make sure to create a new keypair and upload it to the Keycloak UI (Realm Settings).

letsencrypt

We support letsencrypt for our virtual hosts with custom domains. We use the cerbot from debian repositories with a configuration file at /etc/letsencrypt/cli.ini, and a systemd timer to handle renewals.

The setup for a new domain is:

1. Become certbot on caffine with sudo -u certbot bash or similar.
2. Run certbot certonly -c /etc/letsencrypt/cli.ini -d DOMAIN --logs-dir /tmp. The logs-dir isn't important and is only needed for troubleshooting.
3. Set up the Apache site configuration using the example below. (apache config is in /etc/apache2) Note the permanent redirect to https.
4. Make sure to commit your changes when you're done.
5. Reloading apache config is sudo systemctl reload apache2.

<VirtualHost *:80>
    ServerName example.com
    ServerAlias *.example.com
    ServerAdmin example@csclub.uwaterloo.ca

    #DocumentRoot /users/example/www/
    Redirect permanent / https://example.com/

    ErrorLog /var/log/apache2/example-error.log
    CustomLog /var/log/apache2/example-access.log combined
</VirtualHost>

<VirtualHost csclub:443>
    SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
    SSLStrictSNIVHostCheck on

    ServerName example.com
    ServerAlias *.example.com
    ServerAdmin example@csclub.uwaterloo.ca

    DocumentRoot /users/example/www

    ErrorLog /var/log/apache2/example-error.log
    CustomLog /var/log/apache2/example-access.log combined
</VirtualHost>

acme.sh

We are using acme.sh for provisioning SSL certificates for some of our *.csclub.cloud domains. It is currently set up under /root/.acme.sh on biloba.

NOTE: acme.sh has a cron job which automatically renews certificates before they expire and reloads NGINX, so you do not have to do anything after issuing and installing a certificate (i.e. "set-and-forget").

How to add a new SSL cert for a custom domain on CSC cloud

Note: you do not need to acquire a new cert if the requested domain is directly on csclub.cloud, e.g. app1.csclub.cloud. We can re-use our wildcard cert on csclub.cloud for that. However, if a user requests a multi-level domain on csclub.cloud, or a domain hosted on an external registrar, then you will need to create a new cert.

Let's say user ctdalek wants mydomain.com to point to a VM on CSC cloud.
TLDR:

# Obtain the cert.
# If a subdomain was also requested, pass the -d option multiple times, e.g.
# `-d mydomain.com -d sub.mydomain.com`. Make sure the "main" domain is specified first.
acme.sh --issue -d mydomain.com -w /var/www

# Install the cert.
# If a subdomain was also requested, only specify the "main" domain.
acme.sh --install-cert -d mydomain.com \
    --key-file /etc/nginx/ceod/member-ssl/mydomain.com.key \
    --fullchain-file /etc/nginx/ceod/member-ssl/mydomain.com.chain \
    --reloadcmd "/root/bin/reload-nginx.sh"

# Create a vhost file.
# Look at the other files in the same directory for inspiration.
# Make sure the file starts with the username and an underscore, e.g. "ctdalek_",
# because this is how ceod keeps track of the vhosts.
# Make sure to set the custom domain name(s) and paths to the SSL key/cert.
vim /etc/nginx/ceod/member-vhosts/ctdalek_mydomain.com

# Finally, reload NGINX on both biloba and chamomile. The /etc/nginx/ceod directory
# is shared between them.
/root/bin/reload-nginx.sh

Installation

cd /opt    
git clone --depth 1 https://github.com/acmesh-official/acme.sh    
cd acme.sh    
./acme.sh --install -m syscom@csclub.uwaterloo.ca    
. "/root/.acme.sh/acme.sh.env"

Important: If invoking acme.sh from another program, it needs the environment variables set in acme.sh.env. Currently, that is just

LE_WORKING_DIR="/root/.acme.sh"

For testing purposes, make sure to use the Let's Encrypt test server:

acme.sh --set-default-ca --server letsencrypt_test

NGINX setup

mkdir -p /var/www/.well-known/acme-challenge

Add the following snippet to your default NGINX file (e.g. /etc/nginx/sites-enabled/default):

  # For Let's Encrypt
  location /.well-known/acme-challenge/ {
    alias /var/www/.well-known/acme-challenge/;
  }

Now assuming that biloba has the IP address for *.csclub.cloud, you can test that everything is working:

acme.sh --issue -d app.merenber.csclub.cloud -w /var/www

To install a certificate after it's been issued:

acme.sh --install-cert -d app.merenber.csclub.cloud \
    --key-file /etc/nginx/ceod/member-ssl/app.merenber.csclub.cloud.key \
    --fullchain-file /etc/nginx/ceod/member-ssl/app.merenber.csclub.cloud.chain \
    --reloadcmd "/root/bin/reload-nginx.sh"

At this point, you should add your NGINX vhost file which uses that SSL certificate.

To remove a certificate:

acme.sh --remove -d app.merenber.csclub.cloud
rm -r /root/.acme.sh/app.merenber.csclub.cloud
rm /etc/nginx/ceod/member-ssl/app.merenber.csclub.cloud.chain
rm /etc/nginx/ceod/member-ssl/app.merenber.csclub.cloud.key

Don't forget to remove the NGINX vhost file too.

Once you think you're ready, use a real ACME provider, e.g.

acme.sh --set-default-ca --server letsencrypt

Since we have a ZeroSSL account, and ZeroSSL has no rate limit, we are going to use that instead:

acme.sh  --register-account  --server zerossl \
        --eab-kid  xxxxxxxxxxxx  \
        --eab-hmac-key  xxxxxxxxx
acme.sh --set-default-ca  --server zerossl

DNS challenge

To obtain a wildcard certificate (e.g. *.k8s.csclub.cloud), you will need to perform the DNS-01 challenge. We are going to use nsupdate to interact with our BIND9 server on dns1.

On dns1, run:

tsig-keygen csc-cloud

Paste the output into the appropriate section in /etc/bind/named.conf.local. Also paste it into a file somewhere on biloba, e.g. /etc/csc/csc-cloud-tsig.key.

Add the following to the csclub.cloud zone block:

  allow-update {
    !{
      !127.0.0.1;
      !::1;
      !129.97.134.0/24;
      !2620:101:f000:4901::/64;
      any;
    };
    key csc-cloud;
  };

(We're basically trying to restrict updates to the given IP ranges. See https://serverfault.com/a/417229.)

The 'bind' user can't write to files under /etc/bind, so we're going to move our zone file to /var/lib/bind instead. Comment out 'file "/etc/bind/db.csclub.cloud";' from named.conf.local and add this line below it:

  file "/var/lib/bind/db.csclub.cloud";

Then run:

  cp /etc/bind/db.csclub.cloud /var/lib/bind/db.csclub.cloud
  chown bind:bind /var/lib/bind/db.csclub.cloud
  rndc reload

On biloba, check that everything's working:

  nsupdate -k /etc/csc/csc-cloud-tsig.key -v <<EOF
  update add test.csclub.cloud 300 A 0.0.0.0
  send
  EOF

Use a tool such as dig to make sure that the update was successful. If it worked, you can delete the record:

  nsupdate -k /etc/csc/csc-cloud-tsig.key -v <<EOF
  delete test.csclub.cloud
  send
  EOF

Now we are ready to actually perform the challenge with acme.sh:

  export NSUPDATE_SERVER="dns1.csclub.uwaterloo.ca"
  export NSUPDATE_KEY="/etc/csc/csc-cloud-tsig.key"
  acme.sh --issue --dns dns_nsupdate -d 'k8s.csclub.cloud' -d '*.k8s.csclub.cloud'

(If something goes wrong, use the --debug flag.)

If all went well, just install the certificate as usual:

  acme.sh --install-cert -d k8s.csclub.cloud \
    --key-file /etc/nginx/ceod/syscom-ssl/k8s.csclub.cloud.key \
    --fullchain-file /etc/nginx/ceod/syscom-ssl/k8s.csclub.cloud.chain \
    --reloadcmd 'systemctl reload nginx'