Mailing Lists

Systems Mailing Lists

The following mailing lists are the targets of various automated notifications. Subscribe to them to receive these notifications.

git@csclub.uwaterloo.ca

Commits to club git repositories are sent to this list in patch form.

packages@csclub.uwaterloo.ca

Changes to our debian repository are sent to this list.

ceo@csclub.uwaterloo.ca

CEO sends a note to this list every time a new member or club is added.

Technical Details

Most of our mailing lists are handled through Mailman, including the lists for the Executive (exec@csclub.uwaterloo.ca), the Program Committee (progcom@csclub.uwaterloo.ca), and the Systems Committee (syscom@csclub.uwaterloo.ca).

Mailman 3

Starting from April 2021, we now use Mailman 3 for managing our mailing lists. Mailman 3 is split into three independent components:

• Mailman 3 Core is responsible for sending and receiving emails.
• Postorius is the web admin UI for creating and managing mailing lists.
• HyperKitty is the archiver, where past messages can be viewed and searched.

Day-to-Day Operations

The Django admin site for Mailman 3 is accessible from here. Generally, you'll only need to use this to see the list of accounts, and to assign/remove permissions from them.

If you are logged in as a superuser, you should be able to see a list of all the mailing lists from here. After clicking on a list, you should be able to access and modify all of its settings, including subscription policy and message acceptance. Most of the setting names should be self-explanatory. For example, under 'Held Messages', you can see a list of all held messages, and take an action on each one (discard, reject, accept, etc.).

Installation

The steps below describe how Mailman 3 was installed on the mail container and how we migrated the lists from Mailman 2. Note that some lists were not migrated due to inactivity. See /var/lib/mailman/data/aliases in the mail container to see which lists were not migrated.

Database setup

Log into the coffee VM as root, run mysql, and create new databases for Mailman 3 and HyperKitty:

CREATE DATABASE mailman3;    
CREATE USER mailman3 IDENTIFIED BY 'replace_this_password';    
GRANT ALL PRIVILEGES ON mailman3.* TO mailman3;

Repeat the steps above for mailman3web instead of mailman3.

Warning: Make sure the MariaDB version is 10.2 or newer. The default package in Debian 10 and above should be fine.

Mail container setup

First, install some prerequisites which we'll need later:

apt update    
apt install python3-pip python3-mysqldb memcached python3-pylibmc python3-xapian    
pip3 install git+https://github.com/notanumber/xapian-haystack.git

As of this writing (2021-04-17), the package python3-xapian-haystack in the Debian repositories is broken. Make sure to install the latest version off of GitHub, as shown above.

Unfortunately memcached will fail due to Debian's default LXC configuration being unable to create new namespaces. The easiest workaround for this is to just disable mount namespaces in the systemd service. Run systemctl edit memcached.service, then paste the following:

[Service]    
PrivateTmp=false    
ProtectSystem=false    
PrivateDevices=false

Then restart memcached.

Installing the Mailman 3 suite

Get ready, because as soon as you install the full Mailman 3 suite, a lot of services are going to start failing, fast. The worst part is that the mail container is currently set up to email failed cron jobs to the syscom list, and Mailman 3 installs a cron job which runs once per minute, which will cause a lot of spam. Make sure to read these instructions carefully before executing them.

apt install mailman3-full

If the command above works for you on the first try, consider yourself lucky. It certainly didn't for me. If it fails, the first thing you want to do is comment out all the lines in /etc/cron.d/mailman3-web because otherwise the syscom list will get spammed. We are going to edit some files, then try to reinstall the whole suite again.

First, open /etc/mailman3/mailman.cfg. Under [general], set site_owner to your personal email address, temporarily. Under [database], set:

class: mailman.database.mysql.MySQLDatabase
url: mysql+mysqldb://mailman3:my_password@coffee.csclub.uwaterloo.ca/mailman3?charset=utf8&use_unicode=1

Note how we are using the mysqldb (mysqlclient) driver, not the default pymysql. Make sure to replace my_password with the password you created for the mailman3 database.

Under [mta], set <code<smtp_port to 10025 (there's an entry in /etc/postfix/master.cf for Mailman). Also make sure that the following lines are present at the end of the file:

[archiver.hyperkitty]
class: mailman_hyperkitty.Archiver
enable: yes
configuration: /etc/mailman3/mailman-hyperkitty.cfg

The next file we need to edit is /etc/mailman3/mailman-web.py. Go to the DATABASES variable and make sure 'ENGINE' is set to 'django.db.backends.mysql'. Fill in the username, password, host, and DB name for mailman3web. Uncomment the MySQL-specific options under 'OPTIONS'. Also set 'charset' to 'utf8mb4' in 'OPTIONS'. This is important - MariaDB, like MySQL, only uses 3 bytes per Unicode character by default, which means that if someone sends us an email with a 4-byte Unicode emoji in it, HyperKitty will explode. So make sure to use 'utf8mb4'. The final variable should look like this:

DATABASES = { 
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'mailman3web',
        'USER': 'mailman3web',
        'PASSWORD': 'my_password',
        'HOST': 'coffee.csclub.uwaterloo.ca',
        'PORT': '', 
        'OPTIONS': {
            'init_command': "SET sql_mode='STRICT_TRANS_TABLES'",
            'charset': 'utf8mb4',
        },
    }   
}

Some more variables you need to set/unset:

• Set EMAILNAME to 'csclub.uwaterloo.ca'.
• Set HOSTNAME to 'mailman.csclub.uwaterloo.ca'.
• Set POSTORIUS_TEMPLATE_BASE_URL to 'https://mailman.csclub.uwaterloo.ca/mailman3/'. If you want to strip out the /mailman3 part, you'll need to edit the Apache config as well - see the relevant section below.
• Set TIME_ZONE to 'America/Toronto'.
• Comment out 'django_mailman3.lib.auth.fedora' under INSTALLED_APPS.

We need to instruct Django to use memcached for caching. Add the following section to mailman-web.py:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
        'LOCATION': '127.0.0.1:11211',
    }
}

We also need to want to use Xapian as the backend for full-text search (the default engine, Whoosh, is written in pure Python and has horrible performance). Add the following section to mailman-web.py:

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'xapian_backend.XapianEngine',
        'PATH': '/var/lib/mailman3/web/xapian_index',
    }
}