Mailing Lists

Systems Mailing Lists

The following mailing lists are the targets of various automated notifications. Subscribe to them to receive these notifications.

git@csclub.uwaterloo.ca

Commits to club git repositories are sent to this list in patch form.

packages@csclub.uwaterloo.ca

Changes to our debian repository are sent to this list.

ceo@csclub.uwaterloo.ca

CEO sends a note to this list every time a new member or club is added.

Technical Details

Most of our mailing lists are handled through Mailman, including the lists for the Executive (exec@csclub.uwaterloo.ca), the Program Committee (progcom@csclub.uwaterloo.ca), and the Systems Committee (syscom@csclub.uwaterloo.ca).

Mailman 3

Starting from April 2021, we now use Mailman 3 for managing our mailing lists. Mailman 3 is split into three independent components:

• Mailman 3 Core is responsible for sending and receiving emails.
• Postorius is the web admin UI for creating and managing mailing lists.
• HyperKitty is the archiver, where past messages can be viewed and searched.

Day-to-Day Operations

The Django admin site for Mailman 3 is accessible from here. Generally, you'll only need to use this to see the list of accounts, and to assign/remove permissions from them.

If you are logged in as a superuser, you should be able to see a list of all the mailing lists from here. After clicking on a list, you should be able to access and modify all of its settings, including subscription policy and message acceptance. Most of the setting names should be self-explanatory. For example, under 'Held Messages', you can see a list of all held messages, and take an action on each one (discard, reject, accept, etc.).

Installation

The steps below describe how Mailman 3 was installed on the mail container and how we migrated the lists from Mailman 2. Note that some lists were not migrated due to inactivity. See /var/lib/mailman/data/aliases in the mail container to see which lists were not migrated.

Database setup

Log into the coffee VM as root, run mysql, and create new databases for Mailman 3 and HyperKitty:

CREATE DATABASE mailman3;    
CREATE USER mailman3 IDENTIFIED BY 'replace_this_password';    
GRANT ALL PRIVILEGES ON mailman3.* TO mailman3;

Repeat the steps above for mailman3web instead of mailman3.

Warning: Make sure the MariaDB version is 10.2 or newer. The default package in Debian 10 and above should be fine.

Mail container setup

First, install some prerequisites which we'll need later:

apt update    
apt install python3-pip python3-mysqldb memcached python3-pylibmc python3-xapian    
pip3 install git+https://github.com/notanumber/xapian-haystack.git

As of this writing (2021-04-17), the package python3-xapian-haystack in the Debian repositories is broken. Make sure to install the latest version off of GitHub, as shown above.

Unfortunately memcached will fail due to Debian's default LXC configuration being unable to create new namespaces. The easiest workaround for this is to just disable mount namespaces in the systemd service. Run systemctl edit memcached.service, then paste the following:

[Service]    
PrivateTmp=false    
ProtectSystem=false    
PrivateDevices=false

Then restart memcached.

Installing the Mailman 3 suite

Get ready, because as soon as you install the full Mailman 3 suite, a lot of services are going to start failing, fast. The worst part is that the mail container is currently set up to email failed cron jobs to the syscom list, and Mailman 3 installs a cron job which runs once per minute, which will cause a lot of spam. Make sure to read these instructions carefully before executing them.

apt install mailman3-full

If the command above works for you on the first try, consider yourself lucky. It certainly didn't for me. If it fails, the first thing you want to do is comment out all the lines in /etc/cron.d/mailman3-web because otherwise the syscom list will get spammed. We are going to edit some files, then try to reinstall the whole suite again.

First, open /etc/mailman3/mailman.cfg. Under [general], set site_owner to your personal email address, temporarily. Under [database], set:

class: mailman.database.mysql.MySQLDatabase
url: mysql+mysqldb://mailman3:my_password@coffee.csclub.uwaterloo.ca/mailman3?charset=utf8&use_unicode=1

Note how we are using the mysqldb (mysqlclient) driver, not the default pymysql. Make sure to replace my_password with the password you created for the mailman3 database.

Under [mta], set smtp_port to 10025 (there's an entry in /etc/postfix/master.cf for Mailman). Also make sure that the following lines are present at the end of the file:

[archiver.hyperkitty]
class: mailman_hyperkitty.Archiver
enable: yes
configuration: /etc/mailman3/mailman-hyperkitty.cfg

The next file we need to edit is /etc/mailman3/mailman-web.py. Go to the DATABASES variable and make sure 'ENGINE' is set to 'django.db.backends.mysql'. Fill in the username, password, host, and DB name for mailman3web. Uncomment the MySQL-specific options under 'OPTIONS'. Also set 'charset' to 'utf8mb4' in 'OPTIONS'. This is important - MariaDB, like MySQL, only uses 3 bytes per Unicode character by default, which means that if someone sends us an email with a 4-byte Unicode emoji in it, HyperKitty will explode. So make sure to use 'utf8mb4'. The final variable should look like this:

DATABASES = { 
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'mailman3web',
        'USER': 'mailman3web',
        'PASSWORD': 'my_password',
        'HOST': 'coffee.csclub.uwaterloo.ca',
        'PORT': '', 
        'OPTIONS': {
            'init_command': "SET sql_mode='STRICT_TRANS_TABLES'",
            'charset': 'utf8mb4',
        },
    }   
}

Some more variables you need to set/unset:

• Set EMAILNAME to 'csclub.uwaterloo.ca'.
• Set HOSTNAME to 'mailman.csclub.uwaterloo.ca'.
• Set POSTORIUS_TEMPLATE_BASE_URL to 'https://mailman.csclub.uwaterloo.ca/mailman3/'. If you want to strip out the /mailman3 part, you'll need to edit the Apache config as well - see the relevant section below.
• Set TIME_ZONE to 'America/Toronto'.
• Comment out 'django_mailman3.lib.auth.fedora' under INSTALLED_APPS.

We need to instruct Django to use memcached for caching. Add the following section to mailman-web.py:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
        'LOCATION': '127.0.0.1:11211',
    }
}

We also need to want to use Xapian as the backend for full-text search (the default engine, Whoosh, is written in pure Python and has horrible performance). Add the following section to mailman-web.py:

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'xapian_backend.XapianEngine',
        'PATH': '/var/lib/mailman3/web/xapian_index',
    }
}

We also don't want to run one qcluster process on each CPU core, so add the following section as well:

Q_CLUSTER = {
    'timeout': 300,
    'save_limit': 100,
    'orm': 'default',
    'poll': 5,
    'workers': 4,
}

We also want to disable log emails because this will quickly spam our inboxes. (This is also bad for privacy because the HTTP Authorization header is kept in the log, which contains people's base64-encoded passwords). Copy the LOGGING variable from /usr/share/mailman3-web/settings.py into mailman-web.py and remove all mentions of 'mail_admins'.

The next file we need to edit is /etc/mailman3/mailman-hyperkitty.cfg. Set api_key to the same value as MAILMAN_ARCHIVER_KEY in mailman-web.py (without the quotation marks). Also set base_url to https://mailman.csclub.uwaterloo.ca/mailman3/hyperkitty/.

The next file we need to edit is /etc/mailman3/uwsgi.ini. The default buffer size limit for uWSGI is too low and can cause requests to fail. Add the following line to uwsgi.ini:

buffer-size = 32768

Patching the source files

Unfortunately some of the packages which we have installed have bugs in them which we need to fix. The first one is in the Haystack-Xapian backend: occasionally we have very large words which exceed Xapian's limit (see this GitLab issue). Open /usr/local/lib/python3.7/dist-packages/xapian_backend.py, add import traceback at the top of the file, and make sure the bottom of the update() function (around line 500) looks like this:

                # finally, replace or add the document to the database
                database.replace_document(document_id, document)

        except UnicodeDecodeError:
            sys.stderr.write('Chunk failed.\n')
            pass

        except xapian.InvalidArgumentError:
            traceback.print_exc()

        finally:
            try:
                database.close()
            except:
                traceback.print_exc()

(We're just ignoring any errors caused by long words so that the entire indexing job doesn't fail.)

Next, we need to edit Mailman 3's REST client, because there is a bug where if someone's email address contains a space, the REST client doesn't URL-encode it properly (one of us should probably file an issue on GitLab for this at some point). Open /usr/lib/python3/dist-packages/mailmanclient/restbase/connection.py, and on line 94, right before url = urljoin(self.baseurl, path), insert the following line:

path = path.replace(' ', '%20')

Now we are ready to reinstall mailman3-full. Restart the mailman3 and mailman3-web services and make sure they are running:

systemctl restart mailman3
systemctl restart mailman3-web

Now open /etc/default/cron and set the following line:

EXTRA_OPTS="-L 4"

This will prevent cron from spamming the logs, which will inevitably happen since mailman3-web runs very frequent jobs. Restart cron:

systemctl restart cron

Now uncomment the lines in /etc/cron.d/mailman3-web which you commented out earlier.

Postfix changes

Open /etc/postfix/main.cf and add the following lines:

owner_request_special = no
transport_maps = hash:/var/lib/mailman3/data/postfix_lmtp
local_recipient_maps =
        proxy:unix:passwd.byname,
        $alias_maps,
        hash:/var/lib/mailman3/data/postfix_lmtp

Then run:

postfix reload

Apache httpd changes

Copy the contents of /etc/mailman3/apache.conf into /etc/apache2/sites-real/mailman (or wherever the config file is for the Mailman page). You may have to customize some lines. Then run:

a2enmod proxy_uwsgi
systemctl restart apache2

Postorius setup

Now cd into /usr/share/mailman3-web, and run ./manage.py createsuperuser. Use 'sysadmin' as the username and use the syscom mailing list as the email address.

Now go to https://mailman.csclub.uwaterloo.ca/mailman3/postorius/lists/, and sign in as the user you just created. You will get an email confirmation.

You have probably noticed that the site name is still set to 'example.com'. Go to https://mailman.csclub.uwaterloo.ca/mailman3/admin/, then click on 'Sites'. Click on 'example.com', then set 'Domain name' to mailman.csclub.uwaterloo.ca. Set the display name to something reasonable.

Now go to https://mailman.csclub.uwaterloo.ca/mailman3/postorius/domains, click on 'New domain', and create a new domain.

Also click on 'Templates' and add two new templates:

• 'The footer for a regular (non-digest) message.'
• 'The footer for a digest message.'

The contents of these should be:

_______________________________________________
$display_name mailing list -- $listname
To unsubscribe send an email to ${short_listname}-leave@${domain}
https://mailman.csclub.uwaterloo.ca/mailman3/postorius/lists/${list_id}

The reason why we need to specify this is because many of our old lists had footer templates which used %()s placeholders, which have no meaning in Mailman 3. These are the default templates, and these are the placeholders you can use. Also note that you can have domain-wide templates and per-list templates.

Importing lists from Mailman 2

Make sure to read this first.

Before we continue - unfortunately the current version of HyperKitty in the Debian repos doesn't contain a fix which allows bad 'Subject' headers to be skipped (see this issue). So we're going to need to download a more recent version of the file we need:

cd /usr/lib/python3/dist-packages/hyperkitty/management/commands
mv hyperkitty_import.py hyperkitty_import.py.bak
curl -sLOJ 'https://gitlab.com/mailman/hyperkitty/-/raw/master/hyperkitty/management/commands/hyperkitty_import.py?inline=false'

We're also going to need to download a newer version of a specific Mailman 3 file which occasionally causes the import21 command to fail:

cd /usr/lib/python3/dist-packages/mailman/commands
mv cli_import.py cli_import.py.bak
curl -sLOJ 'https://gitlab.com/mailman/mailman/-/raw/master/src/mailman/commands/cli_import.py?inline=false'

We're also going to need to modify a few columns in the database because the csc-industry mailing list has a long description which exceeds the default alotted size. Log back into coffee as root, run mysql, and execute the following:

USE mailman3;
ALTER TABLE mailinglist MODIFY info VARCHAR(1024);
ALTER TABLE mailinglist MODIFY autoresponse_postings_text VARCHAR(1024);

Before migrating a list from Mailman 2, you will need to create a list with exact the same name in Mailman 3. Go to https://mailman.csclub.uwaterloo.ca/mailman3/postorius/lists, click 'New list', and follow the prompts. Go to the list settings and change what you need.

Note: For all of the commands below, it is crucial that you run the mailman command as the list user and the manage.py command as the www-data user. Otherwise, you can create files which only root can access, causing horrible breakage.

In this example, we're going to import the mailing list called 'test'. The process should be the same for the other lists.

su -s /bin/bash list
mailman import21 test@csclub.uwaterloo.ca /var/lib/mailman/lists/test/config.pck
exit
su -s /bin/bash www-data
cd /usr/share/mailman3-web
./manage.py hyperkitty_import -l test@csclub.uwaterloo.ca -v3 /var/lib/mailman/archives/private/test.mbox/test.mbox
./manage.py update_index_one_list -v3 test@csclub.uwaterloo.ca