Mailing Lists

Systems Mailing Lists

The following mailing lists are the targets of various automated notifications. Subscribe to them to receive these notifications.

git@csclub.uwaterloo.ca

Commits to club git repositories are sent to this list in patch form.

packages@csclub.uwaterloo.ca

Changes to our debian repository are sent to this list.

ceo@csclub.uwaterloo.ca

CEO sends a note to this list every time a new member or club is added.

Technical Details

Most of our mailing lists are handled through Mailman, including the lists for the Executive (exec@csclub.uwaterloo.ca), the Program Committee (progcom@csclub.uwaterloo.ca), and the Systems Committee (syscom@csclub.uwaterloo.ca).

Mailman 3

Starting from April 2021, we now use Mailman 3 for managing our mailing lists. Mailman 3 is split into three independent components:

• Mailman 3 Core is responsible for sending and receiving emails.
• Postorius is the web admin UI for creating and managing mailing lists.
• HyperKitty is the archiver, where past messages can be viewed and searched.

Day-to-Day Operations

The Django admin site for Mailman 3 is accessible from here. Generally, you'll only need to use this to see the list of accounts, and to assign/remove permissions from them.

If you are logged in as a superuser, you should be able to see a list of all the mailing lists from here. After clicking on a list, you should be able to access and modify all of its settings, including subscription policy and message acceptance. Most of the setting names should be self-explanatory. For example, under 'Held Messages', you can see a list of all held messages, and take an action on each one (discard, reject, accept, etc.).

Unfortunately it is not possible to subscribe a new member from the web UI, although you can unsubscribe them (open issue here). You can do this from the command line in the mail container. For example, to add people to the exec list:

sudo
su -s /bin/bash list
. /opt/mailman3/bin/activate
mailman addmembers - exec@csclub.uwaterloo.ca
person1@csclub.uwaterloo.ca
person2@csclub.uwaterloo.ca
...
^D

You need to press Ctrl-D (EOF) after entering the list of new members.

Creating a new list

This can be done from the Postorius UI. Unfortunately, ever since we moved Mailman 3 into its container, its copy of postfix_lmtp (and postfix_lmtp.db) is out of sync with the mail container's version (since they each have their own filesystem). So after creating a new list, copy those files over and reload Postfix (from inside the mail container):

# from xylitol
cp -a /vm/mailman/opt/mailman3/data/postfix_lmtp /vm/mail/rootfs/opt/mailman3/data/
cp -a /vm/mailman/opt/mailman3/data/postfix_lmtp.db /vm/mail/rootfs/opt/mailman3/data/
# from mail
postfix reload

I have never actually attempted this before, so I am not certain that it will work.

Make sure you disable the Pipermail archiver (go to Settings -> Archiving and uncheck the Pipermail box). That archiver only works for lists which were migrated from Mailman 2.

User gets subscribed to the same list twice

Sometimes a user gets subscribed to a list twice and it becomes impossible to unsubscribe them. This actually happened to us before, and as of this writing, it is still an open issue. If this happens again, just run the following:

su -s /bin/bash list
cd /opt/mailman3
. bin/activate
mailman shell -r delete_dup_sub Yes

Installation

Update: We now use the pip package of mailman (from PyPI) instead of the Debian package, since the Debian package was giving us too many issues. All of the configs are now in /etc/mailman3, and the data files are in /opt/mailman3. A virtualenv was created in /opt/mailman3.

Update 2: Mailman has been moved into its own systemd-nspawn container. It communicates with the mail container via a dedicated veth pair. Some relevant changes include: apply this patch; edit /opt/mailman3/lib/python3.9/site-packages/mailman/database/alembic/versions/bc0c49c6dda2_use_utf8mb4_in_mysql.py so that the type of autoresponse_postings_text is SAUnicodeLarge instead of SAUnicode4Byte; update main.cf and master.cf (and postfix_lmtp and postfix_lmtp.db in /opt/mailman3/data) to account for the new veth pair IP address of the mailman container.

The steps below describe how Mailman 3 was installed on the mail container and how we migrated the lists from Mailman 2. Note that some lists were not migrated due to inactivity. See /var/lib/mailman/data/aliases in the mail container to see which lists were not migrated.

Database setup

Log into the coffee VM as root, run mysql, and create new databases for Mailman 3 and HyperKitty:

CREATE DATABASE mailman3;    
CREATE USER mailman3 IDENTIFIED BY 'replace_this_password';    
GRANT ALL PRIVILEGES ON mailman3.* TO mailman3;

Repeat the steps above for mailman3web instead of mailman3.

Warning: Make sure the MariaDB version is 10.2 or newer. The default package in Debian 10 and above should be fine.

Mail container setup

First, install some prerequisites which we'll need later:

apt update    
apt install python3-pip python3-mysqldb memcached python3-pylibmc python3-xapian    
pip3 install git+https://github.com/notanumber/xapian-haystack.git

As of this writing (2021-04-17), the package python3-xapian-haystack in the Debian repositories is broken. Make sure to install the latest version off of GitHub, as shown above.

Unfortunately memcached will fail due to Debian's default LXC configuration being unable to create new namespaces. The easiest workaround for this is to just disable mount namespaces in the systemd service. Run systemctl edit memcached.service, then paste the following:

[Service]    
PrivateTmp=false    
ProtectSystem=false    
PrivateDevices=false

Then restart memcached.

Installing the Mailman 3 suite

Get ready, because as soon as you install the full Mailman 3 suite, a lot of services are going to start failing, fast. The worst part is that the mail container is currently set up to email failed cron jobs to the syscom list, and Mailman 3 installs a cron job which runs once per minute, which will cause a lot of spam. Make sure to read these instructions carefully before executing them.

apt install mailman3-full

If the command above works for you on the first try, consider yourself lucky. It certainly didn't for me. If it fails, the first thing you want to do is comment out all the lines in /etc/cron.d/mailman3-web because otherwise the syscom list will get spammed. We are going to edit some files, then try to reinstall the whole suite again.

First, open /etc/mailman3/mailman.cfg. Under [general], set site_owner to your personal email address, temporarily. Under [database], set:

class: mailman.database.mysql.MySQLDatabase
url: mysql+mysqldb://mailman3:my_password@coffee.csclub.uwaterloo.ca/mailman3?charset=utf8&use_unicode=1

Note how we are using the mysqldb (mysqlclient) driver, not the default pymysql. Make sure to replace my_password with the password you created for the mailman3 database.

Under [mta], set smtp_port to 10025 (there's an entry in /etc/postfix/master.cf for Mailman). Also make sure that the following lines are present at the end of the file:

[archiver.hyperkitty]
class: mailman_hyperkitty.Archiver
enable: yes
configuration: /etc/mailman3/mailman-hyperkitty.cfg

The next file we need to edit is /etc/mailman3/mailman-web.py. Go to the DATABASES variable and make sure 'ENGINE' is set to 'django.db.backends.mysql'. Fill in the username, password, host, and DB name for mailman3web. Uncomment the MySQL-specific options under 'OPTIONS'. Also set 'charset' to 'utf8mb4' in 'OPTIONS'. This is important - MariaDB, like MySQL, only uses 3 bytes per Unicode character by default, which means that if someone sends us an email with a 4-byte Unicode emoji in it, HyperKitty will explode. So make sure to use 'utf8mb4'. The final variable should look like this:

DATABASES = { 
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'mailman3web',
        'USER': 'mailman3web',
        'PASSWORD': 'my_password',
        'HOST': 'coffee.csclub.uwaterloo.ca',
        'PORT': '', 
        'OPTIONS': {
            'init_command': "SET sql_mode='STRICT_TRANS_TABLES'",
            'charset': 'utf8mb4',
        },
    }   
}

Some more variables you need to set/unset:

• Set EMAILNAME to 'csclub.uwaterloo.ca'.
• Set HOSTNAME to 'mailman.csclub.uwaterloo.ca'.
• Set POSTORIUS_TEMPLATE_BASE_URL to 'https://mailman.csclub.uwaterloo.ca/mailman3/'. If you want to strip out the /mailman3 part, you'll need to edit the Apache config as well - see the relevant section below.
• Set TIME_ZONE to 'America/Toronto'.
• Comment out 'django_mailman3.lib.auth.fedora' under INSTALLED_APPS.

We need to instruct Django to use memcached for caching. Add the following section to mailman-web.py:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
        'LOCATION': '127.0.0.1:11211',
    }
}

We also need to want to use Xapian as the backend for full-text search (the default engine, Whoosh, is written in pure Python and has horrible performance). Add the following section to mailman-web.py:

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'xapian_backend.XapianEngine',
        'PATH': '/var/lib/mailman3/web/xapian_index',
    }
}

We also don't want to run one qcluster process on each CPU core, so add the following section as well:

Q_CLUSTER = {
    'timeout': 300,
    'save_limit': 100,
    'orm': 'default',
    'poll': 5,
    'workers': 4,
}

We also want to disable log emails because this will quickly spam our inboxes. (This is also bad for privacy because the HTTP Authorization header is kept in the log, which contains people's base64-encoded passwords). Copy the LOGGING variable from /usr/share/mailman3-web/settings.py into mailman-web.py and remove all mentions of 'mail_admins'.

The next file we need to edit is /etc/mailman3/mailman-hyperkitty.cfg. Set api_key to the same value as MAILMAN_ARCHIVER_KEY in mailman-web.py (without the quotation marks). Also set base_url to https://mailman.csclub.uwaterloo.ca/mailman3/hyperkitty/.

The next file we need to edit is /etc/mailman3/uwsgi.ini. The default buffer size limit for uWSGI is too low and can cause requests to fail. Add the following line to uwsgi.ini:

buffer-size = 32768

Patching the source files

Unfortunately some of the packages which we have installed have bugs in them which we need to fix.

Xapian word size limit

The first one is in the Haystack-Xapian backend: occasionally we have very large words which exceed Xapian's limit (see this GitLab issue). Open /usr/local/lib/python3.7/dist-packages/xapian_backend.py, add import traceback at the top of the file, and make sure the bottom of the update() function (around line 500) looks like this:

                # finally, replace or add the document to the database
                database.replace_document(document_id, document)

        except UnicodeDecodeError:
            sys.stderr.write('Chunk failed.\n')
            pass

        except xapian.InvalidArgumentError:
            traceback.print_exc()

        finally:
            try:
                database.close()
            except:
                traceback.print_exc()

(We're just ignoring any errors caused by long words so that the entire indexing job doesn't fail.)

Xapian six import

Same file: open xapian_backend.py and at the top, replace from django.utils import six by import six.

Mailman3 restclient URL quoting

Next, we need to edit Mailman 3's REST client, because there is a bug where if someone's email address contains a space, the REST client doesn't URL-encode it properly (one of us should probably file an issue on GitLab for this at some point). Open /usr/lib/python3/dist-packages/mailmanclient/client.py, and at the top of the file, add the following line:

from urllib.parse import quote

Then, in the `get_user` function, replace

response, content = self._connection.call(
    'users/{0}'.format(address))

with

response, content = self._connection.call(
    'users/{0}'.format(quote(str(address))))

Hyperkitty URL truncation

Hyperkitty truncates each URL in each message to 76 characters. This is way too little. Most Google Docs links will go way past that limit. As of this writing, that limit is unfortunately hard-coded in the Django templates, so we're going to have to edit them.

There's actually another problem with the Django templates, which is the fact that URLs enclosed in angle brackets (<>) don't get quoted properly. So we're going to fix that too. (This might have been fixed in a more recent version of Hyperkitty).

Go to /usr/lib/python3/dist-packages/hyperkitty/templates/hyperkitty and find all files which contain the line urlizetrunc:76, and replace them with urlize. Also make sure that 'urlize' occurs right after 'email.content' in the filter pipeline. You can find these files using:

cd /usr/lib/python3/dist-packages/hyperkitty/templates/hyperkitty
grep -IRl 'urlizetrunc:76' .

For example, in ajax/temp_message.html, the template should look like:

{{ email.content|urlize|snip_quoted|wordwrap:90|escapeemaillinks }}

Installing Mailman 3, again

Now we are ready to reinstall mailman3-full:

apt install mailman3-full

Restart the mailman3 and mailman3-web services and make sure they are running:

systemctl restart mailman3
systemctl restart mailman3-web

Cron configuration

Open /etc/default/cron and set the following line:

EXTRA_OPTS="-L 4"

This will prevent cron from spamming the logs, which will inevitably happen since mailman3-web runs very frequent jobs. Restart cron:

systemctl restart cron

Also uncomment the lines in /etc/cron.d/mailman3-web which you commented out earlier.

Optional, but recommended: change the minutely cron job in /etc/cron.d/mailman3-web to run every 15 minutes instead (or even once an hour). Once a minute just sounds like too much. Example:

*/15       * * * * www-data     [ -f /usr/bin/django-admin ] && flock -n /var/run/mailman3-web/cron.minutely /usr/share/mailman3-web/manage.py runjobs minutely

Installing extra archivers

I wrote some extra archivers for Mailman which are available here. One of them is called 'monthly_archiver' and it just dumps each message into a maildir directory for the current year and month. The motivation for this was to preserve message headers, since HyperKitty does not store headers. The other archiver is called 'pipermail', and it does two things: it passes each message to Pipermail, and it also appends it to the Mailman 2 mbox file. The idea behind this one was to have a backup in case we need to re-install Mailman 3 from scratch. Hopefully we never have to do that.

To install the archivers (as root):

git clone my_username@csclub.uwaterloo.ca:/srv/git/merenber/extra-mailman-archivers.git
cd extra-mailman-archivers
pip3 install .

Open mailman.cfg and add the following lines:

[archiver.monthly_archiver]
class: extra_mailman_archivers.monthly_archiver.MonthlyArchiver
enable: yes 

[archiver.pipermail]
class: extra_mailman_archivers.pipermail.Pipermail
enable: yes 
configuration: /etc/mailman3/mailman-pipermail.cfg

Also create the file /etc/mailman3/mailman-pipermail.cfg and paste the following inside it:

[general]
base_url:

Then restart mailman3.

You should now be able to view 'monthly_archiver' and 'pipermail' as archiver options for each list in Postorius. As a general rule of thumb, you only need monthly_archiver for the lists which are not moderated, so that we can use the headers to delete spam later. The pipermail archiver only really needs to be used for the exec and syscom lists, since those are the most important ones.

Postfix changes

Open /etc/postfix/main.cf and add the following lines:

owner_request_special = no
transport_maps = hash:/var/lib/mailman3/data/postfix_lmtp
local_recipient_maps =
        proxy:unix:passwd.byname,
        $alias_maps,
        hash:/var/lib/mailman3/data/postfix_lmtp

Then run:

postfix reload

Apache httpd changes

Copy the contents of /etc/mailman3/apache.conf into /etc/apache2/sites-real/mailman (or wherever the config file is for the Mailman page). You may have to customize some lines. Then run:

a2enmod proxy_uwsgi
systemctl restart apache2

Postorius setup

Now cd into /usr/share/mailman3-web, and run ./manage.py createsuperuser. Use 'sysadmin' as the username and use the syscom mailing list as the email address.

Now go to https://mailman.csclub.uwaterloo.ca/mailman3/postorius/lists/, and sign in as the user you just created. You will get an email confirmation.

You have probably noticed that the site name is still set to 'example.com'. Go to https://mailman.csclub.uwaterloo.ca/mailman3/admin/, then click on 'Sites'. Click on 'example.com', then set 'Domain name' to mailman.csclub.uwaterloo.ca. Set the display name to something reasonable.

Now go to https://mailman.csclub.uwaterloo.ca/mailman3/postorius/domains, click on 'New domain', and create a new domain.

Also click on 'Templates' and add two new templates:

• 'The footer for a regular (non-digest) message.'
• 'The footer for a digest message.'

The contents of these should be:

_______________________________________________
$display_name mailing list -- $listname
To unsubscribe send an email to ${short_listname}-leave@${domain}
https://mailman.csclub.uwaterloo.ca/mailman3/postorius/lists/${list_id}

The reason why we need to specify this is because many of our old lists had footer templates which used %()s placeholders, which have no meaning in Mailman 3. These are the default templates, and these are the placeholders you can use. Also note that you can have domain-wide templates and per-list templates.

Importing lists from Mailman 2

Make sure to read this first.

Before we continue - unfortunately the current version of HyperKitty in the Debian repos doesn't contain a fix which allows bad 'Subject' headers to be skipped (see this issue). So we're going to need to download a more recent version of the file we need:

cd /usr/lib/python3/dist-packages/hyperkitty/management/commands
mv hyperkitty_import.py hyperkitty_import.py.bak
curl -sLOJ 'https://gitlab.com/mailman/hyperkitty/-/raw/master/hyperkitty/management/commands/hyperkitty_import.py?inline=false'

We're also going to need to download a newer version of a specific Mailman 3 file which occasionally causes the import21 command to fail:

cd /usr/lib/python3/dist-packages/mailman/commands
mv cli_import.py cli_import.py.bak
curl -sLOJ 'https://gitlab.com/mailman/mailman/-/raw/master/src/mailman/commands/cli_import.py?inline=false'

We're also going to need to modify a few columns in the database because the csc-industry mailing list has a long description which exceeds the default alotted size. Log back into coffee as root, run mysql, and execute the following:

USE mailman3;
ALTER TABLE mailinglist MODIFY info VARCHAR(1024);
ALTER TABLE mailinglist MODIFY autoresponse_postings_text VARCHAR(1024);

Before migrating a list from Mailman 2, you will need to create a list with exact the same name in Mailman 3. Go to https://mailman.csclub.uwaterloo.ca/mailman3/postorius/lists, click 'New list', and follow the prompts. Go to the list settings and change what you need.

Note: For all of the commands below, it is crucial that you run the mailman command as the list user and the manage.py command as the www-data user. Otherwise, you can create files which only root can access, causing horrible breakage.

In this example, we're going to import the mailing list called 'test'. The process should be the same for the other lists.

su -s /bin/bash list
mailman import21 test@csclub.uwaterloo.ca /var/lib/mailman/lists/test/config.pck
exit
su -s /bin/bash www-data
cd /usr/share/mailman3-web
./manage.py hyperkitty_import -l test@csclub.uwaterloo.ca -v3 /var/lib/mailman/archives/private/test.mbox/test.mbox
./manage.py update_index_one_list -v3 test@csclub.uwaterloo.ca

Now open /var/lib/mailman/data/aliases and comment out all of the addresses for the list which we just migrated. Then run, as root,

postalias /var/lib/mailman/data/aliases
postfix reload

Once you are done migrating all of the lists, make sure sysadmin (or exec/syscom) is the site owner of Mailman, not your personal email. Open /etc/mailman3/mailman.cfg, and under [mailman], make sure you have the following line:

site_owner: sysadmin@csclub.uwaterloo.ca

Then restart mailman3.

Custom URL

If you wish to strip out the /mailman3 from all of the Mailman 3 URLs, there are several files which you'll need to edit.

First, open /etc/mailman3/mailman-hyperkitty.cfg and take out /mailman3 from base_url.

Next, open etc/mailman3/mailman-web.py and add:

STATIC_URL = '/static/'

Also take out /mailman3 from the POSTORIUS_TEMPLATE_BASE_URL variable.

Now open your Apache config and do the following:

• Remove /mailman3 from Alias /mailman3/favicon.ico and Alias /mailman3/static.
• Add the following lines to redirect people still using the old URLs:

  RedirectMatch permanent "^/mailman3$" "/"
  RedirectMatch permanent "^/mailman3/(.*)$" "/$1"
  

• Add the following to your ProxyPass section:

  ProxyPass /mailman3 !
  

  Also remove /mailman3 from the other ProxyPass entries.

Now run:

systemctl restart mailman3
systemctl restart mailman3-web
systemctl reload apache2

Now go to https://mailman.csclub.uwaterloo.ca/postorius/domains and edit any templates which have hardcoded /mailman3 in a URL. You will need to do this for any list-specific templates as well.

Shutting down Mailman 2 for good

Open /etc/postfix/main.cf and remove hash:/var/lib/mailman/data/aliases from alias_maps.

Run postfix reload.

Open the Apache config and comment out all of the lines relevant to Mailman 2, except for those necessary to show Pipermail. Reload Apache.

Open /etc/cron.d/mailman and comment out everything.

Now run:

systemctl stop mailman
systemctl mask mailman

HyperKitty scripts

There are a couple of scripts in /usr/share/mailman3-web for deleting messages from HyperKitty. The first deletes all messages from HyperKitty (this should only be used if you need to re-migrate a mailing list). Usage:

./hyperkitty_delete_all.py name_of_list

The other script deletes all messages which have been flagged by SpamAssassin as being spam:

./hyperkitty_delete_spam.py --month 2021-April name_of_list

This requires the monthly_archiver to be enabled. Alternatively, if you want to delete spam using the Mailman 2 mbox file:

./hyperkitty_delete_spam.py --mbox name_of_list

Note that using the mbox file will take a very long time.

Mailman 2 to 3 setting mappings

Here is a very rough mapping of Mailman 2 to Mailman 3 (Postorius) settings. Note that some of them are not perfectly equivalent. There are also some settings in Mailman 2 which are not available in Mailman 3 and vice versa.

Mailman 2 setting	Mailman 3 setting
General Options -> owner	Users -> Owners
Non-digest options -> msg_footer	Templates -> 'The footer for a regular (non-digest) message.'
Digest options -> digest_footer	Templates -> 'The footer for a digest message.'
Privacy options -> Subscription rules -> advertised	Settings -> List Identity -> Show list on index page
Privacy options -> Subscription rules -> subscribe_policy	Settings -> Subscription Policy -> Subscription Policy
Privacy options -> Subscription rules -> ban_list	Banned addresses (Note that this only bans subscriptions, not posts.)
Privacy options -> Sender filters -> default_member_moderation, generic_nonmember_action	Settings -> Message Acceptance -> Default action to take when a member posts to the list
Privacy options -> Sender filters -> discard_these_nonmembers	Users -> Non-members -> add a non-member, then go to Non-member Options -> Administration options -> Moderation and choose 'Discard'
Privacy options -> Recipient filters -> require_explicit_destination	Settings -> Message Acceptance -> Require Explicit Destination
Archiving options -> archive, archive_private	Settings -> Archiving -> Archive policy

Note that except for the templates, most settings should be imported correctly via the mailman import21 command.