Systemd

This page contains some tips and tricks for writing systemd units on CSC machines.

Services

• If your service should not be restarted by systemd automatically (e.g. because it has its own retry mechanism), set Restart=no
• If your service should be restarted by systemd automatically, make sure you set RestartSec to a reasonable value so that it does not restart too quickly
• If your service does not need to keep any persistent state on disk, consider using DynamicUser=yes with LoadCredential
• If you are running your service as root just so you can read a secret from a file, consider using DynamicUser=yes with
• Consider using ProtectSystem, ProtectHome, etc. See https://manpages.debian.org/stable/systemd/systemd.exec.5.en.html for details.
• If your service needs to accept network connections (i.e. is a server), use After=network.target
• If your service needs to create network connections (i.e. is a client), use After=network-online.target
• If your service needs to lookup LDAP users, use After=nslcd.service sssd.service
• if your service needs to access a folder on a networked filesystem, use RequiresMountsFor

Timers

Unlike cron, systemd timers do not send email alerts if the job fails. However, you can create your own alerts using OnFailure=. Paste the following into /usr/local/bin/csc-systemd-email and make it executable:

#!/bin/bash

# Adapted from https://wiki.archlinux.org/title/systemd/Timers#MAILTO

set -e

if [[ $# -ne 2 ]]; then
  echo "Usage: $0 <address> <unit>" >&2
  exit 1
fi

FROM="Systemd <root@$HOSTNAME>"
TO="$1"
if ! [[ $TO =~ @ ]]; then
  TO="$TO@csclub.uwaterloo.ca"
fi
SUBJECT="Systemd <root@$HOSTNAME> Unit '$2' failed"
MESSAGE="$(systemctl status --full "$2" || true)"

# Don't use the Postfix sendmail. It creates a new spool file and also
# forks to the background, which we don't want.
if [[ -x /usr/sbin/ssmtp ]]; then
  /usr/sbin/ssmtp -t <<EOF
To: $TO
From: $FROM
Subject: $SUBJECT
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8

$MESSAGE
EOF
elif [[ -x /usr/bin/mutt ]]; then
  EMAIL="$FROM" /usr/bin/mutt -F /dev/null -e "set copy=no" -s "$SUBJECT" -- "$TO" <<< "$MESSAGE"
else
  echo "Could not find program to email" >&2
  exit 1
fi

Next, paste the following into /etc/systemd/system/csc-email-on-failure@.service:

[Unit]
Description=Send email alert when %i fails

[Service]
Type=oneshot
ExecStart=/usr/local/bin/csc-systemd-email root@csclub.uwaterloo.ca %i
# Do not use DynamicUser=true until this issue gets fixed:
# https://github.com/systemd/systemd/issues/22737
User=nobody
# Need to be in the adm group to read journald logs
Group=adm

Then run systemctl daemon-reload. Now, all you need to do is add the following line to the [Unit] of any service for which you would like to receive email alerts:

OnFailure=csc-email-on-failure@%n.service

IMPORTANT: make sure you have the following setting in /etc/ssmtp/ssmtp.conf:

FromLineOverride=NO

Otherwise, Mailman 3 will reject the message because the Envelope From does not have a FQDN.