We use [http://www.openldap.org/ OpenLDAP] for directory services. Our primary LDAP server is [[Machine_List#auth1|auth1]] and our secondary LDAP server is [[Machine_List#auth2|auth2]].

=== ehashman's Guide to Setting up OpenLDAP on Debian ===

Welcome to my nightmare.

==== What is LDAP? ====

<blockquote>'''LDAP:''' Lightweight Directory Access Protocol

An open, vendor-neutral, industry standard application protocol for accessing and maintaining distributed directory information services over an Internet Protocol (IP) network. — [https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol Wikipedia: LDAP]
</blockquote>
In this case, "directory" refers to the user directory, like on an old-school Rolodex. Many groups use LDAP to maintain their user directory, including the University (the "WatIAM" identity management system), the Computer Science Club, and even the UW Amateur Radio Club.

This is a guide documenting how to set up LDAP on a Debian Linux system.

==== First steps ====

<ul>
<li>Ensure that openldap is installed on the machine:
<pre># apt-get install slapd ldap-utils</pre></li>
<li>Debian will do a lot of magic and set up a skeleton LDAP server and get it running. We need to configure that further.</li>
<li>Let's set up logging before we forget. Create the following files in <code>/var/log</code>:
<pre># mkdir /var/log/ldap
# touch /var/log/ldap.log</pre></li>
<li>Set ownership correctly:
<pre># chown openldap:openldap /var/log/ldap</pre></li>
<li>Set up rsyslog to dump the LDAP logs into <code>/var/log/ldap.log</code> by adding the following lines:
<pre># vim /etc/rsyslog.conf
...
# Grab ldap logs, don't duplicate in syslog
local4.* /var/log/ldap.log</pre></li>
<li>Set up log rotation for these by creating the file [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/logrotate.d.ldap <code>/etc/logrotate.d/ldap</code>] with the following contents:
<pre>/var/log/ldap/*log {
weekly
missingok
rotate 1000
compress
delaycompress
notifempty
create 0640 openldap adm
postrotate
if [ -f /var/run/slapd/slapd.pid ]; then
/etc/init.d/slapd restart >/dev/null 2>&1
fi
endscript
}

/var/log/ldap.log {
weekly
missingok
rotate 24
compress
delaycompress
notifempty
}</pre></li>
<li>As of OpenLDAP 2.4, it doesn't actually create a config file for us. Apparently, this is a "feature": LDAP maintainers think we should want to set this up via dynamic queries. We don't, so the first thing we need is our [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/slapd.conf <code>slapd.conf</code>] file.</li></ul>

===== Building <code>slapd.conf</code> from scratch =====

<ul>
<li>Get a copy to work with:
<pre># scp uid@auth1.csclub.uwaterloo.ca:/etc/ldap/slapd.conf /etc/ldap/ ## you need CSC root for this</pre></li>
<li>You'll want to comment out the TLS lines, and anything referring to Kerberos and access for now. You'll also want to comment out lines specifically referring to syscom and office staff.</li>
<li>Make sure you remove the reference to <code>nonMemberTerm</code> as an index, as we're going to remove this field.</li>
<li>You'll also need to generate a root password for the LDAP to bootstrap auth, like so:
<pre># slappasswd
New password:
Re-enter new password:
{SSHA}longhash</pre></li>
<li>Add this line below <code>rootdn</code> in the <code>slapd.conf</code>:
<pre>rootpw {SSHA}longhash</pre></li>
<li>Now we want to edit all instances of "csclub" to be "wics" instead, e.g.:
<pre>suffix "dc=wics,dc=uwaterloo,dc=ca"
rootdn "cn=root,dc=wics,dc=uwaterloo,dc=ca"</pre></li>
<li>Next, we need to grab all the relevant schemas:
<pre>scp -r uid@auth1.csclub.uwaterloo.ca:/etc/ldap/schema/ /tmp/schemas</pre></li>
<li>Use the include directives to help you find the ones you need. I noticed we were missing <code>sudo.schema</code>, <code>csc.schema</code>, and <code>rfc2307bis.schema</code>.</li>
<li>Open up the [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/csc.schema <code>csc.schema</code>] for editing; we're not using it verbatim. Remove the attributes <code>studentid</code> and <code>nonMemberTerm</code> and the objectclass <code>club</code>. Also make sure you change the OID so we don't clash with the CSC. Because we didn't want to go through the process of requesting a [http://pen.iana.org/pen/PenApplication.page PEN number], we chose arbitrarily to use 26338, which belongs to IWICS Inc.</li>
<li>We also need to can the auto-generated config files, so do that:
<pre># rm -rf /etc/openldap/slapd.d/*</pre></li>
<li>Also nuke the auto-generated database:
<pre># rm /var/lib/ldap/__db.*</pre></li>
<li>Configure the database:
<pre># cp /usr/share/slapd/DB_CONFIG /var/lib/ldap/
# chown openldap:openldap /var/lib/ldap/DB_CONFIG </pre></li>
<li>Now we can generate the new configuration files:
<pre># slaptest -f /etc/ldap/slapd.conf -F /etc/ldap/slapd.d/</pre></li>
<li>And ensure that the permissions are all set correctly, lest this break something:
<pre># chown -R openldap:openldap /etc/ldap/slapd.d</pre></li>
<li>If at this point you get a nasty error, such as
<pre>5657d4db hdb_db_open: database "dc=wics,dc=uwaterloo,dc=ca": db_open(/var/lib/ldap/id2entry.bdb) failed: No such file or directory (2).
5657d4db backend_startup_one (type=hdb, suffix="dc=wics,dc=uwaterloo,dc=ca"): bi_db_open failed! (2)
slap_startup failed (test would succeed using the -u switch)</pre>
Just try restarting slapd, and see if that fixes the problem:
<pre># service slapd stop
# service slapd start</pre></li>
<li>Congratulations! Your LDAP service is now configured and running.</li></ul>

==== Getting TLS Up and Running ====

<ul>
<li>Now that we have our LDAP service, we'll want to be able to serve encrypted traffic. This is especially important for any remote access, since binding to LDAP (i.e. sending it a password for auth) occurs over plaintext, and we don't want to leak our admin password.</li>
<li>Our first step is to copy our SSL certificates into the correct places. Public ones go into <code>/etc/ssl/certs/</code> and private ones go into <code>/etc/ssl/private/</code>.</li>
<li>Since the LDAP daemon needs to be able to read our private cert, we need to grant LDAP access to the private folder:
<pre># chgrp openldap /etc/ssl/private
# chmod g+x /etc/ssl/private</pre></li>
<li>Next, uncomment the TLS-related settings in <code>slapd.conf</code>. These are <code>TLSCertificateFile</code> (the public cert), <code>TLSCertificateKeyFile</code> (the private key), <code>TLSCACertificateFile</code> (the intermediate CA cert), and <code>TLSVerifyClient</code> (set to "allow").
<pre># enable TLS connections
TLSCertificateFile /etc/ssl/certs/wics-wildcard.crt
TLSCertificateKeyFile /etc/ssl/private/wics-wildcard.key

# enable TLS client authentication
TLSCACertificateFile /etc/ssl/certs/GlobalSign_Intermediate_Root_SHA256_G2.pem
TLSVerifyClient allow</pre></li>
<li>Update all your LDAP settings:
<pre># rm -rf /etc/openldap/slapd.d/*
# slaptest -f /etc/ldap/slapd.conf -F /etc/ldap/slapd.d/
# chown -R openldap:openldap /etc/ldap/slapd.d</pre></li>
<li>And last, ensure that LDAP will actually serve <code>ldaps://</code> by modifying the init script variables in <code>/etc/default/</code>:
<pre># vim /etc/default/slapd
...
SLAPD_SERVICES="ldap:/// ldapi:/// ldaps:///"
...</pre></li>
<li>Now you can restart the LDAP server:
<pre># service slapd restart</pre></li>
<li>And assuming this is successful, test to ensure LDAP is serving on port 636 for <code>ldaps://</code>:
<pre># netstat -ntaup
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:389 0.0.0.0:* LISTEN 22847/slapd
tcp 0 0 0.0.0.0:636 0.0.0.0:* LISTEN 22847/slapd </pre></li></ul>

==== Populating the Database ====

Now you'll need to start adding objects to the database. While we'll want to mostly do this programmatically, there are a few entries we'll need to bootstrap.

===== Root Entries =====

<ul>
<li>Start by creating a file [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/tree.ldif <code>tree.ldif</code>] to create a few necessary "roots" in our LDAP tree, with the contents:
<pre>dn: dc=wics,dc=uwaterloo,dc=ca
objectClass: dcObject
objectClass: organization
o: Women in Computer Science
dc: wics

dn: ou=People,dc=wics,dc=uwaterloo,dc=ca
objectClass: organizationalUnit
ou: People

dn: ou=Group,dc=wics,dc=uwaterloo,dc=ca
objectClass: organizationalUnit
ou: Group</pre></li>
<li>Now attempt an LDAP add, using the password you set earlier:
<pre># ldapadd -cxWD cn=root,dc=wics,dc=uwaterloo,dc=ca -f tree.ldif
Enter LDAP Password:
adding new entry "dc=wics,dc=uwaterloo,dc=ca"

adding new entry "ou=People,dc=wics,dc=uwaterloo,dc=ca"

adding new entry "ou=Group,dc=wics,dc=uwaterloo,dc=ca"</pre></li>
<li>Test that everything turned out okay, by performing a query of the entire database:
<pre># ldapsearch -x -h localhost
# extended LDIF
#
# LDAPv3
# base <dc=wics,dc=uwaterloo,dc=ca> (default) with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# wics.uwaterloo.ca
dn: dc=wics,dc=uwaterloo,dc=ca
objectClass: dcObject
objectClass: organization
o: Women in Computer Science
dc: wics

# People, wics.uwaterloo.ca
dn: ou=People,dc=wics,dc=uwaterloo,dc=ca
objectClass: organizationalUnit
ou: People

# Group, wics.uwaterloo.ca
dn: ou=Group,dc=wics,dc=uwaterloo,dc=ca
objectClass: organizationalUnit
ou: Group

# search result
search: 2
result: 0 Success

# numResponses: 4
# numEntries: 3</pre></li></ul>

===== Users and Groups =====

<ul>
<li>Next, add users to track the current GID and UID. This will save us from querying the entire database every time we make a new user or group. Create this file, [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/nextxid.ldif <code>nextxid.ldif</code>]:
<pre>dn: uid=nextuid,ou=People,dc=wics,dc=uwaterloo,dc=ca
cn: nextuid
objectClass: account
objectClass: posixAccount
objectClass: top
uidNumber: 20000
gidNumber: 20000
homeDirectory: /dev/null

dn: cn=nextgid,ou=Group,dc=wics,dc=uwaterloo,dc=ca
objectClass: group
objectClass: posixGroup
objectClass: top
gidNumber: 10000</pre>
You'll see here that our first GID is 10000 and our first UID is 20000.</li>
<li>Now add them, like you did with the roots of the tree:
<pre># ldapadd -cxWD cn=root,dc=wics,dc=uwaterloo,dc=ca -f nextxid.ldif
Enter LDAP Password:
adding new entry "uid=nextuid,ou=People,dc=wics,dc=uwaterloo,dc=ca"

adding new entry "cn=nextgid,ou=Group,dc=wics,dc=uwaterloo,dc=ca"</pre></li></ul>

===== Special <code>sudo</code> Entries =====

<ul>
<li>We also need to add a sudoers OU with a defaults object for default sudo settings. We also need entries for syscom, such that members of the syscom group can use sudo on all hosts, and for termcom, whose members can use sudo on only the office terminals. Call this one [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/sudoers.ldif <code>sudoers.ldif</code>]:
<pre>dn: ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca
objectClass: organizationalUnit
ou: SUDOers

dn: cn=defaults,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca
objectClass: top
objectClass: sudoRole
cn: defaults
sudoOption: !lecture
sudoOption: env_reset
sudoOption: listpw=never
sudoOption: mailto="wics-sys@lists.uwaterloo.ca"
sudoOption: shell_noargs

dn: cn=%syscom,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca
objectClass: top
objectClass: sudoRole
cn: %syscom
sudoUser: %syscom
sudoHost: ALL
sudoCommand: ALL
sudoRunAsUser: ALL

dn: cn=%termcom,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca
objectClass: top
objectClass: sudoRole
cn: %termcom
sudoUser: %termcom
sudoHost: honk
sudoHost: hiss
sudoHost: gosling
sudoCommand: ALL
sudoRunAsUser: ALL</pre></li>
<li>Now add them:
<pre># ldapadd -cxWD cn=root,dc=wics,dc=uwaterloo,dc=ca -f sudoers.ldif
Enter LDAP Password:
adding new entry "ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca"

adding new entry "cn=defaults,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca"

adding new entry "cn=%syscom,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca"

adding new entry "cn=%termcom,ou=SUDOers,dc=wics,dc=uwaterloo,dc=ca"</pre></li>
<li>Last, add some special local groups via [https://git.uwaterloo.ca/wics/documentation/blob/master/ldap/local-groups.ldif <code>local-groups.ldif</code>]:
<pre># ldapadd -cxWD cn=root,dc=wics,dc=uwaterloo,dc=ca -f local-groups.ldif</pre>
The local groups are special because they usually are present on all systems, but we want to be able to add users to them at the LDAP level. For instance, the audio group controls access to sound equipment, and the adm group controls log read access.</li>
<li>That's all the entries we have to add manually! Now we can use software for the rest. See [[weo|<code>weo</code>]] for more details.</li></ul>

=== Querying LDAP ===

There are many tools available for issuing LDAP queries. Queries should be issued to <tt>ldap1.csclub.uwaterloo.ca</tt>. The search base you almost certainly want is <tt>dc=csclub,dc=uwaterloo,dc=ca</tt>. Read access is available without authentication; [[Kerberos]] is used to authenticate commands which require it.

Example:

ldapsearch -x -h ldap1.csclub.uwaterloo.ca -b dc=csclub,dc=uwaterloo,dc=ca uid=ctdalek

The <tt>-x</tt> option causes <tt>ldapsearch</tt> to switch to simple authentication rather than trying to authenticate via SASL (which will fail if you do not have a Kerberos ticket).

The University LDAP server (uwldap.uwaterloo.ca) can also be queried like this. Again, use "simple authentication" as read access is available (from on campus) without authentication. SASL authentication will fail without additional parameters.

Example:

ldapsearch -x -h uwldap.uwaterloo.ca -b dc=uwaterloo,dc=ca "cn=Prabhakar Ragde"

=== Replication ===

While <tt>ldap1.csclub.uwaterloo.ca</tt> ([[Machine_List#auth1|auth1]]) is the LDAP master, an up-to-date replica is available on <tt>ldap2.csclub.uwaterloo.ca</tt> ([[Machine_List#auth2|auth2]]).

In order to replicate changes from the master, the slave maintains an authenticated connection to the master which provides it with full read access to all changes.

Specifically, <tt>/etc/systemd/system/k5start-slapd.service</tt> maintains an active Kerberos ticket for <tt>ldap/auth2.csclub.uwaterloo.ca@CSCLUB.UWATERLOO.CA</tt> in <tt>/var/run/slapd/krb5cc</tt>. This is then used to authenticate the slave to the server, who maps this principal to <tt>cn=ldap-slave,dc=csclub,dc=uwaterloo,dc=ca</tt>, which in turn has full read privileges.

In the event of master failure, all hosts should fail LDAP reads seamlessly over to the slave.

[[Category:Software]]

== Changing a user's username ==

Only a member of the Systems Committee can change a user's username. '''At all times, a user's username must match the user's username in WatIAM.'''

# Edit entries in LDAP (<code>ldapvi -Y GSSAPI</code>)
#* Find and replace the user's old username with the new one
# Change the user's Kerberos principal (on auth1, <code>renprinc $OLD $NEW</code>)
# Move the user's home directory (on aspartame)
# Change the user's csc-general (and csc-industry, if subscribed) email address for $OLD@csclub.uwaterloo.ca to $NEW@csclub.uwaterloo.ca
#* https://mailman.csclub.uwaterloo.ca/admin/csc-general
# If the user has vhosts on caffeine, update them to point to their new username

If the user's account has been around for a while, and they request it, forward email from their old username to their new one.

# Edit <code>/etc/aliases</code> on mail. <code>$OLD: $NEW</code>
# Run <code>newaliases</code>

Mirror

2018-12-10T00:06:59Z

Ztseguin: Added information about mirror-dc

The [https://csclub.uwaterloo.ca Computer Science Club] runs a public mirror ([http://mirror.csclub.uwaterloo.ca mirror.csclub.uwaterloo.ca]) on [[Machine_List#potassium-benzoate|potassium-benzoate]].

''We are listed on the ResNet "don't count" list, so downloading from our mirror will not count against one's ResNet quota.''

== Software Mirrored ==

A list of current archives (and their respective disk usage) is listed on our mirror's homepage at [http://mirror.csclub.uwaterloo.ca mirror.csclub.uwaterloo.ca].

=== Mirroring Requests ===

Requests to mirror a particular distribution or archive should be made to [mailto:syscom@csclub.uwaterloo.ca syscom@csclub.uwaterloo.ca].

== Implementation Details ==

=== Syncing ===

==== Storage ====

All of our projects are stored on one of two zfs zpools. There are 8 drives per array, configured as raidz2, and there is an additional drive that can be swapped in (in the event of a disk failure).

* <code>/mirror/root/.cscmirror1</code>
* <code>/mirror/root/.cscmirror2</code>

Each project is given a filesystem under one of the two pools. Symlinks are created <code>/mirror/root</code> to point to the correct pool and file system.

==== Merlin ====

The synchronization process is run by a Python script called "merlin", written by a2brenna. The script is stored in <code>~mirror/merlin</code>.

The list of repositories and their configuration (synch frequency, location, etc.) is configured in <code>merlin.py</code>.

To view the sync status, execute <code>~mirror/merlin/arthur.py status</code>. To force the sync of a project, execute <code>~mirror/merlin/arthur.py sync:PROJECT_NAME</code>.

===== Push Sync =====

Some projects support push syncing via SSH.

We are running a special SSHD instance on mirror.csclub.uwaterloo.ca:22. This instance has been locked down, with the following settings:

* Only SSH key authentication
* Only users of the <code>push</code> group (except <code>mirror</code>) are allowed to connect
* X11 Forwarding, TCP Forwarding, Agent Forwarding, User RC and TTY are disabled
* Users are chrooted to <code>/mirror/merlin</code>

Most projects will connect using the <code>push</code> user. The SSH authorized keys file is located at <code>/home/push/.ssh/authorized_keys</code>. An example entry is:

<pre>
restrict,no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command="arthur sync:ubuntu >/dev/null 2>/dev/null </dev/null &",from="XXX.XXX.XXX.XXX" ssh-rsa ...
</pre>

==== Sync Scripts ====

Our collection of synchronization scripts are located in <code>~mirror/bin</code>. They currently include:

* <code>csc-sync-apache</code>
* <code>csc-sync-debian</code>
* <code>csc-sync-debian-cd</code>
* <code>csc-sync-gentoo</code>
* <code>csc-sync-ssh</code>
* <code>csc-sync-standard</code>

Most of these scripts take the following parameters:

<code>local_dir rsync_host rsync_dir</code>

=== HTTP(s) ===

We use [https://nginx.org nginx] as our webserver.

==== Index ====

An index of the archives we mirror is available at [http://mirror.csclub.uwaterloo.ca mirror.csclub.uwaterloo.ca].

As of Winter 2010, it is now generated by a Python script in <code>~mirror/mirror-index</code>.

<code>~mirror/mirror-index/make-index</code> is scheduled in <code>/etc/cron.d/csc-mirror</code> to be run at 5:40am on the 14th and 28th of each month. The script can be run manually when needed (for example, when the archive list is updated) by running:

<code>sudo -u mirror /home/mirror/mirror-index/make-index.py</code>

This causes an instance of <code>du</code> which computes the size of each directory. This list is then sorted alphabetically by directory name and returned to the Python script. If any errors occur during this process, the script conservatively chooses to exit rather than risk generating an index file that is incorrect.

<code>make-index.py</code> is configured by means of a [https://yaml.org YAML] file, <code>config.yaml</code>, in the same directory. Its format is as follows:

<pre class="yaml">docroot: /mirror/root
duflags: --human-readable --max-depth=1
output: /mirror/root/index.html

exclude:
- include
- lost+found
- pub
# (...)

directories:
apache:
site: apache.org
url: http://www.apache.org/

archlinux:
site: archlinux.org
url: http://www.archlinux.org/

# (...)</pre>
The docroot is the directory which is to be scanned; this will probably always be the mirror root from which Apache serves. <code>duflags</code> specifies the flags to be passed to <code>du</code>. This is here so that it's easy to find and alter. For instance, we could change <code>--human-readable</code> to <code>--si</code> if we ever decided that, like hard disk manufacturers, we want sizes to appear larger than they are. <code>output</code> defines the file to which the generated index will be written.

<code>exclude</code> specifies the list of directories which will not be included in the generated index page (since, by default, all folders are included in the generated index page).

Finally, <code>directories</code> specifies the list of directories to be listed. The format is fairly straightforward: simply name the directory and provide a site (the display name in the "Project Site" column) and URL. One caveat here is that YAML does not allow tabs for whitespace. Indent with two spaces to remain consistent with the existing file format, please. Also note that the directory name is case-sensitive, as is always the case on Unix.

Finally, the HTML index file is generated from <code>index.mako</code>, a Mako template (which is mostly HTML anyhow). If you really can't figure out how it works, look up the Mako documentation.

=== FTP ===

We use [http://www.proftpd.org/ proftpd] (standalone daemon) as our FTP server.

To increase performance, we disable DNS lookups in <code>proftpd.conf</code>:

<pre>UseReverseDNS off
IdentLookups off</pre>
We also limit the amount of CPU/memory resources used (e.g. to minimize [https://en.wikipedia.org/wiki/Globbing Globbing] resources):

<pre>RLimitCPU session 10
RLimitMemory session 4096K</pre>
We allow a maximum of 500 concurrent FTP sessions:

<pre>MaxInstances 500
MaxClients 500</pre>
The contents of <code>/mirror/root/include/motd.msg</code> are displayed when a user connects.

=== rsync ===

We use <code>rsyncd</code> (standalone daemon).

We disable compression and checksumming in <code>rsyncd.conf</code>:

<pre>dont compress = *
refuse options = c delete</pre>
The contents of <code>/mirror/root/include/motd.msg</code> are displayed when a user connects.

== Mirror Administration ==

=== Adding a new project ===

# Find the instructions for mirroring the project. Ideally, try to sync directly from the project’s source repository.
#* Note that some projects provide sync scripts, however we generally won’t use them. We will instead use our custom ones.
# Create a zfs filesystem to store the project in:
#* Find the pool with least current disk usage
#* <code>zfs create cscmirror{1,2}/$PROJECT_NAME</code>
# Change the folder ownership
#* <code>chown mirror:mirror /mirror/root/.cscmirror{1,2}/$PROJECT_NAME</code>
# Create the symlink in <code>/mirror/root</code>
#* <code>ln -s .cscmirror{1,2}/$PROJECT_NAME $PROJECT_NAME</code> ('''NOTE''': The symlink must be relative to the <code>/mirror/root</code> directory. If it isn’t, the symlinks will not work when chrooted)
# Repeat the above steps on mirror-dc. <code>sudo ssh mirror-dc</code> on potassium-benzoate
# Configure the project in merlin (<code>~mirror/merlin/merlin.py</code>)
#* Select the appropriate sync script (typically <code>csc-sync-standard</code>) and supply the appropriate parameters
# Restart merlin: <code>systemctl restart merlin</code>
# Configure the project in zfssync.yml (<code>~mirror/merlin/zfssync.yml</code>)
#* This will kick off the initial sync
#* Check <code>~mirror/merlin/logs/$PROJECT_NAME</code> for errors, <code>~mirror/merlin/logs/transfer.log</code> for transfer progress
# Update the mirror index configuration (<code>~mirror/mirror-index/config.yaml</code>)
# Add the project to rsync (<code>/etc/rsyncd.conf</code>)
#* Restart rsync with <code>systemctl restart rsync</code>

If push mirroring is available/required, see [[#Push_Sync|Push Sync]].

=== Secondary Mirror ===

The School of Computer Science's CSCF has provided us with a secondary mirror machine located in DC. This will limit the downtime of mirror.csclub in the event of an outage affecting the MC machine room.

==== Keepalived ====

Mirror's IP addresses (129.97.134.71 and 2620:101:f000:4901:c5c::f:1055) have been configured has VRRP address on both machines. Keepalived does the monitoring and selecting of the active node.

Potassium-benzoate has higher priority and will typically be the active node. A node's priority is reduced when nginx, proftpd or rsync are not running. Potassium-benzoate starts with a score of 100 and mirror-dc starts with a priority of 90 (higher score wins).

When nginx is unavailable (checked w/ curl), the priority is reduced by 20. When proftpd is unavailable (checked with curl), the priority is reduced by 5. When rsync is unavailable (checking with rsync), the priority is reduced by 15.

The Systems Committee should received an email when the nodes swap position.

==== Project synchronization ====

Only potassium-benzoate is configure with merlin. mirror-dc has the software components, but they are probably not update to date nor configured to run correctly.

When a project sync is complete, merlin will kick off a custom script to sync the zfs dataset to the other node. These scripts live in /usr/local/bin and in ~mirror/merlin.

2018-07-16T22:46:40Z

Ztseguin: /* Timeline */

There is a planned power outage in MC from Tuesday, August 21 to Friday, August 24.

There is also a one-day outage in DC, which will complicate keeping services up during the entire outage.

== Timeline ==

=== Before Monday, August 20 ===

* Complete plan for outage
* Move equipment to DC (if necessary)
* Take backups of LDAP and Kerberos, and download offsite

=== Monday, August 20 ===

* Shutdown general-use computing services
* Transfer computing services to redundant / temporary systems

=== Sometime during the outage window ===

* Shutdown DC systems before the building outage

=== After the outage ===

* Being restoring normal services

== Systems ==

=== Mirror ===

TODO. Syscom is currently working with CSCF to identify a plan for mirror.

=== Website ===

The CSC website is a static site, and will be straightforward to maintain during the outage.

All user and club sites are hosted in home directories (which are unavailable), so we will display an outage page (with a 503 status code).

=== Mail ===

Since the outage is for a week, we need to maintain email services during the outage. An initial plan by ztseguin and jxpryde:

* rsync users' .forward, .procmailrc and .maildir to a local directory on mail, allowing mail to continue as expected

However, this requires:

* Users not reference any scripts, programs, etc. in their procmailrc file that reference things in their home directory

=== Authentication ===

Authentication is located in both MC and PHY.

While the MC node is down, the PHY node can continue to answer to authentication requests. However, updating membership and changing passwords will not be available if the MC node is down.

We may consider moving auth1 to DC for the outage.

=== DNS ===

CSC's DNS service is located in both MC and PHY.

We may consider moving the MC DNS node to DC, but this is not necessary to maintain services during the outage.

''NOTE: The MC node is the master node, and we will need to ensure that the SOA record contains a long enough expiry time so the PHY doesn't stop serving zones.''

== Additional Resources ==

* https://uwaterloo.ca/information-systems-technology/news/important-significant-interruption-service-delivery-due
* https://istns.uwaterloo.ca/uwna/index.php?s=3561