New CSC Machine: Difference between revisions
No edit summary |
(Included a hint for the location of where the passwords are stored for convenience.) |
||
(87 intermediate revisions by 16 users not shown) | |||
Line 1: | Line 1: | ||
= Firmware Updates = |
|||
Draft of how to set up a new CSC machine |
|||
Vendors such as Dell provide firmware updates that should be applied before putting new machines into service. Even if the machine's warranty has expired, security updates are still made available. |
|||
Netboot: |
|||
It is recommended to use the following sequence when updating firmware on the Dell PowerEdge servers ([https://downloads.dell.com/solutions/general-solution-resources/White%20Papers/Recommended%20Workflow%20for%20Performing%20Firmware%20Updates%20on%20PowerEdge%20Servers.pdf]): |
|||
# iDRAC |
|||
# Lifecycle Controller |
|||
# BIOS |
|||
# Diagnostics |
|||
# OS Driver Pack |
|||
# RAID |
|||
# NIC |
|||
# PSU |
|||
# CPLD |
|||
# Other update |
|||
For consumer grade hardware, go to the motherboard vendor's website and find the way to upgrade the firmware. |
|||
= Booting = |
|||
* Put the TFTP image in place (if dist-arch pair installed before, you may skip this). |
* Put the TFTP image in place (if dist-arch pair installed before, you may skip this). |
||
e.g. extract http://mirror.csclub.uwaterloo.ca/ubuntu/dists/oneiric/main/installer-amd64/current/images/netboot/netboot.tar.gz to caffeine:/srv/tftp/oneiric-amd64 |
e.g. extract http://mirror.csclub.uwaterloo.ca/ubuntu/dists/oneiric/main/installer-amd64/current/images/netboot/netboot.tar.gz to caffeine:/srv/tftp/oneiric-amd64 |
||
* Force network boot in the BIOS. This may be called "Legacy LAN" or other such cryptic things. If this doesn't work, boot from CD or USB instead. |
|||
* Configure DHCP server; see caffeine:/etc/dhcp/dhcpd.conf |
|||
[todo: example] |
|||
It is preferred to use the "alternate" Ubuntu installer image, based on debian-installer, instead of the Ubiquity installer. This installer supports software RAID and LVM out of the box, and will generally make your life easier. If installing Debian, this is the usual installer, so don't sweat it. |
|||
* Most of our newer servers (e.g. PowerEdge R815) need non-free firmware in order to boot. This means that if you are using a new netboot image, it is highly recommended to include the entire non-free firmware bundle in the boot image. See [https://wiki.debian.org/DebianInstaller/NetbootFirmware] for more information. |
|||
* For office terminals, create a boot USB (via dd, for example) and boot from USB. |
|||
= Installing = |
|||
== debian-installer == |
|||
At least in expert mode, you can choose a custom mirror (top of the countries list) and give the path for mirror directly. This will make installation super-fast compared to installing from anywhere else. |
|||
Please install to LVM volumes, as this is our standard configuration on all machines where possible. It allows more flexible partitioning across available volumes. Since GRUB 2, even /boot may be on LVM; this is the preferred configuration for simplicity, except when legacy partitioning setups make this inconvenient. |
|||
You may enable unattended upgrades, but do not enable Canonical's remote management service or any such nonsense. This is mostly a straightforward Debian/Ubuntu install. |
|||
= After Installing = |
|||
Add the machine's name to ~git/public/hosts.git, and run the ansible playbook (https://git.uwaterloo.ca/csc/playbooks/blob/master/update-hosts.yml) to distribute the updated hosts file to all machines. |
|||
== apt == |
|||
Delete/clear the file <tt>/etc/apt/sources.list</tt> and paste something like the following into <tt>/etc/apt/sources.list.d/debian.sources</tt> (replace "bookworm" by the the current Debian stable codename): |
|||
<pre> |
|||
Types: deb |
|||
URIs: http://mirror.csclub.uwaterloo.ca/debian |
|||
Suites: bookworm bookworm-updates bookworm-backports |
|||
Components: main contrib non-free non-free-firmware |
|||
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg |
|||
Types: deb |
|||
URIs: http://mirror.csclub.uwaterloo.ca/debian-security |
|||
Suites: bookworm-security |
|||
Components: main contrib non-free non-free-firmware |
|||
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg |
|||
</pre> |
|||
Install the CSC archive signing key: |
|||
<pre> |
|||
wget -O /etc/apt/keyrings/csclub.gpg http://debian.csclub.uwaterloo.ca/csclub.gpg |
|||
</pre> |
|||
Paste the following into <tt>/etc/apt/sources.list.d/csclub.sources</tt> (or copy from another host): |
|||
<pre> |
|||
Types: deb |
|||
URIs: http://debian.csclub.uwaterloo.ca |
|||
Suites: bookworm |
|||
Components: main |
|||
Signed-By: /etc/apt/keyrings/csclub.gpg |
|||
</pre> |
|||
In order to make Debian use packages in our repository by default, set our repository to the highest priority. Create <code>/etc/apt/preferences.d/99-csclub</code>: <syntaxhighlight> |
|||
Package: * |
|||
Pin: origin debian.csclub.uwaterloo.ca |
|||
Pin-Priority: 1001 |
|||
</syntaxhighlight>You should now run <tt>apt-get update</tt> to reflect these changes. |
|||
For unattended upgrades in the future, install the <tt>unattended-upgrades</tt> package and copy <tt>/etc/apt/apt.conf</tt> from another host. |
|||
== Network == |
|||
Note that debian 11 will use NetworkManager or <code>/etc/interfaces</code> by default if you install a desktop environment, which doesn't seem to do DHCPv6 nicely. For simplicity and consistency across machines, we will use <code>systemd-networkd</code>. First stop and disable NetworkManager:<syntaxhighlight lang="bash"> |
|||
systemctl disable --now NetworkManager.service networking.service |
|||
apt autoremove NetworkManager |
|||
</syntaxhighlight>Then, create a network configuration file at <code>/etc/systemd/network/10-wired.network</code>:<syntaxhighlight> |
|||
[Match] |
|||
# Check the interface name using `ip a` |
|||
Name=enp3s0 |
|||
[Network] |
|||
# DHCP for IPv4 should work just fine |
|||
DHCP=ipv4 |
|||
# IPv6 doesn't seem to work properly. Manually set them here |
|||
Address=ALLOCATED_IPv6_ADDRESS |
|||
Gateway=IPv6_GATEWAY |
|||
</syntaxhighlight>Then start and enable <code>systemd-networkd.service</code>. Also remember to specify the campus DNS at <code>/etc/resolve.conf</code>. You can copy it from another CSC machine. |
|||
== Kerberos keys == |
|||
If this is a reinstall of an existing host, copy back the SSH host keys and <tt>/etc/krb5.keytab</tt> from its former incarnation. Otherwise, create a new Kerberos principal and copy the keytab over, as follows (run from the host in question):<syntaxhighlight> |
|||
kadmin -p sysadmin/admin # or any other admin principal; the password for this one is the usual root password |
|||
addprinc -randkey host/[hostname].csclub.uwaterloo.ca |
|||
ktadd host/[hostname].csclub.uwaterloo.ca |
|||
</syntaxhighlight>This will generate a new principal (you can skip this step if one already exists) and add it to the local Kerberos keytab. |
|||
== Configuration == |
|||
=== General === |
|||
Install packages that we will need:<syntaxhighlight lang="bash"> |
|||
apt install krb5-user nfs-common nslcd sudo-ldap |
|||
# This package are automatically installed already, but we need to install our version so that NFS can connect to our crappy NetApp server |
|||
apt install --reinstall libk5crypto3 |
|||
</syntaxhighlight>The following config files are needed to work in the CSC environment (examples given below for an office terminal; perhaps refer to another host if preferred). |
|||
<tt>/etc/nsswitch.conf</tt><syntaxhighlight> |
|||
# /etc/nsswitch.conf |
|||
# |
|||
# Example configuration of GNU Name Service Switch functionality. |
|||
# If you have the `glibc-doc-reference' and `info' packages installed, try: |
|||
# `info libc "Name Service Switch"' for information about this file. |
|||
passwd: files systemd ldap |
|||
group: files systemd ldap |
|||
shadow: files ldap |
|||
gshadow: files ldap |
|||
sudoers: files ldap |
|||
hosts: files dns |
|||
networks: files |
|||
protocols: db files |
|||
services: db files |
|||
ethers: db files |
|||
rpc: db files |
|||
</syntaxhighlight><tt>/etc/ldap/ldap.conf</tt><syntaxhighlight> |
|||
# |
|||
# LDAP Defaults |
|||
# |
|||
# See ldap.conf(5) for details |
|||
# This file should be world readable but not world writable. |
|||
BASE dc=csclub, dc=uwaterloo, dc=ca |
|||
URI ldaps://ldap1.csclub.uwaterloo.ca ldaps://ldap2.csclub.uwaterloo.ca |
|||
SIZELIMIT 0 |
|||
TLS_CACERT /etc/ssl/certs/ca-certificates.crt |
|||
TLS_CACERTFILE /etc/ssl/certs/ca-certificates.crt |
|||
SUDOERS_BASE ou=SUDOers,dc=csclub,dc=uwaterloo,dc=ca |
|||
</syntaxhighlight>Also make <tt>/etc/sudo-ldap.conf</tt> a symlink to the above. On debian, install <tt>sudo-ldap</tt> package too. |
|||
<tt>/etc/nslcd.conf</tt><syntaxhighlight> |
|||
# /etc/nslcd.conf |
|||
# nslcd configuration file. See nslcd.conf(5) |
|||
# for details. |
|||
# The user and group nslcd should run as. |
|||
uid nslcd |
|||
gid nslcd |
|||
# The location at which the LDAP server(s) should be reachable. |
|||
uri ldaps://ldap1.csclub.uwaterloo.ca |
|||
uri ldaps://ldap2.csclub.uwaterloo.ca |
|||
# The search base that will be used for all queries. |
|||
base dc=csclub, dc=uwaterloo, dc=ca |
|||
# The LDAP protocol version to use. |
|||
#ldap_version 3 |
|||
# The DN to bind with for normal lookups. |
|||
#binddn cn=annonymous,dc=example,dc=net |
|||
#bindpw secret |
|||
# The DN used for password modifications by root. |
|||
#rootpwmoddn cn=admin,dc=example,dc=com |
|||
# SSL options |
|||
#ssl off |
|||
tls_reqcert demand |
|||
tls_cacertfile /etc/ssl/certs/ca-certificates.crt |
|||
# The search scope. |
|||
#scope sub |
|||
map group member uniqueMember |
|||
</syntaxhighlight><tt>/etc/krb5.conf</tt><syntaxhighlight> |
|||
[libdefaults] |
|||
default_realm = CSCLUB.UWATERLOO.CA |
|||
forwardable = true |
|||
proxiable = true |
|||
dns_lookup_kdc = false |
|||
dns_lookup_realm = false |
|||
allow_weak_crypto = true |
|||
[realms] |
|||
CSCLUB.UWATERLOO.CA = { |
|||
kdc = kdc1.csclub.uwaterloo.ca |
|||
kdc = kdc2.csclub.uwaterloo.ca |
|||
admin_server = kadmin.csclub.uwaterloo.ca |
|||
} |
|||
(rest omitted for brevity, see any CSC machine) |
|||
</syntaxhighlight>Notably, <tt>allow_weak_crypto</tt> is currently needed to mount <tt>/users</tt> (/music and <tt>/scratch</tt> is sec=sys and thus will always mount, even when krb5 is down and/or broken). Otherwise, you will get a mysterious "permission denied" error (even though the server claims to have authenticated the mount successfully). |
|||
Furthermore, the lines <tt>dns_lookup_kdc</tt> and <tt>dns_lookup_realm</tt> have been added - they are needed to stop the KDC from throwing its arms in the air and giving up if IST's DNS servers ever explode - an event that has happened in the recent past far more often than I'd like it to. |
|||
Change all lines in <tt>/etc/pam.d/common-*</tt> to have <tt>minimum_uid=10000</tt> so that Kerberos won't interfere with local users. Note that pam configs are notably different on syscom-only hosts. Look at an existing syscom-only host to see the difference. |
|||
Alter <tt>/etc/default/nfs-common</tt> <syntaxhighlight> |
|||
# Alter these lines: |
|||
NEED_STATD=1 |
|||
NEED_GSSD=1 |
|||
# -l for gssd is to allow legacy crypto suites |
|||
RPCGSSDOPTS="-v -l" |
|||
</syntaxhighlight>to enable <tt>statd</tt>, and more importantly <tt>gssd</tt> (needed for Kerberos NFS mounts). Start <code>rpc-statd.service</code> and <code>rpc-gssd.service</code> manually for now. |
|||
Add <tt>/users</tt>, <tt>/music</tt> and <tt>/scratch</tt> to <tt>/etc/fstab</tt> (as appropriate for the machine's role), make their mount points and mount them. Note that <tt>/scratch</tt> are sec=sys whereas <tt>/music</tt> and /users is sec=krb5p (with exceptions granted on a case-by-case basis for servers only, office terminals are always sec=krb5p for security reasons). |
|||
To allow single sign-on as <tt>root</tt> (primarily useful for pushing files to all machines simultaneously), put the following in <tt>/root/.k5login</tt>: |
|||
sysadmin/admin@CSCLUB.UWATERLOO.CA |
|||
Also copy the following files from another CSC host: |
|||
* <tt>/etc/ssh/ssh_config</tt> and <tt>/etc/ssh/sshd_config</tt> (for single sign-on) |
|||
* <tt>/etc/ssh/ssh_known_hosts</tt> (to remove hostkey warnings within our network) |
|||
* <tt>/etc/hosts</tt> (for host tab completion and emergency name resolution) |
|||
* <tt>/etc/resolv.conf</tt> (to use IST's nameservers and search csclub/uwaterloo domains. Only required if you are not using <tt>/etc/network/interfaces</tt> to configure DNS) |
|||
=== Audio === |
|||
On an office terminal, copy <tt>/etc/pulse/default.pa</tt> from another office terminal. |
|||
If this is to be the machine that actually plays audio (currently <tt>nullsleep</tt>), the setup is slightly more complicated. You'll need to set up MPD and PipeWire to receive connections, and store the PulseAudio cookie in <tt>~audio</tt>, with appropriate permissions so that only the <tt>audio</tt> group can access it. If this is a new audio machine, you'll also need to change <tt>default.pa</tt> on all office terminals to point to it. |
|||
=== Password === |
|||
Change the root password to the specified password in the usual place under the termcom user. If it's an office terminal, change the local user's password to the one specified in the usual place. |
|||
=== Prevent suspend and hibernation (Office Terminal) === |
|||
Set <code>AllowSuspend</code>, <code>AllowHibernation</code>, <code>AllowSuspendThenHibernate</code> and <code>AllowHybridSleep</code> all to <code>no</code> in <code>/etc/systemd/sleep.conf</code>, and reboot. |
|||
== Records == |
|||
You probably already created the host in the University IPAM system beforehand. If not, please do so. |
|||
Please also add the host to the [[Machine List]] here on the Wiki. |
|||
== Munin (System Monitoring) == |
|||
If the new machine is not a container, you probably want to have it participate in the Munin cluster. Run <tt>apt-get install munin-node</tt> to install the monitoring client, then |
|||
edit the file /etc/munin/munin-node.conf. Look for a line that says <tt>allow ^127\.0\.0\.1$</tt> and add the following on a new line immediately below it: |
|||
<tt>allow ^129\.97\.134\.51$</tt> (this is the IP address for munin.csclub). Save the file, then <tt>/etc/init.d/munin-node restart</tt> and <tt>update-rc.d munin-node defaults</tt>. |
|||
Then, ssh into munin.csclub and edit the file /etc/munin/munin.conf and add the following lines to the end: |
|||
<tt> |
|||
[NEW-MACHINE-NAME.csclub] <br/> |
|||
addr 129.97.134.### <br /> |
|||
use_node_name yes</tt> |
|||
== Prometheus (System Monitoring) == |
|||
We are currently using Prometheus to monitor our systems. On the new machine, install <tt>prometheus-node-exporter</tt> and <tt>stunnel</tt>. |
|||
Change <tt>/etc/default/prometheus-node-exporter</tt> to this: |
|||
ARGS="--web.listen-address=localhost:9101" |
|||
and start <tt>prometheus-node-exporter.service</tt>. |
|||
Then set up stunnel. Create <tt>/etc/stunnel/prometheus-node-exporter.conf</tt> with this content: |
|||
setuid = stunnel4 |
|||
setgid = stunnel4 |
|||
pid = /var/run/stunnel4/exporter.pid |
|||
debug = 7 |
|||
[prometheus-node-exporter] |
|||
accept = 0.0.0.0:9100 |
|||
connect = 127.0.0.1:9101 |
|||
CAfile = /etc/stunnel/tls/server.crt |
|||
cert = /etc/stunnel/tls/node.crt |
|||
key = /etc/stunnel/tls/node.key |
|||
verifyPeer = yes |
|||
Copy <tt>/etc/stunnel/{node.crt, node.key, server.crt}</tt> from <tt>prometheus:/opt/prometheus/tls</tt> or the same location on other machines. |
|||
Finally, start <tt>stunnel4.service</tt>. |
|||
If it's a new machine, you'll also need to add it to the list of monitoring at <tt>prometheus:/opt/prometheus/prometheus.yml</tt>. Add it under a suitable label (or create a new label) in 'node_exporter' job. |
|||
= New Distribution = |
|||
If you're adding a new distribution, there a couple of steps you'll need to take in updating the CSClub Debian repository on [[Machine_List#sodium_benzoate|sodium-benzoate/mirror]]. |
|||
The steps to add a new Debian release (in the examples, jessie) is as follows, modify as necessary: |
|||
=== Step 0: Create a GPG key === |
|||
Use "gpg --gen-key" or something like that. Skip this if you already have one. |
|||
=== Step 1: Add to Uploaders === |
|||
The /srv/debian/conf/uploaders file on mirror contains the list of people who can upload. Add your GPG key id to this file. Use "gpg --list-secret-keys" to find out the key ID. You also need to import your key into the mirror's gpg homedir as follows: |
|||
gpg --export $KEYID | sudo env GNUPGHOME=/srv/debian/gpg gpg --import |
|||
You only need to do this step once. |
|||
=== Step 2: Add Distro === |
|||
Add a new section to /srv/debian/conf/distributions: |
|||
Origin: CSC |
|||
Label: Debian |
|||
Codename: '''jessie''' |
|||
Architectures: alpha amd64 i386 mips mipsel sparc powerpc armel source |
|||
Components: main contrib non-free |
|||
Uploaders: uploaders |
|||
Update: dell chrome |
|||
SignWith: yes |
|||
Log: '''jessie'''.log |
|||
--changes notifier |
|||
And update the '''Allow''' line in /srv/debian/conf/incoming: |
|||
Allow: '''jessie>jessie''' oldstable>squeeze stable>wheezy lucid>lucid maverick>maverick oneiric>oneiric precise>precise quantal>quantal |
|||
=== Step 3: Update from Sources === |
|||
Run: |
|||
sudo env GNUPGHOME=/srv/debian/gpg /srv/debian/bin/rrr-update |
|||
If all went well you should see the new distribution listed at http://debian.csclub.uwaterloo.ca/dists/ |
|||
=== Step 4: CSC Packages === |
|||
Now that we've got our new distribution set up we need to generate our packages and have them uploaded. Namely, ceo and libpam-csc. For libpam-csc: |
|||
Get the package: |
|||
git clone https://git.csclub.uwaterloo.ca/public/libpam-csc.git |
|||
* Force network boot |
|||
cd libpam-csc |
|||
Update change log: |
|||
Else boot CD |
|||
EMAIL=[you]@csclub.uwaterloo.ca NAME="Your Name" dch -i |
|||
Update as necessary, i.e: |
|||
For Ubuntu installer to have lvm support, apt-get install lvm2. If you still can't see the partitions (even if lvscan sees them, but no devices exist), just rename a logical volume something else and back. This makes LVM tell udev about the device. Now the partitioner should be able to see them. We prefer to use LVM for partitions (except /boot, which GRUB prefers not to be on LVM). |
|||
libpam-csc (1.10'''jessie0''') '''jessie'''; urgency=low |
|||
Regenerate the initramfs. |
|||
* Packaging for jessie. |
|||
-- Your Name <[you]@csclub.uwaterloo.ca> Thu, 10 Oct 2013 22:08:48 -0400 |
|||
Build! (You may need to install various dependencies, which it will yell at you if you don't have.) |
|||
mount /dev/vg0/root /mnt |
|||
mount /dev/sda1 /mnt/boot |
|||
chroot /mnt |
|||
apt-get install lvm2 |
|||
debuild -k'''YOURKEYID''' |
|||
You should see an update-initramfs update. Reboot. |
|||
Yay, it built now let's upload it to the repo. The build process which create a PACKAGE.changes file in the parent directory (replace PACKAGE with the actual package name). |
|||
Change Ubuntu mirror to mirror.csclub.uwaterloo.ca, and add debian.csclub.uwaterloo.ca via /etc/apt/sources.list.d. apt-key add the key from mirror (TODO: make this easier to find) |
|||
Copy the dupload file from corn-syrup and dupload: |
|||
Install a bunch of packages from inapt list. Transfer nsswitch, ldap.conf, nslcd.conf, SSL certs, restore SSH key, KRB config, install mod-pam-csc, PAM config (/etc/pam.d/common-account). |
|||
mv /etc/dupload /etc/dupload.bak |
|||
Mount /users, /music and /scratch. |
|||
scp corn-syrup:/etc/dupload /etc/dupload |
|||
/etc/default/nfs-common: yes,yes,yes; start gssd |
|||
dupload libpam-csc_1.10jessie0_amd64.changes |
|||
Finally, log into mirror and type "sudo /srv/debian/bin/rrr-incoming". This is supposed to happen once every few minutes however it is always faster to run it manually. |
|||
On Ubuntu precise or later: alter LDAP schema in nslcd.conf |
|||
And you're done. For CEO, see https://git.csclub.uwaterloo.ca/public/pyceo/src/branch/master/PACKAGING.md |
|||
# use the uniqueMember attribute for group membership |
|||
map group member uniqueMember |
Latest revision as of 16:57, 11 June 2024
Firmware Updates
Vendors such as Dell provide firmware updates that should be applied before putting new machines into service. Even if the machine's warranty has expired, security updates are still made available.
It is recommended to use the following sequence when updating firmware on the Dell PowerEdge servers ([1]):
- iDRAC
- Lifecycle Controller
- BIOS
- Diagnostics
- OS Driver Pack
- RAID
- NIC
- PSU
- CPLD
- Other update
For consumer grade hardware, go to the motherboard vendor's website and find the way to upgrade the firmware.
Booting
- Put the TFTP image in place (if dist-arch pair installed before, you may skip this).
e.g. extract http://mirror.csclub.uwaterloo.ca/ubuntu/dists/oneiric/main/installer-amd64/current/images/netboot/netboot.tar.gz to caffeine:/srv/tftp/oneiric-amd64
- Force network boot in the BIOS. This may be called "Legacy LAN" or other such cryptic things. If this doesn't work, boot from CD or USB instead.
It is preferred to use the "alternate" Ubuntu installer image, based on debian-installer, instead of the Ubiquity installer. This installer supports software RAID and LVM out of the box, and will generally make your life easier. If installing Debian, this is the usual installer, so don't sweat it.
- Most of our newer servers (e.g. PowerEdge R815) need non-free firmware in order to boot. This means that if you are using a new netboot image, it is highly recommended to include the entire non-free firmware bundle in the boot image. See [2] for more information.
- For office terminals, create a boot USB (via dd, for example) and boot from USB.
Installing
debian-installer
At least in expert mode, you can choose a custom mirror (top of the countries list) and give the path for mirror directly. This will make installation super-fast compared to installing from anywhere else.
Please install to LVM volumes, as this is our standard configuration on all machines where possible. It allows more flexible partitioning across available volumes. Since GRUB 2, even /boot may be on LVM; this is the preferred configuration for simplicity, except when legacy partitioning setups make this inconvenient.
You may enable unattended upgrades, but do not enable Canonical's remote management service or any such nonsense. This is mostly a straightforward Debian/Ubuntu install.
After Installing
Add the machine's name to ~git/public/hosts.git, and run the ansible playbook (https://git.uwaterloo.ca/csc/playbooks/blob/master/update-hosts.yml) to distribute the updated hosts file to all machines.
apt
Delete/clear the file /etc/apt/sources.list and paste something like the following into /etc/apt/sources.list.d/debian.sources (replace "bookworm" by the the current Debian stable codename):
Types: deb URIs: http://mirror.csclub.uwaterloo.ca/debian Suites: bookworm bookworm-updates bookworm-backports Components: main contrib non-free non-free-firmware Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg Types: deb URIs: http://mirror.csclub.uwaterloo.ca/debian-security Suites: bookworm-security Components: main contrib non-free non-free-firmware Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
Install the CSC archive signing key:
wget -O /etc/apt/keyrings/csclub.gpg http://debian.csclub.uwaterloo.ca/csclub.gpg
Paste the following into /etc/apt/sources.list.d/csclub.sources (or copy from another host):
Types: deb URIs: http://debian.csclub.uwaterloo.ca Suites: bookworm Components: main Signed-By: /etc/apt/keyrings/csclub.gpg
In order to make Debian use packages in our repository by default, set our repository to the highest priority. Create /etc/apt/preferences.d/99-csclub
:
Package: *
Pin: origin debian.csclub.uwaterloo.ca
Pin-Priority: 1001
You should now run apt-get update to reflect these changes.
For unattended upgrades in the future, install the unattended-upgrades package and copy /etc/apt/apt.conf from another host.
Network
Note that debian 11 will use NetworkManager or /etc/interfaces
by default if you install a desktop environment, which doesn't seem to do DHCPv6 nicely. For simplicity and consistency across machines, we will use systemd-networkd
. First stop and disable NetworkManager:
systemctl disable --now NetworkManager.service networking.service
apt autoremove NetworkManager
Then, create a network configuration file at /etc/systemd/network/10-wired.network
:
[Match]
# Check the interface name using `ip a`
Name=enp3s0
[Network]
# DHCP for IPv4 should work just fine
DHCP=ipv4
# IPv6 doesn't seem to work properly. Manually set them here
Address=ALLOCATED_IPv6_ADDRESS
Gateway=IPv6_GATEWAY
Then start and enable systemd-networkd.service
. Also remember to specify the campus DNS at /etc/resolve.conf
. You can copy it from another CSC machine.
Kerberos keys
If this is a reinstall of an existing host, copy back the SSH host keys and /etc/krb5.keytab from its former incarnation. Otherwise, create a new Kerberos principal and copy the keytab over, as follows (run from the host in question):
kadmin -p sysadmin/admin # or any other admin principal; the password for this one is the usual root password
addprinc -randkey host/[hostname].csclub.uwaterloo.ca
ktadd host/[hostname].csclub.uwaterloo.ca
This will generate a new principal (you can skip this step if one already exists) and add it to the local Kerberos keytab.
Configuration
General
Install packages that we will need:
apt install krb5-user nfs-common nslcd sudo-ldap
# This package are automatically installed already, but we need to install our version so that NFS can connect to our crappy NetApp server
apt install --reinstall libk5crypto3
The following config files are needed to work in the CSC environment (examples given below for an office terminal; perhaps refer to another host if preferred). /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.
passwd: files systemd ldap
group: files systemd ldap
shadow: files ldap
gshadow: files ldap
sudoers: files ldap
hosts: files dns
networks: files
protocols: db files
services: db files
ethers: db files
rpc: db files
/etc/ldap/ldap.conf
#
# LDAP Defaults
#
# See ldap.conf(5) for details
# This file should be world readable but not world writable.
BASE dc=csclub, dc=uwaterloo, dc=ca
URI ldaps://ldap1.csclub.uwaterloo.ca ldaps://ldap2.csclub.uwaterloo.ca
SIZELIMIT 0
TLS_CACERT /etc/ssl/certs/ca-certificates.crt
TLS_CACERTFILE /etc/ssl/certs/ca-certificates.crt
SUDOERS_BASE ou=SUDOers,dc=csclub,dc=uwaterloo,dc=ca
Also make /etc/sudo-ldap.conf a symlink to the above. On debian, install sudo-ldap package too. /etc/nslcd.conf
# /etc/nslcd.conf
# nslcd configuration file. See nslcd.conf(5)
# for details.
# The user and group nslcd should run as.
uid nslcd
gid nslcd
# The location at which the LDAP server(s) should be reachable.
uri ldaps://ldap1.csclub.uwaterloo.ca
uri ldaps://ldap2.csclub.uwaterloo.ca
# The search base that will be used for all queries.
base dc=csclub, dc=uwaterloo, dc=ca
# The LDAP protocol version to use.
#ldap_version 3
# The DN to bind with for normal lookups.
#binddn cn=annonymous,dc=example,dc=net
#bindpw secret
# The DN used for password modifications by root.
#rootpwmoddn cn=admin,dc=example,dc=com
# SSL options
#ssl off
tls_reqcert demand
tls_cacertfile /etc/ssl/certs/ca-certificates.crt
# The search scope.
#scope sub
map group member uniqueMember
/etc/krb5.conf
[libdefaults]
default_realm = CSCLUB.UWATERLOO.CA
forwardable = true
proxiable = true
dns_lookup_kdc = false
dns_lookup_realm = false
allow_weak_crypto = true
[realms]
CSCLUB.UWATERLOO.CA = {
kdc = kdc1.csclub.uwaterloo.ca
kdc = kdc2.csclub.uwaterloo.ca
admin_server = kadmin.csclub.uwaterloo.ca
}
(rest omitted for brevity, see any CSC machine)
Notably, allow_weak_crypto is currently needed to mount /users (/music and /scratch is sec=sys and thus will always mount, even when krb5 is down and/or broken). Otherwise, you will get a mysterious "permission denied" error (even though the server claims to have authenticated the mount successfully).
Furthermore, the lines dns_lookup_kdc and dns_lookup_realm have been added - they are needed to stop the KDC from throwing its arms in the air and giving up if IST's DNS servers ever explode - an event that has happened in the recent past far more often than I'd like it to.
Change all lines in /etc/pam.d/common-* to have minimum_uid=10000 so that Kerberos won't interfere with local users. Note that pam configs are notably different on syscom-only hosts. Look at an existing syscom-only host to see the difference.
Alter /etc/default/nfs-common
# Alter these lines:
NEED_STATD=1
NEED_GSSD=1
# -l for gssd is to allow legacy crypto suites
RPCGSSDOPTS="-v -l"
to enable statd, and more importantly gssd (needed for Kerberos NFS mounts). Start rpc-statd.service
and rpc-gssd.service
manually for now.
Add /users, /music and /scratch to /etc/fstab (as appropriate for the machine's role), make their mount points and mount them. Note that /scratch are sec=sys whereas /music and /users is sec=krb5p (with exceptions granted on a case-by-case basis for servers only, office terminals are always sec=krb5p for security reasons).
To allow single sign-on as root (primarily useful for pushing files to all machines simultaneously), put the following in /root/.k5login:
sysadmin/admin@CSCLUB.UWATERLOO.CA
Also copy the following files from another CSC host:
- /etc/ssh/ssh_config and /etc/ssh/sshd_config (for single sign-on)
- /etc/ssh/ssh_known_hosts (to remove hostkey warnings within our network)
- /etc/hosts (for host tab completion and emergency name resolution)
- /etc/resolv.conf (to use IST's nameservers and search csclub/uwaterloo domains. Only required if you are not using /etc/network/interfaces to configure DNS)
Audio
On an office terminal, copy /etc/pulse/default.pa from another office terminal.
If this is to be the machine that actually plays audio (currently nullsleep), the setup is slightly more complicated. You'll need to set up MPD and PipeWire to receive connections, and store the PulseAudio cookie in ~audio, with appropriate permissions so that only the audio group can access it. If this is a new audio machine, you'll also need to change default.pa on all office terminals to point to it.
Password
Change the root password to the specified password in the usual place under the termcom user. If it's an office terminal, change the local user's password to the one specified in the usual place.
Prevent suspend and hibernation (Office Terminal)
Set AllowSuspend
, AllowHibernation
, AllowSuspendThenHibernate
and AllowHybridSleep
all to no
in /etc/systemd/sleep.conf
, and reboot.
Records
You probably already created the host in the University IPAM system beforehand. If not, please do so.
Please also add the host to the Machine List here on the Wiki.
Munin (System Monitoring)
If the new machine is not a container, you probably want to have it participate in the Munin cluster. Run apt-get install munin-node to install the monitoring client, then edit the file /etc/munin/munin-node.conf. Look for a line that says allow ^127\.0\.0\.1$ and add the following on a new line immediately below it: allow ^129\.97\.134\.51$ (this is the IP address for munin.csclub). Save the file, then /etc/init.d/munin-node restart and update-rc.d munin-node defaults.
Then, ssh into munin.csclub and edit the file /etc/munin/munin.conf and add the following lines to the end:
[NEW-MACHINE-NAME.csclub]
addr 129.97.134.###
use_node_name yes
Prometheus (System Monitoring)
We are currently using Prometheus to monitor our systems. On the new machine, install prometheus-node-exporter and stunnel.
Change /etc/default/prometheus-node-exporter to this:
ARGS="--web.listen-address=localhost:9101"
and start prometheus-node-exporter.service.
Then set up stunnel. Create /etc/stunnel/prometheus-node-exporter.conf with this content:
setuid = stunnel4 setgid = stunnel4 pid = /var/run/stunnel4/exporter.pid debug = 7 [prometheus-node-exporter] accept = 0.0.0.0:9100 connect = 127.0.0.1:9101 CAfile = /etc/stunnel/tls/server.crt cert = /etc/stunnel/tls/node.crt key = /etc/stunnel/tls/node.key verifyPeer = yes
Copy /etc/stunnel/{node.crt, node.key, server.crt} from prometheus:/opt/prometheus/tls or the same location on other machines.
Finally, start stunnel4.service.
If it's a new machine, you'll also need to add it to the list of monitoring at prometheus:/opt/prometheus/prometheus.yml. Add it under a suitable label (or create a new label) in 'node_exporter' job.
New Distribution
If you're adding a new distribution, there a couple of steps you'll need to take in updating the CSClub Debian repository on sodium-benzoate/mirror.
The steps to add a new Debian release (in the examples, jessie) is as follows, modify as necessary:
Step 0: Create a GPG key
Use "gpg --gen-key" or something like that. Skip this if you already have one.
Step 1: Add to Uploaders
The /srv/debian/conf/uploaders file on mirror contains the list of people who can upload. Add your GPG key id to this file. Use "gpg --list-secret-keys" to find out the key ID. You also need to import your key into the mirror's gpg homedir as follows:
gpg --export $KEYID | sudo env GNUPGHOME=/srv/debian/gpg gpg --import
You only need to do this step once.
Step 2: Add Distro
Add a new section to /srv/debian/conf/distributions:
Origin: CSC Label: Debian Codename: jessie Architectures: alpha amd64 i386 mips mipsel sparc powerpc armel source Components: main contrib non-free Uploaders: uploaders Update: dell chrome SignWith: yes Log: jessie.log --changes notifier
And update the Allow line in /srv/debian/conf/incoming:
Allow: jessie>jessie oldstable>squeeze stable>wheezy lucid>lucid maverick>maverick oneiric>oneiric precise>precise quantal>quantal
Step 3: Update from Sources
Run:
sudo env GNUPGHOME=/srv/debian/gpg /srv/debian/bin/rrr-update
If all went well you should see the new distribution listed at http://debian.csclub.uwaterloo.ca/dists/
Step 4: CSC Packages
Now that we've got our new distribution set up we need to generate our packages and have them uploaded. Namely, ceo and libpam-csc. For libpam-csc:
Get the package:
git clone https://git.csclub.uwaterloo.ca/public/libpam-csc.git cd libpam-csc
Update change log:
EMAIL=[you]@csclub.uwaterloo.ca NAME="Your Name" dch -i
Update as necessary, i.e:
libpam-csc (1.10jessie0) jessie; urgency=low * Packaging for jessie. -- Your Name <[you]@csclub.uwaterloo.ca> Thu, 10 Oct 2013 22:08:48 -0400
Build! (You may need to install various dependencies, which it will yell at you if you don't have.)
debuild -kYOURKEYID
Yay, it built now let's upload it to the repo. The build process which create a PACKAGE.changes file in the parent directory (replace PACKAGE with the actual package name).
Copy the dupload file from corn-syrup and dupload:
mv /etc/dupload /etc/dupload.bak scp corn-syrup:/etc/dupload /etc/dupload dupload libpam-csc_1.10jessie0_amd64.changes
Finally, log into mirror and type "sudo /srv/debian/bin/rrr-incoming". This is supposed to happen once every few minutes however it is always faster to run it manually.
And you're done. For CEO, see https://git.csclub.uwaterloo.ca/public/pyceo/src/branch/master/PACKAGING.md