New CSC Machine

Draft of how to set up a new CSC machine

Booting

• Put the TFTP image in place (if dist-arch pair installed before, you may skip this).

e.g. extract http://mirror.csclub.uwaterloo.ca/ubuntu/dists/oneiric/main/installer-amd64/current/images/netboot/netboot.tar.gz to caffeine:/srv/tftp/oneiric-amd64

• Configure DHCP server; see caffeine:/etc/dhcp/dhcpd.conf. This allow caffeine to identify the host by its MAC address and give it the correct IP and boot image. e.g.

host bit-shifter {
  hardware ethernet 6C:F0:49:08:49:48;
  fixed-address 129.97.134.111;
  next-server 129.97.134.17;
  filename "precise-amd64/pxelinux.0";
}

• Force network boot in the BIOS. This may be called "Legacy LAN" or other such cryptic things. If this doesn't work, boot from CD or USB instead.

It is preferred to use the "alternate" Ubuntu installer image, based on debian-installer, instead of the Ubiquity installer. This installer supports software RAID and LVM out of the box, and will generally make your life easier. If installing Debian, this is the usual installer, so don't sweat it.

Installing

debian-installer

At least in expert mode, you can choose a custom mirror (top of the countries list) and give the path for mirror directly. This will make installation super-fast compared to installing from anywhere else.

Please install to LVM volumes, as this is our standard configuration on all machines where possible. It allows more flexible partitioning across available volumes. Since GRUB 2, even /boot may be on LVM; this is the preferred configuration for simplicity, except when legacy partitioning setups make this inconvenient.

You may enable unattended upgrades, but do not enable Canonical's remote management service or any such nonsense. This is mostly a straightforward Debian/Ubuntu install.

Ubiquity

Ubiquity is the Ubuntu GUI installer. For it to have lvm support, run:

apt-get install lvm2

If you still can't see the partitions (even if lvscan sees them, but no devices exist), run vgscan and vgchange -ay as root. Now the partitioner should be able to see them. We prefer to use LVM for partitions. Since GRUB 2, even /boot may be on LVM; this is the preferred configuration for simplicity, except when legacy partitioning setups make this inconvenient.

After installing with Ubiquity, you must also add LVM support to the newly installed system, and in particular its initramfs.

mount /dev/vg0/root /mnt
mount /dev/sda1 /mnt/boot
chroot /mnt
apt-get install lvm2

You should see an update-initramfs update. Reboot.

After Installing

apt

If you did not during installation, change all references in /etc/apt/sources.list to use mirrorinstead of the usual mirrors.

Also add support for the CSC packages. Add the following to /etc/apt/sources.list.d/csclub.list (or copy from another host):

deb http://debian.csclub.uwaterloo.ca/ <distribution> main contrib non-free
deb-src http://debian.csclub.uwaterloo.ca/ <distribution> main contrib non-free

You'll also need the CSC archive signing key (if curl is not installed, install it).

curl -s http://debian.csclub.uwaterloo.ca/csclub.asc | apt-key add -

You should now run apt-get update to reflect these changes.

Next, install inapt (it is in the CSC Debian archive). If it hasn't previously been built for the current platform, clone and build it (TODO: describe how to do this).

Clone ~git/public/packages.git, update it if necessary (notably updating nodes.ia to reflect the distribution and role of the machine), then run:

inapt *.ia

(Due to a bug, if a warning is thrown, this will segfault. Until fixed, just temporarily remove whatever packages it complains about from the list.)

Warning: this will take a long time due to the large number of packages being installed. Some of the below can be done once the relevant packages are installed, but while other packages are still being installed.

Keys

If this is a reinstall of an existing host, copy back the SSH host keys and /etc/krb5.keytab from its former incarnation. Otherwise, create a new Kerberos principal and copy the keytab over, as follows (run from the host in question):

kadmin -p sysadmin/admin   # or any other admin principal; the password for this one is the usual root password
addprinc -randkey host/[hostname].csclub.uwaterloo.ca
ktadd host/[hostname].csclub.uwaterloo.ca

This will generate a new principal (you can skip this step if one already exists) and add it to the local Kerberos keytab.

Also copy /etc/ssl/certs/GlobalSign_Intermediate_Root.pem from another host, as many of our services use a certificate issued by this CA.

Configuration

The following config files are needed to work in the CSC environment (examples given below for an office terminal; perhaps refer to another host if preferred).

/etc/nsswitch.conf

# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd:         files ldap
group:          files ldap
shadow:         files
sudoers:        files ldap

hosts:          files dns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis

/etc/ldap/ldap.conf

# $OpenLDAP: pkg/ldap/libraries/libldap/ldap.conf,v 1.9 2000/09/04 19:57:01 kurt Exp $
#
# LDAP Defaults
#

# See ldap.conf(5) for details
# This file should be world readable but not world writable.

BASE   dc=csclub, dc=uwaterloo, dc=ca
URI     ldaps://ldap1.csclub.uwaterloo.ca ldaps://ldap2.csclub.uwaterloo.ca

SIZELIMIT      0

TLS_CACERT      /etc/ssl/certs/GlobalSign_Intermediate_Root.pem
TLS_CACERTFILE /etc/ssl/certs/GlobalSign_Intermediate_Root.pem

SUDOERS_BASE    ou=SUDOers,dc=csclub,dc=uwaterloo,dc=ca

/etc/nslcd.conf

# /etc/nslcd.conf
# nslcd configuration file. See nslcd.conf(5)
# for details.

# The user and group nslcd should run as.
uid nslcd
gid nslcd

# The location at which the LDAP server(s) should be reachable.
uri ldap://ldap1.csclub.uwaterloo.ca
uri ldap://ldap2.csclub.uwaterloo.ca

# The search base that will be used for all queries.
base dc=csclub,dc=uwaterloo,dc=ca

# use the uniqueMember attribute for group membership
map group member uniqueMember

/etc/krb5.conf

[libdefaults]
        default_realm = CSCLUB.UWATERLOO.CA
        forwardable = true
        proxiable = true

        # remove this once NFS mounts without it
        allow_weak_crypto = true

[realms]
        CSCLUB.UWATERLOO.CA = {
                kdc = kdc1.csclub.uwaterloo.ca
                kdc = kdc2.csclub.uwaterloo.ca
                admin_server = kadmin.csclub.uwaterloo.ca
        }
(rest omitted for brevity)

Notably, allow_weak_crypto is currently needed to mount /users (though oddly, not /music or /scratch). Otherwise, you will get a mysterious "permission denied" error (even though the server claims to have authenticated the mount successfully).

/etc/pam.d/common-account

#
# /etc/pam.d/common-account - authorization settings common to all services
#

# here are the per-package modules (the "Primary" block)
account        [success=1 new_authtok_reqd=done default=ignore]        pam_unix.so 
# here's the fallback if no module succeeds
account        requisite                       pam_deny.so
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
account        required                        pam_permit.so
# and here are more per-package modules (the "Additional" block)
account        required                        pam_krb5.so minimum_uid=10000
# end of pam-auth-update config

# Make sure the user is up to date. System accounts and syscom are exempt.
account [success=2 default=ignore]     pam_succeed_if.so quiet uid < 10000
account [success=1 default=ignore]     pam_succeed_if.so quiet user ingroup syscom
account required        pam_csc.so

This file is notably different on syscom-only hosts. Look at an existing syscom-only host to see the difference.

Alter /etc/default/nfs-common to enable statd, and more importantly gssd (needed for Kerberos NFS mounts). Start both daemons manually for now.

Add /users, /music and /scratch to /etc/fstab (as appropriate for the machine's role), make their mount points and mount them.

/etc/ssh/sshd_config (for single sign-on)

/etc/ssh/ssh_known_hosts (to remove hostkey warnings within our network)

(TODO: DM configuration etc.)

New Distribution

If new distribution, create in conf/distributions AND conf/uploaders, and do rrr-update as well as rrr-incoming. (TODO: expand this to describe building and pushing core CSC packages)