Intro to Authentication

CSC's user directory and authentication system consists of 3 major parts:

LDAP

directory service. stores all public user information (your name, program, WatIAM, UNIX groups, things like that)

Kerberos (krb5)

authentication service. stores passwords, provide authentication to user logins/inter-server integrations (ceo and NFS uses krb5)

pyceo

CSC's home-grown frontend for interacting with user directory and passwords

Keycloak

SSO support for web applications, allows you to use WatIAM/CSC OTP to log into web based services

Basically, this is a UNIX-based Active Directory system. Why not using Microsoft's offering? Well, we love UNIX, just look at the office, there's no Windows there.

Client Side

LDAP

Read more on: LDAP

LDAP itself is just a protocol. In practice, it has 2 parts: a server that stores and serves user information, and a client that queries these information for various applications. All servers in CSC, no matter general-use or syscom-only, are hooked up to LDAP so that they can automatically sync users and groups so that we don't need to manually create/delete account for every member. It's also responsible for tracking which terms do a member have a valid membership of, and that's how we implement general-use server's access control based on membership.

In essence, it's just a database that keeps track of a bunch of unique entities (called Distinguished Name, or DN in LDAP world) and each entity can have a bunch of attributes associated with them. Here's the LDAP entry for our club mascot, C.T. Dalek:

dn: uid=ctdalek,ou=People,dc=csclub,dc=uwaterloo,dc=ca
uid: ctdalek
homeDirectory: /users/ctdalek
cn: Calum T. Dalek
gecos: Calum T. Dalek,MC 3036,,,0
uidNumber: 20000
description: Prototypical Member Account
gidNumber: 20000
objectClass: account
objectClass: member
objectClass: posixAccount
objectClass: shadowAccount
objectClass: top
objectClass: inetLocalMailRecipient
loginShell: /bin/false
userPassword: {SASL}ctdalek@CSCLUB.UWATERLOO.CA
term: f2016
term: f2017
term: f2018
givenName: Calum
sn: Dalek
mailLocalAddress: ctdalek@csclub.uwaterloo.ca
program: Alumni

It's entirely possible to do all of these in a NoSQL database, but of course LDAP is a standard, and almost all SSO-capable system can speak LDAP, so that's why we use it.

Kerberos

Read more on: Kerberos

Kerberos is a very complicated protocol and deserve its own lecture, but the crash course is that you do an initial authentication with Kerberos (usually with username/password), and it grants you a "ticket" so that you can use such ticket to access other services without going through the central authentication node again.

One use case you might find useful is if you want to hop between CSC servers (for example, you are outside the campus network but you want to access a termcom machine that is campus-network only), you can use kinit to get yourself a ticket, and you won't need to input password any more because your SSH client will send the ticket first and will be accepted as your proof of identity.

Other than user authentication, we also use Kerberos for inter-machine authentication. All CSC machines have a host/$HOSTNAME.csclub.uwaterloo.ca ticket installed on them, and they use this to authenticate and mount the /users NFS file share, so that you can use one single home folder on all of the CSC machines.

pyceo

Read more on: CEO

Since editing LDAP databases is tediously and dangerous (you might bring down the whole fleet!), account management is handled by pyceo. Things like adding/renewing members and resetting passwords are in essence just modifying LDAP/Kerberos databases, but pyceo does autofilling and sanity check on day-to-day operations.

Note that pyceo's scope has expanded quite a bit after the introduction of CSC cloud, now it's more than just a frontend for LDAP/Krb5, but more of a frontend for most of the member services.

Keycloak

Read more on: Keycloak

For security reasons, the university requires us to implement 2FA (two factor authentication) in some way for all of our services. For SSH we did it via pam_duo, and for web we use Keycloak. Keycloak behaves like an adapter to provide the "web" way of authentication (i.e. OpenID Connect) using data from the "UNIX" way of authentication (LDAP, Krb5), and provide additional security features like OTP (one time password).

A special thing we do with Keycloak is that we use it as an identity broker for WatIAM, which allow you to login via the university's WatIAM login portal and then use that session to log into CSC services without another password/OTP prompt.

Server implementations

We have two nspawn containers, auth1 and auth2, on two physical locations (xylitol@MC and cobalamin@Science Machine Room). Both of them run slapd (the free LDAP server implementation) for LDAP and krb5-kdc + krb5-admin-server for Kerberos.

If you have read the LDAP spec or slapd/OpenLDAP documentation closely, you will know that there's an authentication mechanism in LDAP as well. But the documentation will also tell you it's generally a good idea to separate the public LDAP server and the secret-keeping ones, and that's exactly what we did: we store (hashed) passwords in Kerberos. But a lot of services (example being Nextcloud) uses the LDAP authentication; thus, when that happens, we proxy the authentication to Kerberos and just return the result. This is done via configuring slapd to use SASL auth mechanism, and run saslauthd on auth1/2 with backend set to kerberos5.

auth1 is the master server, and it will either actively send changes to auth2 (slapd) or auth2 will periodically pull changes from auth1 (krb5). There's also a cron job to periodically dump the complete database onto NFS.