Sysadmin Guide - Revision history

Merenber at 00:58, 5 June 2022

2022-06-05T00:58:23Z

← Older revision		Revision as of 19:58, 4 June 2022
Line 77:		Line 77:
	</li>		</li>
	</ul>		</ul>

			=== Membership lifecycle ===
			We should (but often don't) make sure that we clean up resources created by members who have expired. Otherwise, these just take up space. For cloud resources, we have a Python class called CloudResourceManager in the [https://git.csclub.uwaterloo.ca/public/pyceo/ pyceo] repo which deletes resources for expired members and sends a warning email one week in advance (the implementation is pretty sketchy and could be improved). If you decide to host some service where members can create their own resources, please update the pyceo code as necessary so that those resources are deleted when the owners' memberships expire.

R389li: Members don't get top-level UW subdomains

2022-06-05T00:50:44Z

Members don't get top-level UW subdomains

← Older revision		Revision as of 19:50, 4 June 2022
Line 58:		Line 58:
	Make sure to read [[Web Hosting]] first.		Make sure to read [[Web Hosting]] first.

	If a ~~member or~~ club requests a uwaterloo.ca subdomain, first make sure that their website is being hosted on our servers. Then, forward the email to hostmaster (at) uwaterloo.ca, and ask them to make the domain a CNAME for caffeine.csclub.uwaterloo.ca. You will also need to add a VirtualHost entry in /etc/apache2 on caffeine, redirecting requests to /users/club_name/www.		If a club requests a uwaterloo.ca subdomain, first make sure that their website is being hosted on our servers. Then, forward the email to hostmaster (at) uwaterloo.ca, and ask them to make the domain a CNAME for caffeine.csclub.uwaterloo.ca. You will also need to add a VirtualHost entry in /etc/apache2 on caffeine, redirecting requests to /users/club_name/www.

	Make sure to create a new Let's Encrypt certificate for the domain.		Make sure to create a new Let's Encrypt certificate for the domain.

			Members do not get their own top-level uwaterloo.ca subdomains, they can instead request a <WatIAM>.members.csclub.uwaterloo.ca subdomain.

	=== Mailing list subscriptions ===		=== Mailing list subscriptions ===

Merenber: /* Post-Outage */

2021-07-18T05:22:24Z

Post-Outage

← Older revision		Revision as of 00:22, 18 July 2021
Line 37:		Line 37:
	<ul>		<ul>
	<li>		<li>
	Log back into each MC machine and make sure that <code>/users</code> was mounted correctly. If not, check /etc/network/interfaces to get the name of the VLAN device, and use <code>ip addr</code> to see if the interface is up. If it is not up, try to use ifup; if that doesn't work, manually bring up the device and assign it the appropriate IP addresses using iproute2.		Log back into each MC machine and make sure that <code>/users</code> was mounted correctly. If not, check /etc/network/interfaces to get the name of the VLAN device, and use <code>ip addr</code> to see if the interface is up. If it is not up, try to use ifup; if that doesn't work, manually bring up the device and assign it the appropriate IP addresses using iproute2:
			<pre>
			# check /etc/network/interfaces for the interface name and IP
			ip link add name ens3.530 link ens3 type vlan id 530
			ip addr add dev ens3.530 172.19.168.49/27
			ip addr add dev ens3.530 fd74:6b6a:8eca:4903:c5c::49/64
			ip link set dev ens3.530 up
			</pre>
	</li>		</li>
	</ul>		</ul>

Merenber: Update mailman link

2021-07-17T04:49:43Z

Update mailman link

← Older revision		Revision as of 23:49, 16 July 2021
Line 10:		Line 10:
	<ul>		<ul>
	<li>		<li>
	Send an email to csc-general announcing the outage (example [https://mailman.csclub.uwaterloo.ca/~~pipermail~~/csc-general/2016-November/000694.html here])		Send an email to csc-general announcing the outage (example [https://mailman.csclub.uwaterloo.ca/private/csc-general/2016-November/000694.html here])
	</li>		</li>
	<li>		<li>
Line 33:		Line 33:
	</li>		</li>
	</ul>		</ul>

	==== Post-Outage ====		==== Post-Outage ====
	<ul>		<ul>

Merenber: /* uwaterloo.ca subdomains */

2021-03-26T21:22:58Z

uwaterloo.ca subdomains

← Older revision		Revision as of 16:22, 26 March 2021
Line 51:		Line 51:

	If a member or club requests a uwaterloo.ca subdomain, first make sure that their website is being hosted on our servers. Then, forward the email to hostmaster (at) uwaterloo.ca, and ask them to make the domain a CNAME for caffeine.csclub.uwaterloo.ca. You will also need to add a VirtualHost entry in /etc/apache2 on caffeine, redirecting requests to /users/club_name/www.		If a member or club requests a uwaterloo.ca subdomain, first make sure that their website is being hosted on our servers. Then, forward the email to hostmaster (at) uwaterloo.ca, and ask them to make the domain a CNAME for caffeine.csclub.uwaterloo.ca. You will also need to add a VirtualHost entry in /etc/apache2 on caffeine, redirecting requests to /users/club_name/www.

			Make sure to create a new Let's Encrypt certificate for the domain.

	=== Mailing list subscriptions ===		=== Mailing list subscriptions ===

Merenber: /* Let's Encrypt certificates */

2021-03-22T05:47:54Z

Let's Encrypt certificates

← Older revision		Revision as of 00:47, 22 March 2021
Line 44:		Line 44:

	We handle LE certs for members and clubs who host their websites on our servers. The certs should be renewed automatically; if they do not, then something is very wrong. There are plans underway to migrate from <code>certbot</code> to <code>dehydrate</code> since the apt version of certbot appears to be broken.		We handle LE certs for members and clubs who host their websites on our servers. The certs should be renewed automatically; if they do not, then something is very wrong. There are plans underway to migrate from <code>certbot</code> to <code>dehydrate</code> since the apt version of certbot appears to be broken.

			If you get an email LE warning you that a cert is about to expire, login to caffeine and check /var/log/letsencrypt/letsencrypt.log. There should usually be some clue as to what went wrong. Often, a club or member decides that they no longer want to host their website on our servers, in which case the cert can safely be removed via <code>certbot delete --cert-name CERT_NAME</code>. Make sure to also delete the corresponding Apache config files. Sometimes a subset of the domains for a member have become invalid, in which case they must be removed from the cert. One way to do this is via <code>certbot certonly --webroot -d domain1.com -d domain2.com ...</code>. Only list the domains which are still valid with the <code>-d</code> flags; omitted domains will be removed. Make sure to update the corresponding Apache config files.

	=== uwaterloo.ca subdomains ===		=== uwaterloo.ca subdomains ===

Merenber at 03:38, 22 March 2021

2021-03-22T03:38:40Z

← Older revision		Revision as of 22:38, 21 March 2021
Line 48:		Line 48:
	Make sure to read [[Web Hosting]] first.		Make sure to read [[Web Hosting]] first.

	If a member or club requests a uwaterloo.ca subdomain, first make sure that their website is being hosted on our servers.. Then, forward the email to hostmaster (at) uwaterloo.ca, and make ~~their~~ domain a CNAME for caffeine.csclub.uwaterloo.ca.		If a member or club requests a uwaterloo.ca subdomain, first make sure that their website is being hosted on our servers. Then, forward the email to hostmaster (at) uwaterloo.ca, and ask them to make the domain a CNAME for caffeine.csclub.uwaterloo.ca. You will also need to add a VirtualHost entry in /etc/apache2 on caffeine, redirecting requests to /users/club_name/www.

			=== Mailing list subscriptions ===
			At the very least, you need to be subscribed to the [https://mailman.csclub.uwaterloo.ca/listinfo/syscom syscom] and [https://mailman.csclub.uwaterloo.ca/listinfo/exec exec] mailing lists. You may also wish to subscribe to the following:
			<ul>
			<li>
			[https://mailman.csclub.uwaterloo.ca/listinfo/git git]: Get alerts on some git commits (mainly Apache configs)
			</li>
			<li>
			[https://mailman.csclub.uwaterloo.ca/listinfo/packages packages]: Get alerts of when packages are added to our Debian repository
			</li>
			<li>
			[https://mailman.csclub.uwaterloo.ca/listinfo/ceo ceo]: Get alerts on new user accounts
			</li>
			</ul>

Merenber: /* Power Outages */

2021-03-20T22:44:47Z

Power Outages

← Older revision		Revision as of 17:44, 20 March 2021
Line 7:		Line 7:
	=== Power Outages ===		=== Power Outages ===
	Occasionally MC will undergo planned power outages. These usually last from the morning until the evening. a2brenna or someone from IST will hopefully give us a notice in advance. When this happens, you should:		Occasionally MC will undergo planned power outages. These usually last from the morning until the evening. a2brenna or someone from IST will hopefully give us a notice in advance. When this happens, you should:
			==== Pre-Outage ====
	<ul>		<ul>
	<li>		<li>
Line 30:		Line 31:
	<br>		<br>
	TODO: Consider using keepalived to automate this process.		TODO: Consider using keepalived to automate this process.
			</li>
			</ul>
			==== Post-Outage ====
			<ul>
			<li>
			Log back into each MC machine and make sure that <code>/users</code> was mounted correctly. If not, check /etc/network/interfaces to get the name of the VLAN device, and use <code>ip addr</code> to see if the interface is up. If it is not up, try to use ifup; if that doesn't work, manually bring up the device and assign it the appropriate IP addresses using iproute2.
	</li>		</li>
	</ul>		</ul>

Merenber: /* Power Outages */

2021-03-20T05:47:00Z

Power Outages

← Older revision		Revision as of 00:47, 20 March 2021
Line 15:		Line 15:
	</li>		</li>
	<li>		<li>
	~~Update~~ the ~~topic~~ in the #csc IRC channel ~~announcing~~ the outage		Announce the outage in the #csc IRC channel and update the channel topic to show outage information
	</li>		</li>
	<li>		<li>
	Schedule the shutdown the night before the outage using the <code>shutdown</code> command on all of our MC machines, e.g.		Schedule the shutdown the night before the outage using the <code>shutdown</code> command on all of our MC machines, e.g.
	<pre>		<pre>
	sudo shutdown 08:00 "CSC systems will be unavailable for a power outage ~~8am~~ -> ~~12pm~~. This machine will shutdown at 8:00AM EDT."		sudo shutdown 06:00 "CSC systems will be unavailable for a power outage 7am -> 5pm. This machine will shutdown at 6:00AM EDT."
	</pre>		</pre>
			</li>
			<li>
			If the real machines hosting the web server (phosphoric-acid) and mirror (potassium-benzoate) cannot be kept up during the outage,
			set up a backup web server in an LXC container on a machine which is not located in MC (currently there is a container named dr-website
			on biloba). '''After''' the MC machines have shutdown, assign the IP addresses of csclub.uwaterloo.ca and mirror.csclub.uwaterloo.ca to
			the backup container.
			<br>
			TODO: Consider using keepalived to automate this process.
	</li>		</li>
	</ul>		</ul>

Merenber: /* Power Outages */

2021-03-19T21:18:56Z

Power Outages

← Older revision		Revision as of 16:18, 19 March 2021
Line 6:		Line 6:

	=== Power Outages ===		=== Power Outages ===
	Occasionally MC will undergo planned power outages. These usually last from the morning until the evening. a2brenna or someone from IST will hopefully give us a notice in advance. When this happens, you should ~~send an email to csc-general announcing the outage (example [https~~://mailman.csclub.uwaterloo.ca/pipermail/csc-general/2016-November/000694.html here]). You should plan the shutdown the night before the outage using the <code>shutdown</code> command on all of our MC machines, e.g.		Occasionally MC will undergo planned power outages. These usually last from the morning until the evening. a2brenna or someone from IST will hopefully give us a notice in advance. When this happens, you should:
			<ul>
			<li>
			Send an email to csc-general announcing the outage (example [https://mailman.csclub.uwaterloo.ca/pipermail/csc-general/2016-November/000694.html here])
			</li>
			<li>
			Create an announcement on our main website announcing the outage
			</li>
			<li>
			Update the topic in the #csc IRC channel announcing the outage
			</li>
			<li>
			Schedule the shutdown the night before the outage using the <code>shutdown</code> command on all of our MC machines, e.g.
	<pre>		<pre>
	sudo shutdown 08:00 "CSC systems will be unavailable for a power outage 8am -> 12pm. This machine will shutdown at 8:00AM EDT."		sudo shutdown 08:00 "CSC systems will be unavailable for a power outage 8am -> 12pm. This machine will shutdown at 8:00AM EDT."
	</pre>		</pre>
			</li>
			</ul>

	=== Let's Encrypt certificates ===		=== Let's Encrypt certificates ===