Web Hosting - Revision history

K57chan: /* Fighting AI scrappers */

2025-06-30T17:02:08Z

Fighting AI scrappers

← Older revision		Revision as of 12:02, 30 June 2025
Line 320:		Line 320:
	See <code>caffeine:/etc/anubis</code> for details.		See <code>caffeine:/etc/anubis</code> for details.

	Anubis might break services running on the CSC servers (e.g. Renovate Bot). To fix this, simply add the service into the list of Anubis' exceptions (gitea.botPolicies.json). Afterwards, restart Anubis ~~and~~ Apache. ~~WARNING: This might break~~ ~~Git~~.		Anubis might break services running on the CSC servers (e.g. Renovate Bot). To fix this, simply add the service into the list of Anubis' exceptions (gitea.botPolicies.json). Afterwards, restart Anubis, Apache, and Gitea.

	=== Sample Apache config for website with both a custom domain and a UW subdomain ===		=== Sample Apache config for website with both a custom domain and a UW subdomain ===

2025-06-28T00:08:59Z

← Older revision		Revision as of 19:08, 27 June 2025
Line 320:		Line 320:
	See <code>caffeine:/etc/anubis</code> for details.		See <code>caffeine:/etc/anubis</code> for details.

	Anubis might break ~~certain bots~~ running on the CSC servers (e.g. Renovate Bot). To fix this, simply add the service into the list of Anubis' exceptions (gitea.botPolicies.json). This might break Git.		Anubis might break services running on the CSC servers (e.g. Renovate Bot). To fix this, simply add the service into the list of Anubis' exceptions (gitea.botPolicies.json). Afterwards, restart Anubis and Apache. WARNING: This might break Git.

	=== Sample Apache config for website with both a custom domain and a UW subdomain ===		=== Sample Apache config for website with both a custom domain and a UW subdomain ===

2025-06-28T00:02:55Z

Added documentation on fixing Anubis when it kills other bots.

← Older revision		Revision as of 19:02, 27 June 2025
Line 319:		Line 319:

	See <code>caffeine:/etc/anubis</code> for details.		See <code>caffeine:/etc/anubis</code> for details.

			Anubis might break certain bots running on the CSC servers (e.g. Renovate Bot). To fix this, simply add the service into the list of Anubis' exceptions (gitea.botPolicies.json). This might break Git.

	=== Sample Apache config for website with both a custom domain and a UW subdomain ===		=== Sample Apache config for website with both a custom domain and a UW subdomain ===

2025-04-28T22:49:20Z

add anubis: fighting ai scrappers

← Older revision		Revision as of 17:49, 28 April 2025
Line 310:		Line 310:
	* The script provides a 1-month grace period (corresponding to the grace period of pam-csc)		* The script provides a 1-month grace period (corresponding to the grace period of pam-csc)
	* The expired page returns HTTP status code of 503 (Service Unavailable)		* The expired page returns HTTP status code of 503 (Service Unavailable)

			=== Fighting AI scrappers ===
			We use [https://github.com/TecharoHQ/anubis Anubis] to block excess AI scrappers. This sits between Apache2 and the real application to insert a proof-of-work challenge for the browser to solve, before proceeding to provide the actual content.

			Since Anubis requires JavaScript, it is only selectively enabled on sites that a) is currently getting excessively scraped and b) need JavaScript already.

			Currently, only [[Git Hosting]] is protected.

			See <code>caffeine:/etc/anubis</code> for details.

	=== Sample Apache config for website with both a custom domain and a UW subdomain ===		=== Sample Apache config for website with both a custom domain and a UW subdomain ===

2022-10-31T14:25:28Z

Setting up a git repository

← Older revision		Revision as of 09:25, 31 October 2022
Line 92:		Line 92:
	#set -x		#set -x
	shopt -s dotglob		shopt -s dotglob
			# FOR CLUB REPS ONLY: set the following variable to e.g. /users/myclub/www
	DEPLOYMENT_DIR=~/www		DEPLOYMENT_DIR=~/www

2022-08-11T21:51:10Z

How do I make a website?

← Older revision		Revision as of 16:51, 11 August 2022
Line 31:		Line 31:
	* [https://www.getzola.org/ Zola] (accepts Markdown and Tera)		* [https://www.getzola.org/ Zola] (accepts Markdown and Tera)
	* [https://blog.getpelican.com/ Pelican] (accepts Markdown, reStructuredText and Jinja2)		* [https://blog.getpelican.com/ Pelican] (accepts Markdown, reStructuredText and Jinja2)

			[https://astro.build/ Astro] is an excellent static site builder which integrates with a wide variety of JS-based frameworks (including React, Vue, Svelte and Solid), but requires a bit more coding experience.

	These SSGs require some experience with React.js:		These SSGs require some experience with React.js:
Line 41:		Line 43:
	* [https://nuxtjs.org/ Nuxt.js]		* [https://nuxtjs.org/ Nuxt.js]
	* [https://vuepress.vuejs.org/ Vuepress]		* [https://vuepress.vuejs.org/ Vuepress]
			* [https://vitepress.vuejs.org/ Vitepress]
	* [https://gridsome.org/ Gridsome]		* [https://gridsome.org/ Gridsome]

2022-07-05T03:31:46Z

Disabling insecure or infringing sites

← Older revision		Revision as of 22:31, 4 July 2022
Line 290:		Line 290:
	</Directory>		</Directory>

	For WordPress sites specifically, ~~use~~ the ~~shared~~ ~~snippet~~:		For WordPress sites specifically, insert a snippet similar to the following into conf-enabled/disable-wordpress.conf:

	<Directory "/users/$BADCLUB/www">		<Directory "/users/$BADCLUB/www">
	Include snippets/disable-wordpress.conf		Include snippets/disable-wordpress.conf
	</Directory>		</Directory>

	If a user/club does not have their own file under sites-enabled, use conf-enabled/disable-wordpress.conf instead.

	=== Expired Websites ===		=== Expired Websites ===

2022-06-28T13:16:58Z

Setting up a git repository

← Older revision		Revision as of 08:16, 28 June 2022
Line 129:		Line 129:
	</pre>		</pre>
	</li>		</li>
	<li>For club reps:		<li>For club reps:<br>
	Use the full path of the repo in your club user's home directory, e.g.		Use the full path of the repo in your club user's home directory, e.g.
	<pre>		<pre>

2022-06-28T13:16:26Z

Setting up a git repository

← Older revision		Revision as of 08:16, 28 June 2022
Line 118:		Line 118:
	</li>		</li>
	<li>		<li>
	If you have not done so already, add your public SSH key to your ~/.ssh/authorized_keys file (on the CSC machines).		If you have not done so already, add your public SSH key to your ~/.ssh/authorized_keys file (on the CSC machines). See [https://git-scm.com/book/en/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key here] for a tutorial.
	</li>		</li>
	<li>		<li>

2022-06-28T13:15:19Z

Setting up a git repository

← Older revision		Revision as of 08:15, 28 June 2022
Line 112:		Line 112:
	</pre>		</pre>

	<b>For club reps</b>: ~~The~~ ~~<code>chgrp</code>~~ ~~and~~ ~~<code>chmod</code>~~ ~~commands~~ in ~~the script are~~ ~~necessary~~ to ~~ensure~~ ~~that~~ ~~other members~~ of your club ~~have~~ ~~write permissions to the files which you create.~~		<b>For club reps</b>: Make sure the www directory is group-writable. Switch to the Unix user of your club and run:
			<pre>
			chmod g+w ~/www
			</pre>
	</li>		</li>
	<li>		<li>