Mirror - Revision history

Y266shen: /* Writing Mirror News & Banner */

2024-12-20T18:45:20Z

Writing Mirror News & Banner

← Older revision		Revision as of 13:45, 20 December 2024
Line 164:		Line 164:
	banner_text = "CSC Mirror will be down on Dec 22, 2024 from 7am to 4pm (EST/UTC-5). [More](/news/dec-22-2024-scheduled-downtime/)"		banner_text = "CSC Mirror will be down on Dec 22, 2024 from 7am to 4pm (EST/UTC-5). [More](/news/dec-22-2024-scheduled-downtime/)"
	</pre>		</pre>
	And once the ~~warning~~ is over just set ~~warning~~ to be false.		And once the incident is over just set banner to be false.

	=== Adding a new project ===		=== Adding a new project ===

Y266shen: /* Writing Mirror News & Warning */

2024-12-20T18:40:04Z

Writing Mirror News & Warning

← Older revision		Revision as of 13:40, 20 December 2024
Line 142:		Line 142:
	Everything in the <code>~mirror</code> is managed by git (so a monorepo containing all sub-projects like Merlin and mirror-index). To make changes, switch to the mirror user and commit with <code>--author "FirstName LastName <email@csc></code> to show who made the change. Then run <code>git push</code> to push the changes. The remote is using the HTTPS URL, so just enter your CSC credentials.		Everything in the <code>~mirror</code> is managed by git (so a monorepo containing all sub-projects like Merlin and mirror-index). To make changes, switch to the mirror user and commit with <code>--author "FirstName LastName <email@csc></code> to show who made the change. Then run <code>git push</code> to push the changes. The remote is using the HTTPS URL, so just enter your CSC credentials.

	=== Writing Mirror News & ~~Warning~~ ===		=== Writing Mirror News & Banner ===
	You can add news by putting a Markdown file into <code>~mirror/mirror-index/content/news</code>. A minimal post looks like this:		You can add news by putting a Markdown file into <code>~mirror/mirror-index/content/news</code>. A minimal post looks like this:
	<pre>		<pre>
Line 154:		Line 154:
	If you experienced any usability issues due to browser compatibility, please let us know on [syscom@csclub.uwaterloo.ca](syscom@csclub.uwaterloo.ca).		If you experienced any usability issues due to browser compatibility, please let us know on [syscom@csclub.uwaterloo.ca](syscom@csclub.uwaterloo.ca).
	</pre>		</pre>
	You can also put up a big ~~warning~~ on the front page. Just edit <code>~mirror/mirror-index/config.toml</code>:		You can also put up a big banner on the front page for notifying critical information. Just edit <code>~mirror/mirror-index/config.toml</code>:
	<pre>		<pre>
	[params]		[params]
	~~warning~~ = true		banner = true
			# Supported options: blue orange red
			banner_color = "orange"
			banner_title = "Scheduled Downtime"
	# You can write markdown here		# You can write markdown here
	~~warning_text~~ = "CSC Mirror will be down on Dec 22, 2024 from 7am to 4pm (EST/UTC-5). [More](/news/dec-22-2024-scheduled-downtime/)"		banner_text = "CSC Mirror will be down on Dec 22, 2024 from 7am to 4pm (EST/UTC-5). [More](/news/dec-22-2024-scheduled-downtime/)"
	</pre>		</pre>
	And once the warning is over just set warning to be false.		And once the warning is over just set warning to be false.

Y266shen: /* Mirror Administration */

2024-12-19T18:35:48Z

Mirror Administration

@@ Line 141: / Line 141: @@
 === Making changes ===
 Everything in the <code>~mirror</code> is managed by git (so a monorepo containing all sub-projects like Merlin and mirror-index). To make changes, switch to the mirror user and commit with <code>--author "FirstName LastName <email@csc></code> to show who made the change. Then run <code>git push</code> to push the changes. The remote is using the HTTPS URL, so just enter your CSC credentials.
 === Adding a new project ===

Ztseguin: /* Adding a new project */

2023-07-01T23:39:41Z

Adding a new project

← Older revision		Revision as of 18:39, 1 July 2023
Line 153:		Line 153:
	#*<code>ln -s .cscmirror0/$PROJECT_NAME $PROJECT_NAME</code> ('''NOTE''': The symlink must be relative to the <code>/mirror/root</code> directory. If it isn’t, the symlinks will not work when chrooted)		#*<code>ln -s .cscmirror0/$PROJECT_NAME $PROJECT_NAME</code> ('''NOTE''': The symlink must be relative to the <code>/mirror/root</code> directory. If it isn’t, the symlinks will not work when chrooted)
	# Repeat the above steps on mirror-phys. <code>sudo ssh mirror-dc</code> on potassium-benzoate ['''NOTE: This machine is currently unavailable]'''		# Repeat the above steps on mirror-phys. <code>sudo ssh mirror-dc</code> on potassium-benzoate ['''NOTE: This machine is currently unavailable]'''
	# Configure the project in merlin (<code>~mirror/merlin/merlin.py</code>)		# Configure the project in merlin (<code>~mirror/merlin/merlin-config.ini</code>)
	#* Select the appropriate sync script (typically <code>csc-sync-standard</code>) and supply the appropriate parameters		#* Select the appropriate sync script (typically <code>csc-sync-standard</code>) and supply the appropriate parameters
	# Restart merlin: <code>systemctl restart merlin</code>		# Restart merlin: <code>systemctl restart merlin-go</code>
	#* This will kick off the initial sync		#* This will kick off the initial sync
	#* Check <code>~mirror/merlin/~~logs~~/$PROJECT_NAME</code> for errors, <code>~mirror/merlin/~~logs~~/~~transfer~~.log</code> for transfer progress		#* Check <code>~mirror/merlin/log/$PROJECT_NAME</code> for errors, <code>~mirror/merlin/log-$PROTOCOL/$PROJECT_NAME-*.log</code> for transfer progress
	# Configure the project in zfssync.yml (<code>~mirror/merlin/zfssync.yml</code>)		# Configure the project in zfssync.yml (<code>~mirror/merlin/zfssync.yml</code>) ['''NOTE: The backup machine is currently unavailable, so this step is not currently needed]'''
	# Update the mirror index configuration (<code>~mirror/mirror-index-ng/synctask2project/config.toml</code>)		# Update the mirror index configuration (<code>~mirror/mirror-index-ng/synctask2project/config.toml</code>)
	# Add the project to rsync (<code>/etc/rsyncd.conf</code>)		# Add the project to rsync (<code>/etc/rsyncd.conf</code>)

Y266shen at 04:27, 21 June 2023

2023-06-21T04:27:40Z

← Older revision		Revision as of 23:27, 20 June 2023
Line 138:		Line 138:

	== Mirror Administration ==		== Mirror Administration ==

			=== Making changes ===
			Everything in the <code>~mirror</code> is managed by git (so a monorepo containing all sub-projects like Merlin and mirror-index). To make changes, switch to the mirror user and commit with <code>--author "FirstName LastName <email@csc></code> to show who made the change. Then run <code>git push</code> to push the changes. The remote is using the HTTPS URL, so just enter your CSC credentials.

	=== Adding a new project ===		=== Adding a new project ===

Y266shen at 04:21, 21 June 2023

2023-06-21T04:21:14Z

@@ Line 74: / Line 74: @@
 An index of the archives we mirror is available at [http://mirror.csclub.uwaterloo.ca mirror.csclub.uwaterloo.ca].
-As of Winter 2010, it is now generated by a Python script in <code>~mirror/mirror-index</code>.
+As of Spring 2023, it is now generated by Hugo.
-<code>~mirror/mirror-index/make-index</code> is scheduled in <code>/etc/cron.d/csc-mirror</code> to be run hourly. The script can be run manually when needed (for example, when the archive list is updated) by running:
+<code>~mirror/mirror-index/deploy.sh</code> is scheduled in <code>/etc/cron.d/csc-mirror</code> to be run every minute.
 === FTP ===

Ztseguin: /* Rename project */

2023-06-19T20:57:22Z

Rename project

← Older revision		Revision as of 15:57, 19 June 2023
Line 178:		Line 178:
	# Add a symlink for the old name (in <code>/mirror/root</code>) so that existing users won't be broken by the change		# Add a symlink for the old name (in <code>/mirror/root</code>) so that existing users won't be broken by the change
	#* <code>ln -s NEW_DIR OLD_DIR</code>		#* <code>ln -s NEW_DIR OLD_DIR</code>
			# Update the rsync daemon
			#* Edit <code>/etc/rsyncd.conf</code>, adding a new entry for the new name (keep the old name too). Restart with <code>systemctl restart rsync</code>
	# Modify index page generator config		# Modify index page generator config
	#* At <code>~mirror/mirror-index-ng/synctask2project/config.toml</code>		#* At <code>~mirror/mirror-index-ng/synctask2project/config.toml</code>

Ztseguin: /* Rename project */

2023-06-19T20:56:09Z

Rename project

← Older revision		Revision as of 15:56, 19 June 2023
Line 176:		Line 176:
	#* <code>rm OLD_DIR</code>		#* <code>rm OLD_DIR</code>
	#* <code>ln -s .cscmirror0/NEW_DIR NEW_DIR</code>		#* <code>ln -s .cscmirror0/NEW_DIR NEW_DIR</code>
			# Add a symlink for the old name (in <code>/mirror/root</code>) so that existing users won't be broken by the change
			#* <code>ln -s NEW_DIR OLD_DIR</code>
	# Modify index page generator config		# Modify index page generator config
	#* At <code>~mirror/mirror-index-ng/synctask2project/config.toml</code>		#* At <code>~mirror/mirror-index-ng/synctask2project/config.toml</code>
			# Update an mirror registrations with the project to ensure the new URLs are used

	=== Secondary Mirror ===		=== Secondary Mirror ===

Y266shen at 18:52, 13 June 2023

2023-06-13T18:52:31Z

← Older revision		Revision as of 13:52, 13 June 2023
Line 160:		Line 160:
	#* Check <code>~mirror/merlin/logs/$PROJECT_NAME</code> for errors, <code>~mirror/merlin/logs/transfer.log</code> for transfer progress		#* Check <code>~mirror/merlin/logs/$PROJECT_NAME</code> for errors, <code>~mirror/merlin/logs/transfer.log</code> for transfer progress
	# Configure the project in zfssync.yml (<code>~mirror/merlin/zfssync.yml</code>)		# Configure the project in zfssync.yml (<code>~mirror/merlin/zfssync.yml</code>)
	# Update the mirror index configuration (<code>~mirror/mirror-index/config.~~yaml~~</code>)		# Update the mirror index configuration (<code>~mirror/mirror-index-ng/synctask2project/config.toml</code>)
	# Add the project to rsync (<code>/etc/rsyncd.conf</code>)		# Add the project to rsync (<code>/etc/rsyncd.conf</code>)
	#* Restart rsync with <code>systemctl restart rsync</code>		#* Restart rsync with <code>systemctl restart rsync</code>

	If push mirroring is available/required, see [[#Push_Sync\|Push Sync]].		If push mirroring is available/required, see [[#Push_Sync\|Push Sync]].

			=== Rename project ===

			# Change project name (title) and local_dir in <code>merlin-config.ini</code>
			# Change zfs dataset name
			#* <code>zfs rename cscmirror0/OLD_NAME cscmirror0/NEW_NAME</code>
			# Reload merlin config
			#* <code>systemctl reload merlin-go.service</code>
			# Remove old symlink and create new symlink in mirror root
			#* <code>rm OLD_DIR</code>
			#* <code>ln -s .cscmirror0/NEW_DIR NEW_DIR</code>
			# Modify index page generator config
			#* At <code>~mirror/mirror-index-ng/synctask2project/config.toml</code>

	=== Secondary Mirror ===		=== Secondary Mirror ===

	The School of Computer Science's CSCF has provided us with a secondary mirror machine located in DC. This will limit the downtime of mirror.csclub in the event of an outage affecting the MC machine room.		The School of Computer Science's CSCF has provided us with a secondary mirror machine located in DC. This will limit the downtime of mirror.csclub in the event of an outage affecting the MC machine room.

			As of June 2023, CSCF mirror is down. CSCF is planing to bring it back with new hardware but no ETA.

	==== Keepalived ====		==== Keepalived ====

Merenber: /* FTP */

2023-05-01T00:46:45Z

FTP

← Older revision		Revision as of 19:46, 30 April 2023
Line 112:		Line 112:

	=== FTP ===		=== FTP ===

			<b>UPDATE</b>: We now use vsftpd instead. See /etc/vsftpd.conf for details. Official documentation can be found [https://manpages.debian.org/stable/vsftpd/vsftpd.conf.5.en.html here].

	We use [http://www.proftpd.org/ proftpd] (standalone daemon) as our FTP server.		We use [http://www.proftpd.org/ proftpd] (standalone daemon) as our FTP server.