Difference between revisions of "Mirror"
(removed VLC from proposed archives; as of writing, we are currently mirroring it)
|Line 10:||Line 10:|
== Implementation Details ==
== Implementation Details ==
Revision as of 10:33, 20 January 2014
We currently run a public mirror (mirror.csclub.uwaterloo.ca) on sodium-benzoate. We are listed on the ResNet "don't count" list so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to email@example.com. We also have a bandwidth graph you can look at.
For a list of what is currently mirrored and their respective disk usage see http://mirror.csclub.uwaterloo.ca/
Proposed Archives to Mirror
The mirroring is done by one of three scripts. The latter two are based on anonftpsync. merlin is used to call one of these scripts. Most of the scripts and such used to maintain the mirror are available in the public mirror.git repository.
ftpsync is the official Debian mirror synchronization tool, and is used to rsync the Debian repository. It's located in ~mirror/debian. Its invocation takes a few steps (this is more or less how merlin invokes it:
export BASEDIR=/home/mirror/debian cd $BASEDIR ./bin/ftpsync sync:stage1 ./bin/ftpsync sync:stage2
This is used to sync debian-style repositories. Its usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]
If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.
This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. Its usage is:
csc-sync-standard local_dir rsync_host rsync_dir
The synchronization process is run by a Python script called "merlin", written by a2brenna, stored in ~mirror/merlin. The repository list, sync time, etc. is maintained in merlin.py.
We use Apache as our web server. Here's a snippet of the worker configuration:
<IfModule mpm_worker_module> ServerLimit 64 ThreadLimit 64 StartServers 2 MaxClients 4096 MinSpareThreads 16 MaxSpareThreads 48 ThreadsPerChild 64 MaxRequestsPerChild 0 </IfModule>
We use the bwbar application to display current bandwidth in the footer of mirror pages.
An index of the archives we mirror is available at http://mirror.csclub.uwaterloo.ca/. As of Winter 2010, it is now generated by a Python script in ~mirror/mirror-index.
~mirror/mirror-index/make-index.py is scheduled in mirror's crontab to be run at 5:40 AM on the 14th and 28th of each month. The script can be run manually when needed (for example, when an archive is removed) as follows:
sudo -u mirror /home/mirror/mirror-index/make-index.py
This causes an instance of du to be run which computes the size of each directory. This list is then sorted alphabetically by directory name and returned to the Python script. If any errors occur during this process, the script conservatively chooses to exit rather than risk generating an index file that is incorrect.
make-index.py is configured by means of a YAML file, config.yaml, in the same directory. Its format is as follows:
docroot: /mirror/root duflags: --human-readable --max-depth=1 output: /mirror/root/index.html directories: apache: site: apache.org url: http://www.apache.org/ archlinux: site: archlinux.org url: http://www.archlinux.org/ # (...)
The docroot is the directory which is to be scanned; this will probably always be the mirror root from which Apache serves. duflags specifies the flags to be passed to du. This is here so that it's easy to find and alter. For instance, we could change --human-readable to --si if we ever decided that, like hard disk manufacturers, we want sizes to appear larger than they are. output defines the file to which the generated index will be written.
Finally, directories specifies the list of directories to be listed. No directories not listed here will be shown. If you add a new archive and it doesn't appear, that's why. The format is fairly straightforward: simply name the directory and provide a site (the display name in the "Project Site" column) and URL.
One caveat here is that YAML does not allow tabs for whitespace. Indent with two spaces to remain consistent with the existing file format, please. Also note that the directory name is case-sensitive, as is always the case on Unix.
Finally, the HTML index file is generated from index.mako, a Mako template (which is mostly HTML anyhow). If you really can't figure out how it works, look up the Mako documentation.
We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:
UseReverseDNS off IdentLookups off
We also limit the amount of CPU/memory resources used (e.g. to minimize Globbing resources):
RLimitCPU session 10 RLimitMemory session 4096K
We allow a maximum of 200 concurrent ftp sessions:
MaxInstances 500 MaxClients 500
We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:
dont compress = * refuse options = c delete
For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.