Mirror
We currently run a public mirror (mirror.csclub.uwaterloo.ca) on sodium-benzoate. We are listed on the ResNet "don't count" list so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a bandwidth graph you can look at.
Archives Mirrored
Total Size: 4.4 TiB
| Archive | Upstream/Listing | Size | Notes |
|---|---|---|---|
| Apache | rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ |
23 GiB | - |
| Arch Linux | rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List |
75 GiB | - |
| Blastwave | rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php |
11 GiB | - |
| CentOS | rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ |
131 GiB | - |
| CPAN | rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ |
6.8 GiB | - |
| CRAN | rsync://cran.r-project.org/CRAN/ | 41 GiB | We should get added as an official mirror. |
| CTAN | rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ |
16 GiB | - |
| Cygwin | rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html |
11 GiB | - |
| Damn Small Linux | rsync://ftp.belnet.be/packages/damnsmalllinux/ | 18 GiB | Sent a request to be added. Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html. |
| Debian | rsync://debian.mirror.rafal.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ |
446 GiB | Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises.. |
| Debian-backports | rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html |
30 GiB | Submitted a request to be added to the mirror list long ago with no response. |
| Debian-cd | rsync://cdimage.debian.org/debian-cd/ | 70 GiB | Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...). |
| Debian-multimedia | rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php |
5.3 GiB | - |
| Debian-ports | rsync://ftp.debian-ports.org/debian/ | 124G | - |
| Debian-security | rsync://security.debian.org/debian-security/ | 36 GiB | Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively. |
| Debian-unofficial | rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html |
970 MiB | - |
| Debian-volatile | rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors |
3.3 GiB | - |
| Eclipse | rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ |
226 GiB | - |
| Emdebian | rsync://www.emdebian.org/emdebian/ | 3.7 GiB | Emdebian does not currently list mirrors. |
| Fedora | http://fedoraproject.org/wiki/Infrastructure/Mirroring http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager/site/647 rsync://ftp.muug.mb.ca/pub/fedora/linux/core/ |
548 GiB | - |
| FreeBSD | rsync://ftp1.ca.freebsd.org/ | 673 GiB | We should get added as an official mirror. |
| Gentoo (portage) | rsync://rsync1.us.gentoo.org/gentoo-portage/ | 581 MiB | We are rsync4.ca.gentoo.org. |
| Gentoo (sources) | rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ |
159 GiB | See ~sysadmin/passwords/gentoo for rsync password. |
| GNOME | rsync://ftp.gnome.org/gnome/ | 86 GiB | - |
| GNU | rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html |
20 GiB | - |
| KDE | rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ |
69 GiB | - |
| kernel.org | rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html |
130 GiB | - |
| Linux Mint (releases) | rsync://ftp.heanet.ie/pub/linuxmint.com/ http://www.linuxmint.com/mirrors.php |
23 GiB | - |
| Linux Mint (packages) | rsync://packages.linuxmint.com/packages/ http://www.linuxmint.com/mirrors.php |
4.1 GiB | - |
| mozdev | rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html |
6.0 GiB | Currently in the North American rotation, but could request to be added to the global rotation. |
| mozilla.org | rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html |
105 GiB | - |
| MySQL | rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html |
148 GiB | - |
| non-GNU | rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ |
15 GiB | - |
| Openoffice (extended set) | rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ |
102 GiB | - |
| OpenSUSE (opensuse-full) | rsync://stage.opensuse.org/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/all.html |
161 GiB | - |
| Slackware | - | 139 GiB | We should get added as an official mirror. |
| Ubuntu | rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors |
286 GiB | We used to be the official Canadian mirror (i.e., ca.archive.ubuntu.com); when we get more bandwidth get us added back. |
| Ubuntu-ports | rsync://ports.ubuntu.com/ubuntu-ports/ | 394 GiB | - |
| Ubuntu-ports-releases | rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ | 36 GiB | - |
| Ubuntu-releases | rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download |
39 GiB | Are the official Canadian mirror (i.e., ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu. |
| xorg.freedesktop.org | rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download |
5.3 GiB | - |
| Xubuntu-releases | rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get |
18 GiB | - |
| wine-budgetdedicated | http://wine.budgetdedicated.com/apt/ | 151 MiB | - |
Proposed Archives to Mirror
- Mandriva (774 GiB)
- OpenBSD (209 GiB)
- NetBSD (340 GiB)
- PCLinuxOS
Implementation Details
The mirroring is done by one of two scripts. Both are based on anonftpsync. Various cronjobs are scheduled to call one of these scripts.
csc-sync-debian
This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]
If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.
csc-sync-standard
This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:
csc-sync-standard local_dir rsync_host rsync_dir
merlin
The synchronization process is run by a Python script called "merlin", ostensibly written by mspang, stored in ~mirror/merlin. The repository list, sync time, etc. is maintained in merlin.py.
HTTP
We use Apache as our web server. Here's a snippet of the worker configuration:
<IfModule mpm_worker_module>
ServerLimit 64
ThreadLimit 64
StartServers 2
MaxClients 4096
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>
We use the bwbar application to display current bandwidth in the footer of mirror pages.
FTP
We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:
UseReverseDNS off IdentLookups off
We also limit the amount of CPU/memory resources used (e.g. to minimize Globbing resources):
RLimitCPU session 10 RLimitMemory session 4096K
We allow a maximum of 200 concurrent ftp sessions:
MaxInstances 500 MaxClients 500
Rsync
We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:
dont compress = * refuse options = c delete
For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.