Mirror
Jump to navigation
Jump to search
We currently run a public mirror (mirror.csclub.uwaterloo.ca) on sodium-benzoate. We are listed on the ResNet "don't count" list so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.
Archives Mirrored
Total Size: 2.28 TiB
| Archive | Upstream/Listing | Size | Notes |
|---|---|---|---|
| Apache | rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ |
17 GiB | - |
| Arch Linux | rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List |
23 GiB | - |
| Blastwave | rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php |
15 GiB | Blastwave does not currently list mirrors. |
| CentOS | rsync://us-msync.centos.org/CentOS/ | 106 GiB | - |
| CPAN | rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ |
5 GiB | - |
| CRAN | rsync://cran.r-project.org/CRAN/ | 20 GiB | - |
| CTAN | rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ |
10 GiB | - |
| Cygwin | rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html |
5 GiB | - |
| Damn Small Linux | rsync://ftp.belnet.be/packages/damnsmalllinux/ | 17 GiB | - |
| Debian | rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ |
316 GiB | Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises.. |
| Debian-backports | rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html |
36 GiB | Submitted a request to be added to the mirror list. |
| Debian-multimedia | rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php |
2 GiB | - |
| Debian-security | rsync://security.debian.org/debian-security/ | 46 GiB | Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively. |
| Debian-unofficial | rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html |
1 GiB | - |
| Debian-volatile | rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors |
4 GiB | - |
| Eclipse | rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ |
138 GiB | - |
| Emdebian | rsync://www.emdebian.org/emdebian/ | 3 GiB | Emdebian does not currently list mirrors. |
| Gentoo (portage) | rsync://rsync1.us.gentoo.org/gentoo-portage/ | 1 GiB | We are rsync4.ca.gentoo.org. |
| Gentoo (sources) | rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ |
92 GiB | See ~sysadmin/passwords/gentoo for rsync password. |
| GNOME | rsync://ftp.gnome.org/gnome/ | 65 GiB | - |
| GNU | rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html |
14 GiB | - |
| Gobuntu-releases | rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ | 7 GiB | Canonical does not currently list mirrors. |
| KDE | rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ |
65 GiB | - |
| kernel.org | rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html |
82 GiB | - |
| mozdev | rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html |
5 GiB | Currently in the North American rotation, but could request to be added to the global rotation. |
| mozilla.org | rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html |
95 GiB | - |
| MySQL | rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html |
76 GiB | - |
| non-GNU | rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html |
8 GiB | - |
| Openoffice (extended set) | rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://ftp.stardiv.de/mirmon/mirror-state.html#ca |
136 GiB | - |
| OpenSUSE (opensuse-full) | rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://download.opensuse.org/distribution/11.0/iso/dvd/MD5SUMS?mirrorlist |
230 GiB | - |
| Slackware | - | 27 GiB | - |
| Ubuntu | rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors |
261 GiB | Are the official Canadian mirror (i.e., ca.archive.ubuntu.com) |
| Ubuntu-releases | rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download |
37 GiB | Are the official Canadian mirror (i.e., ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu. |
| xorg.freedesktop.org | rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download |
5 GiB | - |
| Xubuntu-releases | rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get |
20 GiB | - |
Propsed Archives to Mirror
- openSUSE (306 GiB)
- Mandriva (774 GiB)
- OpenBSD (54 GiB)
- FreeBSD (500 GiB)
- NetBSD (340 GiB)
Implementation Details
The mirroring is done by one of two scripts. Both are based on anonftpsync. Various cronjobs are scheduled to call one of these scripts.
csc-sync-debian
This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]
If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.
csc-sync-standard
This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:
csc-sync-standard local_dir rsync_host rsync_dir
Crontab
All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:
# m h dom mon dow command # # bi-hourly # 5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org 5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net 5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org 5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net 5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org 5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org 5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com 5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases 5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/ 5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/ # # daily # 5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian 5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository 5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases 5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN 5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN 5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan 5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice 5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp 5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/ 5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases 5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/ 5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/ 5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist 5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror 5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql 5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp 5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev 5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/ 5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux 5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian 5 5,17 * * * ~/bin/csc-sync-debian-cd # # other # 29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo 15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage 5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases
HTTP
We use Apache as our web server. Here's a snippet of the worker configuration:
<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>
We use the bwbar application to display current bandwidth in the footer of mirror pages.
FTP
We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:
UseReverseDNS off IdentLookups off
We also limit the amount of CPU/memory resources used (e.g. to minimize Globbing resources):
RLimitCPU session 10 RLimitMemory session 4096K
We allow a maximum of 200 concurrent ftp sessions:
MaxInstances 500 MaxClients 500
Rsync
We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:
dont compress = * refuse options = c delete
For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.