Mirror
Jump to navigation
Jump to search
We currently run a public mirror (mirror.csclub.uwaterloo.ca) on sodium-benzoate. We are listed on the ResNet "don't count" list so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.
Archives Mirrored
Total Size: 4.5 TiB
| Archive | Upstream/Listing | Size | Notes |
|---|---|---|---|
| Apache | rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ |
20 GiB | - |
| Arch Linux | rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List |
41 GiB | - |
| Blastwave | rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php |
10 GiB | - |
| CentOS | rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ |
106 GiB | - |
| CPAN | rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ |
5 GiB | - |
| CRAN | rsync://cran.r-project.org/CRAN/ | 36 GiB | We should get added as an official mirror. |
| CTAN | rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ |
10 GiB | - |
| Cygwin | rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html |
5 GiB | - |
| Damn Small Linux | rsync://ftp.belnet.be/packages/damnsmalllinux/ | 17 GiB | - |
| Debian | rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ |
316 GiB | Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises.. |
| Debian-backports | rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html |
36 GiB | Submitted a request to be added to the mirror list. |
| Debian-multimedia | rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php |
2 GiB | - |
| Debian-security | rsync://security.debian.org/debian-security/ | 46 GiB | Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively. |
| Debian-unofficial | rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html |
1 GiB | - |
| Debian-volatile | rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors |
4 GiB | - |
| Eclipse | rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ |
138 GiB | - |
| Emdebian | rsync://www.emdebian.org/emdebian/ | 3 GiB | Emdebian does not currently list mirrors. |
| Fedora | rsync://archive.linux.duke.edu/fedora-enchilada/ | 873 GiB | We should get added as an official mirror. |
| FreeBSD | rsync://ftp1.ca.freebsd.org/ | 718 GiB | We should get added as an official mirror. |
| Gentoo (portage) | rsync://rsync1.us.gentoo.org/gentoo-portage/ | 1 GiB | We are rsync4.ca.gentoo.org. |
| Gentoo (sources) | rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ |
92 GiB | See ~sysadmin/passwords/gentoo for rsync password. |
| GNOME | rsync://ftp.gnome.org/gnome/ | 65 GiB | - |
| GNU | rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html |
14 GiB | - |
| Gobuntu-releases | rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ | 7 GiB | Canonical does not currently list mirrors. |
| KDE | rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ |
65 GiB | - |
| kernel.org | rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html |
82 GiB | - |
| mozdev | rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html |
5 GiB | Currently in the North American rotation, but could request to be added to the global rotation. |
| mozilla.org | rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html |
95 GiB | - |
| MySQL | rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html |
76 GiB | - |
| non-GNU | rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ |
8 GiB | - |
| Openoffice (extended set) | rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ |
136 GiB | - |
| OpenSUSE (opensuse-full) | rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://download.opensuse.org/distribution/11.0/iso/dvd/MD5SUMS?mirrorlist |
230 GiB | - |
| Slackware | - | 27 GiB | - |
| Ubuntu | rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors |
261 GiB | Are the official Canadian mirror (i.e., ca.archive.ubuntu.com) |
| Ubuntu-releases | rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download |
37 GiB | Are the official Canadian mirror (i.e., ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu. |
| xorg.freedesktop.org | rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download |
5 GiB | - |
| Xubuntu-releases | rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get |
20 GiB | - |
Propsed Archives to Mirror
- openSUSE (306 GiB)
- Mandriva (774 GiB)
- OpenBSD (54 GiB)
- FreeBSD (500 GiB)
- NetBSD (340 GiB)
Implementation Details
The mirroring is done by one of two scripts. Both are based on anonftpsync. Various cronjobs are scheduled to call one of these scripts.
csc-sync-debian
This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]
If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.
csc-sync-standard
This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:
csc-sync-standard local_dir rsync_host rsync_dir
Crontab
All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:
# m h dom mon dow command # # bi-hourly # 5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org 5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net 5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org 5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net 5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org 5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org 5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com 5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases 5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/ 5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/ # # daily # 5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian 5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository 5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases 5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN 5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN 5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan 5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice 5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp 5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/ 5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases 5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/ 5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/ 5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist 5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror 5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql 5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp 5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev 5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/ 5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux 5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian 5 5,17 * * * ~/bin/csc-sync-debian-cd 5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive 5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome 5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS 5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/ 5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/ # # other # 29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo 15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage 5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases
HTTP
We use Apache as our web server. Here's a snippet of the worker configuration:
<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>
We use the bwbar application to display current bandwidth in the footer of mirror pages.
We use mod_bw to ensure every connection is at least 100 KiB/s.
FTP
We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:
UseReverseDNS off IdentLookups off
We also limit the amount of CPU/memory resources used (e.g. to minimize Globbing resources):
RLimitCPU session 10 RLimitMemory session 4096K
We allow a maximum of 200 concurrent ftp sessions:
MaxInstances 500 MaxClients 500
Rsync
We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:
dont compress = * refuse options = c delete
For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.