Mirror
Jump to navigation
Jump to search
We currently run a public mirror (mirror.csclub.uwaterloo.ca) on sodium-benzoate. We are listed on the ResNet "don't count" list so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a bandwidth graph you can look at.
Archives Mirrored
Total Size: 4.4 TiB
| Archive | Upstream/Listing | Size | Notes |
|---|---|---|---|
| Apache | rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ |
23 GiB | - |
| Arch Linux | rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List |
75 GiB | - |
| Blastwave | rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php |
11 GiB | - |
| CentOS | rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ |
131 GiB | - |
| CPAN | rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ |
6.8 GiB | - |
| CRAN | rsync://cran.r-project.org/CRAN/ | 41 GiB | We should get added as an official mirror. |
| CTAN | rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ |
16 GiB | - |
| Cygwin | rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html |
11 GiB | - |
| Damn Small Linux | rsync://ftp.belnet.be/packages/damnsmalllinux/ | 18 GiB | Sent a request to be added. Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html. |
| Debian | rsync://debian.mirror.rafal.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ |
446 GiB | Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises.. |
| Debian-backports | rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html |
30 GiB | Submitted a request to be added to the mirror list long ago with no response. |
| Debian-cd | rsync://cdimage.debian.org/debian-cd/ | 70 GiB | Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...). |
| Debian-multimedia | rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php |
5.3 GiB | - |
| Debian-ports | rsync://ftp.debian-ports.org/debian/ | 124G | - |
| Debian-security | rsync://security.debian.org/debian-security/ | 36 GiB | Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively. |
| Debian-unofficial | rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html |
970 MiB | - |
| Debian-volatile | rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors |
3.3 GiB | - |
| Eclipse | rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ |
226 GiB | - |
| Emdebian | rsync://www.emdebian.org/emdebian/ | 3.7 GiB | Emdebian does not currently list mirrors. |
| Fedora | http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager/site/647 rsync://ftp.muug.mb.ca/pub/fedora/linux/core/ |
548 GiB | - |
| FreeBSD | rsync://ftp1.ca.freebsd.org/ | 673 GiB | We should get added as an official mirror. |
| Gentoo (portage) | rsync://rsync1.us.gentoo.org/gentoo-portage/ | 581 MiB | We are rsync4.ca.gentoo.org. |
| Gentoo (sources) | rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ |
159 GiB | See ~sysadmin/passwords/gentoo for rsync password. |
| GNOME | rsync://ftp.gnome.org/gnome/ | 86 GiB | - |
| GNU | rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html |
20 GiB | - |
| KDE | rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ |
69 GiB | - |
| kernel.org | rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html |
130 GiB | - |
| Linux Mint (releases) | rsync://ftp.heanet.ie/pub/linuxmint.com/ http://www.linuxmint.com/mirrors.php |
23 GiB | - |
| Linux Mint (packages) | rsync://packages.linuxmint.com/packages/ http://www.linuxmint.com/mirrors.php |
4.1 GiB | - |
| mozdev | rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html |
6.0 GiB | Currently in the North American rotation, but could request to be added to the global rotation. |
| mozilla.org | rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html |
105 GiB | - |
| MySQL | rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html |
148 GiB | - |
| non-GNU | rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ |
15 GiB | - |
| Openoffice (extended set) | rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ |
102 GiB | - |
| OpenSUSE (opensuse-full) | rsync://stage.opensuse.org/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/all.html |
161 GiB | - |
| Slackware | - | 139 GiB | We should get added as an official mirror. |
| Ubuntu | rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors |
286 GiB | We used to be the official Canadian mirror (i.e., ca.archive.ubuntu.com); when we get more bandwidth get us added back. |
| Ubuntu-ports | rsync://ports.ubuntu.com/ubuntu-ports/ | 394 GiB | - |
| Ubuntu-ports-releases | rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ | 36 GiB | - |
| Ubuntu-releases | rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download |
39 GiB | Are the official Canadian mirror (i.e., ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu. |
| xorg.freedesktop.org | rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download |
5.3 GiB | - |
| Xubuntu-releases | rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get |
18 GiB | - |
| wine-budgetdedicated | http://wine.budgetdedicated.com/apt/ | 151 MiB | - |
Proposed Archives to Mirror
- Mandriva (774 GiB)
- OpenBSD (209 GiB)
- NetBSD (340 GiB)
- PCLinuxOS
Implementation Details
The mirroring is done by one of two scripts. Both are based on anonftpsync. Various cronjobs are scheduled to call one of these scripts.
csc-sync-debian
This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]
If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.
csc-sync-standard
This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:
csc-sync-standard local_dir rsync_host rsync_dir
Crontab
All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:
# m h dom mon dow command # make torrents */10 * * * * /home/mirror/bin/make-torrents > /dev/null 2> /dev/null # # bi-hourly # 5 */2 * * * ~/bin/csc-sync-debian debian debian.mirror.rafal.ca debian ftp-master.debian.org 5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net 5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org 5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile-master.debian.org debian-volatile volatile-master.debian.org 5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org 5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org 5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com 5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases rsync.releases.ubuntu.com releases 5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/ 5 */2 * * * ~/bin/csc-sync-debian ubuntu-ports ports.ubuntu.com ubuntu-ports drescher.canonical.com 5 */2 * * * ~/bin/csc-sync-debian linuxmint-packages packages.linuxmint.com packages # # daily # 5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org debian 5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN 5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN 5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan 5 3,15 * * * ~/bin/csc-sync-standard openoffice rsync.services.openoffice.org openoffice-extended # 5 3,15 * * * ~/bin/csc-sync-standard fedora/epel fedora-archives.ibiblio.org fedora-epel && ~/bin/report_mirror >/dev/null 5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp 5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/ 5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases --ignore-errors 5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/ 5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/ 5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist 5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror 5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql 5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp 5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev 5 5,17 * * * ~/bin/csc-sync-standard blastwave master.rsync.blastwave.org blastwave 5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux 5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian --ignore-errors 5 5,17 * * * ~/bin/csc-sync-standard slackware slackware.cs.utah.edu slackware 5 5,17 * * * ~/bin/csc-sync-debian-cd 5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive 5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome 5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS 5 6,18 * * * ~/bin/csc-sync-standard opensuse stage.opensuse.org opensuse-full/opensuse/ #"--exclude distribution/.timestamp_invisible" 5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.heanet.ie mirrors/damnsmalllinux.org/ 5 7,19 * * * ~/bin/csc-sync-standard FreeBSD ftp1.ca.freebsd.org freebsd # 5 7,19 * * * ~/bin/csc-sync-standard fedora/linux fedora-archives.ibiblio.org fedora-enchilada/linux/ --ignore-errors && ~/bin/report_mirror >/dev/null 5 7,19 * * * ~/bin/csc-sync-standard linuxmint ftp.heanet.ie pub/linuxmint.com/ 5 7,19 * * * ~/bin/csc-sync-standard ubuntu-ports-releases cdimage.ubuntu.com cdimage/ports/releases/ # # other # 29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo 15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage 5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases
HTTP
We use Apache as our web server. Here's a snippet of the worker configuration:
<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>
We use the bwbar application to display current bandwidth in the footer of mirror pages.
We use mod_bw to ensure every connection is at least 100 KiB/s.
FTP
We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:
UseReverseDNS off IdentLookups off
We also limit the amount of CPU/memory resources used (e.g. to minimize Globbing resources):
RLimitCPU session 10 RLimitMemory session 4096K
We allow a maximum of 200 concurrent ftp sessions:
MaxInstances 500 MaxClients 500
Rsync
We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:
dont compress = * refuse options = c delete
For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.