Mirror

We currently run a public mirror (mirror.csclub.uwaterloo.ca) on sodium-benzoate. We are listed on the ResNet "don't count" list so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a bandwidth graph you can look at.

Archives Mirrored

Total Size: 4.4 TiB

Archive	Upstream/Listing	Size	Notes
Apache	rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/	23 GiB	-
Arch Linux	rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List	75 GiB	-
Blastwave	rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php	11 GiB	-
CentOS	rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/	131 GiB	-
CPAN	rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/	6.8 GiB	-
CRAN	rsync://cran.r-project.org/CRAN/	41 GiB	We should get added as an official mirror.
CTAN	rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/	16 GiB	-
Cygwin	rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html	11 GiB	-
Damn Small Linux	rsync://ftp.belnet.be/packages/damnsmalllinux/	18 GiB	Sent a request to be added. Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
Debian	rsync://debian.mirror.rafal.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/	446 GiB	Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
Debian-backports	rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html	30 GiB	Submitted a request to be added to the mirror list long ago with no response.
Debian-cd	rsync://cdimage.debian.org/debian-cd/	70 GiB	Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
Debian-multimedia	rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php	5.3 GiB	-
Debian-ports	rsync://ftp.debian-ports.org/debian/	124G	-
Debian-security	rsync://security.debian.org/debian-security/	36 GiB	Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
Debian-unofficial	rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html	970 MiB	-
Debian-volatile	rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors	3.3 GiB	-
Eclipse	rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/	226 GiB	-
Emdebian	rsync://www.emdebian.org/emdebian/	3.7 GiB	Emdebian does not currently list mirrors.
Fedora	http://fedoraproject.org/wiki/Infrastructure/Mirroring http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager/site/647 rsync://ftp.muug.mb.ca/pub/fedora/linux/core/	548 GiB	-
FreeBSD	rsync://ftp1.ca.freebsd.org/	673 GiB	We should get added as an official mirror.
Gentoo (portage)	rsync://rsync1.us.gentoo.org/gentoo-portage/	581 MiB	We are rsync4.ca.gentoo.org.
Gentoo (sources)	rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/	159 GiB	See ~sysadmin/passwords/gentoo for rsync password.
GNOME	rsync://ftp.gnome.org/gnome/	86 GiB	-
GNU	rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html	20 GiB	-
KDE	rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/	69 GiB	-
kernel.org	rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html	130 GiB	-
Linux Mint (releases)	rsync://ftp.heanet.ie/pub/linuxmint.com/ http://www.linuxmint.com/mirrors.php	23 GiB	-
Linux Mint (packages)	rsync://packages.linuxmint.com/packages/ http://www.linuxmint.com/mirrors.php	4.1 GiB	-
mozdev	rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html	6.0 GiB	Currently in the North American rotation, but could request to be added to the global rotation.
mozilla.org	rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html	105 GiB	-
MySQL	rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html	148 GiB	-
non-GNU	rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/	15 GiB	-
Openoffice (extended set)	rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/	102 GiB	-
OpenSUSE (opensuse-full)	rsync://stage.opensuse.org/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/all.html	161 GiB	-
Slackware	-	139 GiB	We should get added as an official mirror.
Ubuntu	rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors	286 GiB	We used to be the official Canadian mirror (i.e., ca.archive.ubuntu.com); when we get more bandwidth get us added back.
Ubuntu-ports	rsync://ports.ubuntu.com/ubuntu-ports/	394 GiB	-
Ubuntu-ports-releases	rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/	36 GiB	-
Ubuntu-releases	rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download	39 GiB	Are the official Canadian mirror (i.e., ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
xorg.freedesktop.org	rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download	5.3 GiB	-
Xubuntu-releases	rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get	18 GiB	-
wine-budgetdedicated	http://wine.budgetdedicated.com/apt/	151 MiB	-

Proposed Archives to Mirror

• Mandriva (774 GiB)
• OpenBSD (209 GiB)
• NetBSD (340 GiB)
• PCLinuxOS

Implementation Details

The mirroring is done by one of two scripts. Both are based on anonftpsync. Various cronjobs are scheduled to call one of these scripts.

csc-sync-debian

This is used to sync debian-style repositories. It's usage is:

csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

csc-sync-standard

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

merlin

The synchronization process is run by a Python script called "merlin", ostensibly written by mspang, stored in ~mirror/merlin. The repository list, sync time, etc. is maintained in merlin.py.

HTTP

We use Apache as our web server. Here's a snippet of the worker configuration:

 <IfModule mpm_worker_module>
     ServerLimit          64
     ThreadLimit          64
     StartServers          2
     MaxClients         4096
     MinSpareThreads      16
     MaxSpareThreads      48
     ThreadsPerChild      64
     MaxRequestsPerChild   0
 </IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

FTP

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS           off
IdentLookups            off

We also limit the amount of CPU/memory resources used (e.g. to minimize Globbing resources):

RLimitCPU               session 10
RLimitMemory            session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances            500
MaxClients              500

Rsync

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.