Mirror

We currently run a public mirror (mirror.csclub.uwaterloo.ca) on sodium-benzoate. We are listed on the ResNet "don't count" list so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.

Archives Mirrored

Total Size: 4.5 TiB

Archive	Upstream/Listing	Size	Notes
Apache	rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/	20 GiB	-
Arch Linux	rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List	41 GiB	-
Blastwave	rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php	10 GiB	-
CentOS	rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/	125 GiB	-
CPAN	rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/	6 GiB	-
CRAN	rsync://cran.r-project.org/CRAN/	36 GiB	We should get added as an official mirror.
CTAN	rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/	16 GiB	-
Cygwin	rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html	10 GiB	-
Damn Small Linux	rsync://ftp.belnet.be/packages/damnsmalllinux/	18 GiB	We should get added as an official mirror. Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
Debian	rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/	316 GiB	Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
Debian-backports	rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html	28 GiB	Submitted a request to be added to the mirror list.
Debian-cd	rsync://cdimage.debian.org/debian-cd/	68 GiB	Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
Debian-multimedia	rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php	5 GiB	-
Debian-security	rsync://security.debian.org/debian-security/	46 GiB	Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
Debian-unofficial	rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html	1 GiB	-
Debian-volatile	rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors	4 GiB	-
Eclipse	rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/	138 GiB	-
Emdebian	rsync://www.emdebian.org/emdebian/	3 GiB	Emdebian does not currently list mirrors.
Fedora	rsync://archive.linux.duke.edu/fedora-enchilada/	873 GiB	We should get added as an official mirror.
FreeBSD	rsync://ftp1.ca.freebsd.org/	718 GiB	We should get added as an official mirror.
Gentoo (portage)	rsync://rsync1.us.gentoo.org/gentoo-portage/	1 GiB	We are rsync4.ca.gentoo.org.
Gentoo (sources)	rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/	92 GiB	See ~sysadmin/passwords/gentoo for rsync password.
GNOME	rsync://ftp.gnome.org/gnome/	65 GiB	-
GNU	rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html	14 GiB	-
Gobuntu-releases	rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/	7 GiB	Canonical does not currently list mirrors.
KDE	rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/	65 GiB	-
kernel.org	rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html	82 GiB	-
mozdev	rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html	5 GiB	Currently in the North American rotation, but could request to be added to the global rotation.
mozilla.org	rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html	95 GiB	-
MySQL	rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html	76 GiB	-
non-GNU	rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/	8 GiB	-
Openoffice (extended set)	rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/	136 GiB	-
OpenSUSE (opensuse-full)	rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://download.opensuse.org/distribution/11.0/iso/dvd/MD5SUMS?mirrorlist	230 GiB	-
Slackware	-	27 GiB	-
Ubuntu	rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors	261 GiB	Are the official Canadian mirror (i.e., ca.archive.ubuntu.com)
Ubuntu-releases	rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download	37 GiB	Are the official Canadian mirror (i.e., ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
xorg.freedesktop.org	rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download	5 GiB	-
Xubuntu-releases	rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get	20 GiB	-

Propsed Archives to Mirror

• openSUSE (306 GiB)
• Mandriva (774 GiB)
• OpenBSD (54 GiB)
• FreeBSD (500 GiB)
• NetBSD (340 GiB)

Implementation Details

The mirroring is done by one of two scripts. Both are based on anonftpsync. Various cronjobs are scheduled to call one of these scripts.

csc-sync-debian

This is used to sync debian-style repositories. It's usage is:

csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

csc-sync-standard

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

Crontab

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m  h     dom mon dow command

#
# bi-hourly
#
  5  */2   *   *   *   ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
  5  */2   *   *   *   ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
  5  */2   *   *   *   ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
  5  */2   *   *   *   ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
  5  */2   *   *   *   ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
  5  3,15  *  *   *   ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
  5  3,15  *  *   *   ~/bin/csc-sync-standard nexenta nexenta.org repository
  5  3,15  *  *   *   ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
  5  3,15  *  *   *   ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
  5  3,15  *  *   *   ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
  5  3,15  *  *   *   ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
  5  3,15  *  *   *   ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
  5  4,16  *  *   *   ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
  5  4,16  *  *   *   ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
  5  4,16  *  *   *   ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
  5  4,16  *  *   *   ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
  5  4,16  *  *   *   ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
  5  4,16  *  *   *   ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
  5  4.16  *  *   *   ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
  5  5,17  *  *   *   ~/bin/csc-sync-standard mysql mysql.he.net mysql
  5  5,17  *  *   *   ~/bin/csc-sync-standard kde master.kde.org kdeftp
  5  5,17  *  *   *   ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
  5  5,17  *  *   *   ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
  5  5,17  *  *   *   ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
  5  5,17  *  *   *   ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
  5  5,17  *  *   *   ~/bin/csc-sync-debian-cd
  5  6,18  *  *   *   ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
  5  6,18  *  *   *   ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
  5  6,18  *  *   *   ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
  5  6,18  *  *   *   ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
  5  6,18  *  *   *   ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
  29 */4   *  *   *   RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
  15,45 *  *  *   *   ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
  5,35  *  *  *   *   ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

HTTP

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
    ServerLimit          32
    ThreadLimit          64
    StartServers          2
    MaxClients         2048
    MinSpareThreads      16
    MaxSpareThreads      48
    ThreadsPerChild      64
    MaxRequestsPerChild   0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

FTP

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS           off
IdentLookups            off

We also limit the amount of CPU/memory resources used (e.g. to minimize Globbing resources):

RLimitCPU               session 10
RLimitMemory            session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances            500
MaxClients              500

Rsync

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.