Mirror

From CSCWiki
Jump to navigation Jump to search

We currently run a public mirror (mirror.csclub.uwaterloo.ca) on sodium-benzoate. We are listed on the ResNet "don't count" list so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.

Archives Mirrored

Total Size: 4.5 TiB

Archive Upstream/Listing Size Notes
Apache rsync://rsync.us.apache.org/apache-dist/
http://www.apache.org/mirrors/
20 GiB -
Arch Linux rsync://mirror.rit.edu/archlinux
http://www.archlinux.org/download/
http://wiki.archlinux.org/index.php/Mirrors#Mirror_List
41 GiB -
Blastwave rsync://www.ibiblio.org/sun-packages/csw/
http://www.blastwave.org/mirrors.php
10 GiB -
CentOS rsync://us-msync.centos.org/CentOS/
http://mirror-status.centos.org/
106 GiB -
CPAN rsync://rsync.nic.funet.fi/CPAN/
http://mirror.cpan.org/
5 GiB -
CRAN rsync://cran.r-project.org/CRAN/ 36 GiB We should get added as an official mirror.
CTAN rsync://carroll.aset.psu.edu/ctan/
http://www.dante.de/cgi-bin/ctan/list.cgi
http://www.dante.de/mirmon/
10 GiB -
Cygwin rsync://cygwin.com/cygwin-ftp/
http://www.cygwin.com/mirrors.html
5 GiB -
Damn Small Linux rsync://ftp.belnet.be/packages/damnsmalllinux/ 17 GiB -
Debian rsync://gulus.usherbrooke.ca/debian/
http://www.debian.org/mirror/list
http://www.de.debian.org/dmc/today/
316 GiB Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
Debian-backports rsync://www.backports.org/backports.org/
http://www.backports.org/debian/README.mirrors.html
36 GiB Submitted a request to be added to the mirror list.
Debian-multimedia rsync://www.debian-multimedia.org/debian/
http://www.debian-multimedia.org/debian-m.php
2 GiB -
Debian-security rsync://security.debian.org/debian-security/ 46 GiB Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
Debian-unofficial rsync://ftp.debian-unofficial.org/debian/
http://www.debian-unofficial.org/mirrors.html
1 GiB -
Debian-volatile rsync://volatile.debian.org/debian-volatile/
http://www.debian.org/volatile/volatile-mirrors
4 GiB -
Eclipse rsync://download.eclipse.org/eclipseMirror
http://www.eclipse.org/downloads/download.php?file=/
138 GiB -
Emdebian rsync://www.emdebian.org/emdebian/ 3 GiB Emdebian does not currently list mirrors.
Fedora rsync://archive.linux.duke.edu/fedora-enchilada/ 873 GiB We should get added as an official mirror.
FreeBSD rsync://ftp1.ca.freebsd.org/ 718 GiB We should get added as an official mirror.
Gentoo (portage) rsync://rsync1.us.gentoo.org/gentoo-portage/ 1 GiB We are rsync4.ca.gentoo.org.
Gentoo (sources) rsync://masterdistfiles.gentoo.org/gentoo/
http://www.gentoo.org/main/en/mirrors.xml
http://mirrorstats.gentoo.org/
92 GiB See ~sysadmin/passwords/gentoo for rsync password.
GNOME rsync://ftp.gnome.org/gnome/ 65 GiB -
GNU rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/
http://www.gnu.org/order/ftp.html
14 GiB -
Gobuntu-releases rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ 7 GiB Canonical does not currently list mirrors.
KDE rsync://master.kde.org/kdeftp/
http://www.kde.org/mirrors/
65 GiB -
kernel.org rsync://kernel.org/pub/linux/
rsync://kernel.org/pub/software/
http://kernel.org/mirrors/countries/html/CA.html
82 GiB -
mozdev rsync://rsync.mozdev.org/mozdev/
http://mirrors.mozdev.org/index.html
5 GiB Currently in the North American rotation, but could request to be added to the global rotation.
mozilla.org rsync://releases-rsync.mozilla.org/mozilla-releases/
http://www.mozilla.org/mirrors.html
95 GiB -
MySQL rsync://mysql.mirrors.pair.com/mysql/
http://dev.mysql.com/downloads/mirrors.html
76 GiB -
non-GNU rsync://dl.sv.gnu.org/releases/
http://dl.sv.gnu.org/releases/00_MIRRORS.html
http://download.savannah.gnu.org/mirmon/
8 GiB -
Openoffice (extended set) rsync://rsync.services.openoffice.org/openoffice-extended/
http://distribution.openoffice.org/mirrors/#mirrors
http://www.ooodev.org/mirmon/
136 GiB -
OpenSUSE (opensuse-full) rsync://suse.mirrors.tds.net/opensuse-full/opensuse/
http://download.opensuse.org/distribution/11.0/iso/dvd/MD5SUMS?mirrorlist
230 GiB -
Slackware - 27 GiB -
Ubuntu rsync://archives.ubuntu.com/ubuntu/
https://launchpad.net/ubuntu/+archivemirrors
261 GiB Are the official Canadian mirror (i.e., ca.archive.ubuntu.com)
Ubuntu-releases rsync://releases.ubuntu.com/releases/
https://launchpad.net/ubuntu/+cdmirrors
http://www.ubuntu.com/getubuntu/download
http://www.kubuntu.org/download.php
http://www.edubuntu.org/Download
37 GiB Are the official Canadian mirror (i.e., ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
xorg.freedesktop.org rsync://xorg.freedesktop.org/xorg-archive/
http://www.x.org/wiki/Releases/Download
5 GiB -
Xubuntu-releases rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/
http://www.xubuntu.org/get
20 GiB -

Propsed Archives to Mirror

  • openSUSE (306 GiB)
  • Mandriva (774 GiB)
  • OpenBSD (54 GiB)
  • FreeBSD (500 GiB)
  • NetBSD (340 GiB)

Implementation Details

The mirroring is done by one of two scripts. Both are based on anonftpsync. Various cronjobs are scheduled to call one of these scripts.

csc-sync-debian

This is used to sync debian-style repositories. It's usage is:

csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

csc-sync-standard

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

Crontab

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m  h     dom mon dow command

#
# bi-hourly
#
  5  */2   *   *   *   ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
  5  */2   *   *   *   ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
  5  */2   *   *   *   ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
  5  */2   *   *   *   ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
  5  */2   *   *   *   ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
  5  3,15  *  *   *   ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
  5  3,15  *  *   *   ~/bin/csc-sync-standard nexenta nexenta.org repository
  5  3,15  *  *   *   ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
  5  3,15  *  *   *   ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
  5  3,15  *  *   *   ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
  5  3,15  *  *   *   ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
  5  3,15  *  *   *   ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
  5  4,16  *  *   *   ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
  5  4,16  *  *   *   ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
  5  4,16  *  *   *   ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
  5  4,16  *  *   *   ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
  5  4,16  *  *   *   ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
  5  4,16  *  *   *   ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
  5  4.16  *  *   *   ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
  5  5,17  *  *   *   ~/bin/csc-sync-standard mysql mysql.he.net mysql
  5  5,17  *  *   *   ~/bin/csc-sync-standard kde master.kde.org kdeftp
  5  5,17  *  *   *   ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
  5  5,17  *  *   *   ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
  5  5,17  *  *   *   ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
  5  5,17  *  *   *   ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
  5  5,17  *  *   *   ~/bin/csc-sync-debian-cd
  5  6,18  *  *   *   ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
  5  6,18  *  *   *   ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
  5  6,18  *  *   *   ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
  5  6,18  *  *   *   ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
  5  6,18  *  *   *   ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
  29 */4   *  *   *   RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
  15,45 *  *  *   *   ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
  5,35  *  *  *   *   ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

HTTP

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
    ServerLimit          32
    ThreadLimit          64
    StartServers          2
    MaxClients         2048
    MinSpareThreads      16
    MaxSpareThreads      48
    ThreadsPerChild      64
    MaxRequestsPerChild   0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

FTP

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS           off
IdentLookups            off

We also limit the amount of CPU/memory resources used (e.g. to minimize Globbing resources):

RLimitCPU               session 10
RLimitMemory            session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances            500
MaxClients              500

Rsync

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.