Mirror

From CSCWiki
Revision as of 17:56, 6 December 2009 by Jbroman (talk | contribs) (fixed typo in crontab)
Jump to navigation Jump to search

We currently run a public mirror (mirror.csclub.uwaterloo.ca) on sodium-benzoate. We are listed on the ResNet "don't count" list so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a bandwidth graph you can look at.

Archives Mirrored

Total Size: 4.4 TiB

Archive Upstream/Listing Size Notes
Apache rsync://rsync.us.apache.org/apache-dist/
http://www.apache.org/mirrors/
23 GiB -
Arch Linux rsync://mirror.rit.edu/archlinux
http://www.archlinux.org/download/
http://wiki.archlinux.org/index.php/Mirrors#Mirror_List
75 GiB -
Blastwave rsync://www.ibiblio.org/sun-packages/csw/
http://www.blastwave.org/mirrors.php
11 GiB -
CentOS rsync://us-msync.centos.org/CentOS/
http://mirror-status.centos.org/
131 GiB -
CPAN rsync://rsync.nic.funet.fi/CPAN/
http://mirror.cpan.org/
6.8 GiB -
CRAN rsync://cran.r-project.org/CRAN/ 41 GiB We should get added as an official mirror.
CTAN rsync://carroll.aset.psu.edu/ctan/
http://www.dante.de/cgi-bin/ctan/list.cgi
http://www.dante.de/mirmon/
16 GiB -
Cygwin rsync://cygwin.com/cygwin-ftp/
http://www.cygwin.com/mirrors.html
11 GiB -
Damn Small Linux rsync://ftp.belnet.be/packages/damnsmalllinux/ 18 GiB Sent a request to be added. Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
Debian rsync://debian.mirror.rafal.ca/debian/
http://www.debian.org/mirror/list
http://www.de.debian.org/dmc/today/
446 GiB Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
Debian-backports rsync://www.backports.org/backports.org/
http://www.backports.org/debian/README.mirrors.html
30 GiB Submitted a request to be added to the mirror list long ago with no response.
Debian-cd rsync://cdimage.debian.org/debian-cd/ 70 GiB Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
Debian-multimedia rsync://www.debian-multimedia.org/debian/
http://www.debian-multimedia.org/debian-m.php
5.3 GiB -
Debian-ports rsync://ftp.debian-ports.org/debian/ 124G -
Debian-security rsync://security.debian.org/debian-security/ 36 GiB Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
Debian-unofficial rsync://ftp.debian-unofficial.org/debian/
http://www.debian-unofficial.org/mirrors.html
970 MiB -
Debian-volatile rsync://volatile.debian.org/debian-volatile/
http://www.debian.org/volatile/volatile-mirrors
3.3 GiB -
Eclipse rsync://download.eclipse.org/eclipseMirror
http://www.eclipse.org/downloads/download.php?file=/
226 GiB -
Emdebian rsync://www.emdebian.org/emdebian/ 3.7 GiB Emdebian does not currently list mirrors.
Fedora http://mirrors.fedoraproject.org/publiclist
https://admin.fedoraproject.org/mirrormanager/site/647
rsync://ftp.muug.mb.ca/pub/fedora/linux/core/
548 GiB -
FreeBSD rsync://ftp1.ca.freebsd.org/ 673 GiB We should get added as an official mirror.
Gentoo (portage) rsync://rsync1.us.gentoo.org/gentoo-portage/ 581 MiB We are rsync4.ca.gentoo.org.
Gentoo (sources) rsync://masterdistfiles.gentoo.org/gentoo/
http://www.gentoo.org/main/en/mirrors.xml
http://mirrorstats.gentoo.org/
159 GiB See ~sysadmin/passwords/gentoo for rsync password.
GNOME rsync://ftp.gnome.org/gnome/ 86 GiB -
GNU rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/
http://www.gnu.org/order/ftp.html
20 GiB -
KDE rsync://master.kde.org/kdeftp/
http://www.kde.org/mirrors/
69 GiB -
kernel.org rsync://kernel.org/pub/linux/
rsync://kernel.org/pub/software/
http://kernel.org/mirrors/countries/html/CA.html
130 GiB -
Linux Mint (releases) rsync://ftp.heanet.ie/pub/linuxmint.com/
http://www.linuxmint.com/mirrors.php
23 GiB -
Linux Mint (packages) rsync://packages.linuxmint.com/packages/
http://www.linuxmint.com/mirrors.php
4.1 GiB -
mozdev rsync://rsync.mozdev.org/mozdev/
http://mirrors.mozdev.org/index.html
6.0 GiB Currently in the North American rotation, but could request to be added to the global rotation.
mozilla.org rsync://releases-rsync.mozilla.org/mozilla-releases/
http://www.mozilla.org/mirrors.html
105 GiB -
MySQL rsync://mysql.mirrors.pair.com/mysql/
http://dev.mysql.com/downloads/mirrors.html
148 GiB -
non-GNU rsync://dl.sv.gnu.org/releases/
http://dl.sv.gnu.org/releases/00_MIRRORS.html
http://download.savannah.gnu.org/mirmon/
15 GiB -
Openoffice (extended set) rsync://rsync.services.openoffice.org/openoffice-extended/
http://distribution.openoffice.org/mirrors/#mirrors
http://www.ooodev.org/mirmon/
102 GiB -
OpenSUSE (opensuse-full) rsync://stage.opensuse.org/opensuse-full/opensuse/
http://mirrors.opensuse.org/list/all.html
161 GiB -
Slackware - 139 GiB We should get added as an official mirror.
Ubuntu rsync://archives.ubuntu.com/ubuntu/
https://launchpad.net/ubuntu/+archivemirrors
286 GiB We used to be the official Canadian mirror (i.e., ca.archive.ubuntu.com); when we get more bandwidth get us added back.
Ubuntu-ports rsync://ports.ubuntu.com/ubuntu-ports/ 394 GiB -
Ubuntu-ports-releases rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ 36 GiB -
Ubuntu-releases rsync://releases.ubuntu.com/releases/
https://launchpad.net/ubuntu/+cdmirrors
http://www.ubuntu.com/getubuntu/download
http://www.kubuntu.org/download.php
http://www.edubuntu.org/Download
39 GiB Are the official Canadian mirror (i.e., ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
xorg.freedesktop.org rsync://xorg.freedesktop.org/xorg-archive/
http://www.x.org/wiki/Releases/Download
5.3 GiB -
Xubuntu-releases rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/
http://www.xubuntu.org/get
18 GiB -
wine-budgetdedicated http://wine.budgetdedicated.com/apt/ 151 MiB -

Proposed Archives to Mirror

  • Mandriva (774 GiB)
  • OpenBSD (209 GiB)
  • NetBSD (340 GiB)
  • PCLinuxOS

Implementation Details

The mirroring is done by one of two scripts. Both are based on anonftpsync. Various cronjobs are scheduled to call one of these scripts.

csc-sync-debian

This is used to sync debian-style repositories. It's usage is:

csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

csc-sync-standard

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

Crontab

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m  h     dom mon dow command

# make torrents
  */10 *  *   *   *     /home/mirror/bin/make-torrents > /dev/null 2> /dev/null

#
# bi-hourly
#
  5  */2   *   *   *   ~/bin/csc-sync-debian debian debian.mirror.rafal.ca debian ftp-master.debian.org
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-volatile volatile-master.debian.org debian-volatile volatile-master.debian.org
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
  5  */2   *   *   *   ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
  5  */2   *   *   *   ~/bin/csc-sync-standard ubuntu-releases rsync.releases.ubuntu.com releases
  5  */2   *   *   *   ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
  5  */2   *   *   *   ~/bin/csc-sync-debian ubuntu-ports ports.ubuntu.com ubuntu-ports	drescher.canonical.com
  5  */2   *   *   *   ~/bin/csc-sync-debian linuxmint-packages packages.linuxmint.com packages

#
# daily
#
  5  3,15  *  *   *   ~/bin/csc-sync-debian emdebian www.emdebian.org debian
  5  3,15  *  *   *   ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
  5  3,15  *  *   *   ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
  5  3,15  *  *   *   ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
  5  3,15  *  *   *   ~/bin/csc-sync-standard openoffice rsync.services.openoffice.org openoffice-extended
#  5  3,15  *  *   *   ~/bin/csc-sync-standard fedora/epel fedora-archives.ibiblio.org fedora-epel && ~/bin/report_mirror >/dev/null
  5  4,16  *  *   *   ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
  5  4,16  *  *   *   ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
  5  4,16  *  *   *   ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases --ignore-errors
  5  4,16  *  *   *   ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
  5  4,16  *  *   *   ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
  5  4,16  *  *   *   ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
  5  4.16  *  *   *   ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
  5  5,17  *  *   *   ~/bin/csc-sync-standard mysql mysql.he.net mysql
  5  5,17  *  *   *   ~/bin/csc-sync-standard kde master.kde.org kdeftp
  5  5,17  *  *   *   ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
  5  5,17  *  *   *   ~/bin/csc-sync-standard blastwave master.rsync.blastwave.org blastwave
  5  5,17  *  *   *   ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
  5  5,17  *  *   *   ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian --ignore-errors
  5  5,17  *  *   *   ~/bin/csc-sync-standard slackware slackware.cs.utah.edu slackware
  5  5,17  *  *   *   ~/bin/csc-sync-debian-cd
  5  6,18  *  *   *   ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
  5  6,18  *  *   *   ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
  5  6,18  *  *   *   ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
  5  6,18  *  *   *   ~/bin/csc-sync-standard opensuse stage.opensuse.org opensuse-full/opensuse/ #"--exclude distribution/.timestamp_invisible"
  5  6,18  *  *   *   ~/bin/csc-sync-standard damnsmalllinux ftp.heanet.ie mirrors/damnsmalllinux.org/
  5  7,19  *  *   *   ~/bin/csc-sync-standard FreeBSD ftp1.ca.freebsd.org freebsd
#  5  7,19  *  *   *   ~/bin/csc-sync-standard fedora/linux fedora-archives.ibiblio.org fedora-enchilada/linux/ --ignore-errors && ~/bin/report_mirror >/dev/null
  5  7,19  *  *   *   ~/bin/csc-sync-standard linuxmint ftp.heanet.ie pub/linuxmint.com/
  5  7,19  *  *   *   ~/bin/csc-sync-standard ubuntu-ports-releases cdimage.ubuntu.com cdimage/ports/releases/

#
# other
#
  29 */4   *  *   *   RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
  15,45 *  *  *   *   ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
  5,35 *   *  *   *   ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

HTTP

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
    ServerLimit          32
    ThreadLimit          64
    StartServers          2
    MaxClients         2048
    MinSpareThreads      16
    MaxSpareThreads      48
    ThreadsPerChild      64
    MaxRequestsPerChild   0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

FTP

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS           off
IdentLookups            off

We also limit the amount of CPU/memory resources used (e.g. to minimize Globbing resources):

RLimitCPU               session 10
RLimitMemory            session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances            500
MaxClients              500

Rsync

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.