Mirror: Difference between revisions

From CSCWiki
Jump to navigation Jump to search
No edit summary
Line 97: Line 97:


== Monitoring ==
== Monitoring ==
[http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440]
[http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 Bandwidth graph]


== Implementation Details ==
== Implementation Details ==

Revision as of 03:24, 13 August 2009

We currently run a public mirror (mirror.csclub.uwaterloo.ca) on sodium-benzoate. We are listed on the ResNet "don't count" list so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.

Archives Mirrored

Total Size: 4.5 TiB

Archive Upstream/Listing Size Notes
Apache rsync://rsync.us.apache.org/apache-dist/
http://www.apache.org/mirrors/
20 GiB -
Arch Linux rsync://mirror.rit.edu/archlinux
http://www.archlinux.org/download/
http://wiki.archlinux.org/index.php/Mirrors#Mirror_List
41 GiB -
Blastwave rsync://www.ibiblio.org/sun-packages/csw/
http://www.blastwave.org/mirrors.php
10 GiB -
CentOS rsync://us-msync.centos.org/CentOS/
http://mirror-status.centos.org/
125 GiB -
CPAN rsync://rsync.nic.funet.fi/CPAN/
http://mirror.cpan.org/
6 GiB -
CRAN rsync://cran.r-project.org/CRAN/ 36 GiB We should get added as an official mirror.
CTAN rsync://carroll.aset.psu.edu/ctan/
http://www.dante.de/cgi-bin/ctan/list.cgi
http://www.dante.de/mirmon/
16 GiB -
Cygwin rsync://cygwin.com/cygwin-ftp/
http://www.cygwin.com/mirrors.html
10 GiB -
Damn Small Linux rsync://ftp.belnet.be/packages/damnsmalllinux/ 18 GiB We should get added as an official mirror. Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
Debian rsync://gulus.usherbrooke.ca/debian/
http://www.debian.org/mirror/list
http://www.de.debian.org/dmc/today/
410 GiB Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
Debian-backports rsync://www.backports.org/backports.org/
http://www.backports.org/debian/README.mirrors.html
28 GiB Submitted a request to be added to the mirror list.
Debian-cd rsync://cdimage.debian.org/debian-cd/ 68 GiB Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
Debian-multimedia rsync://www.debian-multimedia.org/debian/
http://www.debian-multimedia.org/debian-m.php
5 GiB -
Debian-ports rsync://ftp.debian-ports.org/debian/ 124G -
Debian-security rsync://security.debian.org/debian-security/ 29 GiB Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
Debian-unofficial rsync://ftp.debian-unofficial.org/debian/
http://www.debian-unofficial.org/mirrors.html
1 GiB -
Debian-volatile rsync://volatile.debian.org/debian-volatile/
http://www.debian.org/volatile/volatile-mirrors
4 GiB -
Eclipse rsync://download.eclipse.org/eclipseMirror
http://www.eclipse.org/downloads/download.php?file=/
234 GiB -
Emdebian rsync://www.emdebian.org/emdebian/ 3 GiB Emdebian does not currently list mirrors.
Fedora rsync://archive.linux.duke.edu/fedora-enchilada/ 721 GiB We should get added as an official mirror.
FreeBSD rsync://ftp1.ca.freebsd.org/ 718 GiB We should get added as an official mirror.
Gentoo (portage) rsync://rsync1.us.gentoo.org/gentoo-portage/ 1 GiB We are rsync4.ca.gentoo.org.
Gentoo (sources) rsync://masterdistfiles.gentoo.org/gentoo/
http://www.gentoo.org/main/en/mirrors.xml
http://mirrorstats.gentoo.org/
92 GiB See ~sysadmin/passwords/gentoo for rsync password.
GNOME rsync://ftp.gnome.org/gnome/ 65 GiB -
GNU rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/
http://www.gnu.org/order/ftp.html
14 GiB -
Gobuntu-releases rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ 7 GiB Canonical does not currently list mirrors.
KDE rsync://master.kde.org/kdeftp/
http://www.kde.org/mirrors/
65 GiB -
kernel.org rsync://kernel.org/pub/linux/
rsync://kernel.org/pub/software/
http://kernel.org/mirrors/countries/html/CA.html
82 GiB -
Linux Mint rsync://ftp.heanet.ie/pub/linuxmint.com/ 21 GiB We should get added as an official mirror.
mozdev rsync://rsync.mozdev.org/mozdev/
http://mirrors.mozdev.org/index.html
5 GiB Currently in the North American rotation, but could request to be added to the global rotation.
mozilla.org rsync://releases-rsync.mozilla.org/mozilla-releases/
http://www.mozilla.org/mirrors.html
95 GiB -
MySQL rsync://mysql.mirrors.pair.com/mysql/
http://dev.mysql.com/downloads/mirrors.html
76 GiB -
non-GNU rsync://dl.sv.gnu.org/releases/
http://dl.sv.gnu.org/releases/00_MIRRORS.html
http://download.savannah.gnu.org/mirmon/
8 GiB -
Openoffice (extended set) rsync://rsync.services.openoffice.org/openoffice-extended/
http://distribution.openoffice.org/mirrors/#mirrors
http://www.ooodev.org/mirmon/
136 GiB -
OpenSUSE (opensuse-full) rsync://suse.mirrors.tds.net/opensuse-full/opensuse/
http://mirrors.opensuse.org/list/11.1.html
230 GiB We are no longer on the download list - find out why.
Slackware - 27 GiB -
Ubuntu rsync://archives.ubuntu.com/ubuntu/
https://launchpad.net/ubuntu/+archivemirrors
261 GiB Are the official Canadian mirror (i.e., ca.archive.ubuntu.com)
Ubuntu-releases rsync://releases.ubuntu.com/releases/
https://launchpad.net/ubuntu/+cdmirrors
http://www.ubuntu.com/getubuntu/download
http://www.kubuntu.org/download.php
http://www.edubuntu.org/Download
37 GiB Are the official Canadian mirror (i.e., ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
xorg.freedesktop.org rsync://xorg.freedesktop.org/xorg-archive/
http://www.x.org/wiki/Releases/Download
5 GiB -
Xubuntu-releases rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/
http://www.xubuntu.org/get
20 GiB -

Propsed Archives to Mirror

  • Mandriva (774 GiB)
  • OpenBSD (54 GiB)
  • NetBSD (340 GiB)
  • PCLinuxOS

Monitoring

Bandwidth graph

Implementation Details

The mirroring is done by one of two scripts. Both are based on anonftpsync. Various cronjobs are scheduled to call one of these scripts.

csc-sync-debian

This is used to sync debian-style repositories. It's usage is:

csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

csc-sync-standard

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

Crontab

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m  h     dom mon dow command

#
# bi-hourly
#
  5  */2   *   *   *   ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
  5  */2   *   *   *   ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
  5  */2   *   *   *   ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
  5  */2   *   *   *   ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
  5  */2   *   *   *   ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
  5  */2   *   *   *   ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
  5  3,15  *  *   *   ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
  5  3,15  *  *   *   ~/bin/csc-sync-standard nexenta nexenta.org repository
  5  3,15  *  *   *   ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
  5  3,15  *  *   *   ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
  5  3,15  *  *   *   ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
  5  3,15  *  *   *   ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
  5  3,15  *  *   *   ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
  5  4,16  *  *   *   ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
  5  4,16  *  *   *   ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
  5  4,16  *  *   *   ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
  5  4,16  *  *   *   ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
  5  4,16  *  *   *   ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
  5  4,16  *  *   *   ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
  5  4.16  *  *   *   ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
  5  5,17  *  *   *   ~/bin/csc-sync-standard mysql mysql.he.net mysql
  5  5,17  *  *   *   ~/bin/csc-sync-standard kde master.kde.org kdeftp
  5  5,17  *  *   *   ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
  5  5,17  *  *   *   ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
  5  5,17  *  *   *   ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
  5  5,17  *  *   *   ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
  5  5,17  *  *   *   ~/bin/csc-sync-debian-cd
  5  6,18  *  *   *   ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
  5  6,18  *  *   *   ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
  5  6,18  *  *   *   ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
  5  6,18  *  *   *   ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
  5  6,18  *  *   *   ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
  29 */4   *  *   *   RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
  15,45 *  *  *   *   ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
  5,35  *  *  *   *   ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

HTTP

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
    ServerLimit          32
    ThreadLimit          64
    StartServers          2
    MaxClients         2048
    MinSpareThreads      16
    MaxSpareThreads      48
    ThreadsPerChild      64
    MaxRequestsPerChild   0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

FTP

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS           off
IdentLookups            off

We also limit the amount of CPU/memory resources used (e.g. to minimize Globbing resources):

RLimitCPU               session 10
RLimitMemory            session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances            500
MaxClients              500

Rsync

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.