Mirror: Difference between revisions

From CSCWiki
Jump to navigation Jump to search
(crontab -> merlin)
Line 126: Line 126:


We use Apache as our web server. Here's a snippet of the worker configuration:
We use Apache as our web server. Here's a snippet of the worker configuration:
<IfModule mpm_worker_module>

ServerLimit 64
<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
ThreadLimit 64
StartServers 2
StartServers 2
MaxClients 4096
MaxClients 2048
MinSpareThreads 16
MinSpareThreads 16
MaxSpareThreads 48
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
ThreadsPerChild 64
</IfModule>
MaxRequestsPerChild 0
</IfModule>


We use the bwbar application to display current bandwidth in the footer of mirror pages.
We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.


=== FTP ===
=== FTP ===

Revision as of 08:05, 25 February 2010

We currently run a public mirror (mirror.csclub.uwaterloo.ca) on sodium-benzoate. We are listed on the ResNet "don't count" list so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a bandwidth graph you can look at.

Archives Mirrored

Total Size: 4.4 TiB

Archive Upstream/Listing Size Notes
Apache rsync://rsync.us.apache.org/apache-dist/
http://www.apache.org/mirrors/
23 GiB -
Arch Linux rsync://mirror.rit.edu/archlinux
http://www.archlinux.org/download/
http://wiki.archlinux.org/index.php/Mirrors#Mirror_List
75 GiB -
Blastwave rsync://www.ibiblio.org/sun-packages/csw/
http://www.blastwave.org/mirrors.php
11 GiB -
CentOS rsync://us-msync.centos.org/CentOS/
http://mirror-status.centos.org/
131 GiB -
CPAN rsync://rsync.nic.funet.fi/CPAN/
http://mirror.cpan.org/
6.8 GiB -
CRAN rsync://cran.r-project.org/CRAN/ 41 GiB We should get added as an official mirror.
CTAN rsync://carroll.aset.psu.edu/ctan/
http://www.dante.de/cgi-bin/ctan/list.cgi
http://www.dante.de/mirmon/
16 GiB -
Cygwin rsync://cygwin.com/cygwin-ftp/
http://www.cygwin.com/mirrors.html
11 GiB -
Damn Small Linux rsync://ftp.belnet.be/packages/damnsmalllinux/ 18 GiB Sent a request to be added. Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
Debian rsync://debian.mirror.rafal.ca/debian/
http://www.debian.org/mirror/list
http://www.de.debian.org/dmc/today/
446 GiB Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
Debian-backports rsync://www.backports.org/backports.org/
http://www.backports.org/debian/README.mirrors.html
30 GiB Submitted a request to be added to the mirror list long ago with no response.
Debian-cd rsync://cdimage.debian.org/debian-cd/ 70 GiB Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
Debian-multimedia rsync://www.debian-multimedia.org/debian/
http://www.debian-multimedia.org/debian-m.php
5.3 GiB -
Debian-ports rsync://ftp.debian-ports.org/debian/ 124G -
Debian-security rsync://security.debian.org/debian-security/ 36 GiB Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
Debian-unofficial rsync://ftp.debian-unofficial.org/debian/
http://www.debian-unofficial.org/mirrors.html
970 MiB -
Debian-volatile rsync://volatile.debian.org/debian-volatile/
http://www.debian.org/volatile/volatile-mirrors
3.3 GiB -
Eclipse rsync://download.eclipse.org/eclipseMirror
http://www.eclipse.org/downloads/download.php?file=/
226 GiB -
Emdebian rsync://www.emdebian.org/emdebian/ 3.7 GiB Emdebian does not currently list mirrors.
Fedora http://fedoraproject.org/wiki/Infrastructure/Mirroring
http://mirrors.fedoraproject.org/publiclist
https://admin.fedoraproject.org/mirrormanager/site/647
rsync://ftp.muug.mb.ca/pub/fedora/linux/core/
548 GiB -
FreeBSD rsync://ftp1.ca.freebsd.org/ 673 GiB We should get added as an official mirror.
Gentoo (portage) rsync://rsync1.us.gentoo.org/gentoo-portage/ 581 MiB We are rsync4.ca.gentoo.org.
Gentoo (sources) rsync://masterdistfiles.gentoo.org/gentoo/
http://www.gentoo.org/main/en/mirrors.xml
http://mirrorstats.gentoo.org/
159 GiB See ~sysadmin/passwords/gentoo for rsync password.
GNOME rsync://ftp.gnome.org/gnome/ 86 GiB -
GNU rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/
http://www.gnu.org/order/ftp.html
20 GiB -
KDE rsync://master.kde.org/kdeftp/
http://www.kde.org/mirrors/
69 GiB -
kernel.org rsync://kernel.org/pub/linux/
rsync://kernel.org/pub/software/
http://kernel.org/mirrors/countries/html/CA.html
130 GiB -
Linux Mint (releases) rsync://ftp.heanet.ie/pub/linuxmint.com/
http://www.linuxmint.com/mirrors.php
23 GiB -
Linux Mint (packages) rsync://packages.linuxmint.com/packages/
http://www.linuxmint.com/mirrors.php
4.1 GiB -
mozdev rsync://rsync.mozdev.org/mozdev/
http://mirrors.mozdev.org/index.html
6.0 GiB Currently in the North American rotation, but could request to be added to the global rotation.
mozilla.org rsync://releases-rsync.mozilla.org/mozilla-releases/
http://www.mozilla.org/mirrors.html
105 GiB -
MySQL rsync://mysql.mirrors.pair.com/mysql/
http://dev.mysql.com/downloads/mirrors.html
148 GiB -
non-GNU rsync://dl.sv.gnu.org/releases/
http://dl.sv.gnu.org/releases/00_MIRRORS.html
http://download.savannah.gnu.org/mirmon/
15 GiB -
Openoffice (extended set) rsync://rsync.services.openoffice.org/openoffice-extended/
http://distribution.openoffice.org/mirrors/#mirrors
http://www.ooodev.org/mirmon/
102 GiB -
OpenSUSE (opensuse-full) rsync://stage.opensuse.org/opensuse-full/opensuse/
http://mirrors.opensuse.org/list/all.html
161 GiB -
Slackware - 139 GiB We should get added as an official mirror.
Ubuntu rsync://archives.ubuntu.com/ubuntu/
https://launchpad.net/ubuntu/+archivemirrors
286 GiB We used to be the official Canadian mirror (i.e., ca.archive.ubuntu.com); when we get more bandwidth get us added back.
Ubuntu-ports rsync://ports.ubuntu.com/ubuntu-ports/ 394 GiB -
Ubuntu-ports-releases rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ 36 GiB -
Ubuntu-releases rsync://releases.ubuntu.com/releases/
https://launchpad.net/ubuntu/+cdmirrors
http://www.ubuntu.com/getubuntu/download
http://www.kubuntu.org/download.php
http://www.edubuntu.org/Download
39 GiB Are the official Canadian mirror (i.e., ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
xorg.freedesktop.org rsync://xorg.freedesktop.org/xorg-archive/
http://www.x.org/wiki/Releases/Download
5.3 GiB -
Xubuntu-releases rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/
http://www.xubuntu.org/get
18 GiB -
wine-budgetdedicated http://wine.budgetdedicated.com/apt/ 151 MiB -

Proposed Archives to Mirror

  • Mandriva (774 GiB)
  • OpenBSD (209 GiB)
  • NetBSD (340 GiB)
  • PCLinuxOS

Implementation Details

The mirroring is done by one of two scripts. Both are based on anonftpsync. Various cronjobs are scheduled to call one of these scripts.

csc-sync-debian

This is used to sync debian-style repositories. It's usage is:

csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

csc-sync-standard

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

merlin

The synchronization process is run by a Python script called "merlin", ostensibly written by mspang, stored in ~mirror/merlin. The repository list, sync time, etc. is maintained in merlin.py.

HTTP

We use Apache as our web server. Here's a snippet of the worker configuration: <IfModule mpm_worker_module>

   ServerLimit          64
   ThreadLimit          64
   StartServers          2
   MaxClients         4096
   MinSpareThreads      16
   MaxSpareThreads      48
   ThreadsPerChild      64
   MaxRequestsPerChild   0

</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

FTP

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS           off
IdentLookups            off

We also limit the amount of CPU/memory resources used (e.g. to minimize Globbing resources):

RLimitCPU               session 10
RLimitMemory            session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances            500
MaxClients              500

Rsync

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.