CSCWiki - User contributions [en]

Past Executive

2010-08-20T01:37:51Z

Dtbartle: /* Winter */

__NOTOC__
= Definitions =
#define PR President
#define VP Vice-president
#define TR Treasurer
#define SE Secretary
#define SA Sysadmin
#define OF Office Manager
#define LI Librarian
#define FL Flasher
#define DE Diety
#define WW Webmaster
#define SE-TR Secretary-Treasurer (Position was split)

=Founding 1967-1968=

Sponsor - J. Peter Sprung
PR: K. Rugger
VP: R. Jaques
SE-TR: G. Sutherland

Founding Members:
B. Kindree
R. Melen
V. Neglia
R. Charney
R. Truman
Glenn Berry
D. Meek

=1968-1969=

SE-TR: Glenn Berry

=1969-1970=

Unknown, only one letter found in the folder 'ACM History' addressed to Glenn Berry, which makes it likely that he was SE-TR once again. May be indicated in membership lists. The club appears to have died this academic year.

===A note on ACM affiliation===

The first attempt at joining the ACM was started with an informal inquiry Dec 5, 1967. This lead to a series of constitution edits (working towards affiliation) in Winter 1968. There was a break for the spring (no correspondence found, I presume we were waiting on a reply). In the fall records indicate that our constitution and chartering was rejected, further correspondence was sent in Fall 1968 by Glenn Berry. A new inquiry, seemingly unaware of the first was sent Dec 7, 1970

=1971=

James W. Welch is the only person sending letters for the club. Later signs them Vice-President. Continues this for most of the year. Either he was VP throughout 3 terms or we were on calendar year elections then.

=1972=

It appears we visited Western and Western visited us this year (there is some reference to a similar occurrence the year previous). Documents from 1973 indicate a termly exec structure, this probably goes back to 1972.

===Winter===

PR: Mike Campbell
SE: Doug Lacy

there is also stuff from James W. Welch without a position. Considering his work in the founding and that the letters are speaker invitations he is, with high probability, VP this term.

===Spring===

===Fall===

PR: Ian McIntosh

=1973=

Faculty Sponsor: M. Gentleman

===Winter===
===Spring===

Either secretary or VP: Jim Parry (sent the only letter on file, but it is a speaker invite, did not sign his position).

===Fall===

PR: Jim Parry
VP: Ray Walden
TR: Slavko Stemberger
SE: Mario Festival

=1974-1988=

James B- was president in Winter 88 (signature hard to read).
Tim Timar - cc'd on memos/mentioned on mathsoc minutes in 1987/88.
The Sysadmin and Office Manager positions seem to have been created somewhere in here. The 'Record Management Profile' that Robyn Stewart did as an assignment in 1991-1992 for some class at UBC
indicates the existence of both positions. As we acquired an HP-9000 in 1987 and this is likely our first computer (previously we only had a terminal?), the sysadmin was created then.

=1989=
===Winter===

PR: Kevin W. Smith ?? (derived from a signature, may be a bad reading)
VP: Angela Chambers
TR:
SE: Sean Goggin

===Spring===
===Fall===

PR: Sean Goggin

=1990=
===Winter===
===Spring===
===Fall===
PR: Wade Richards
Kivi Shapiro - attended mathsoc meeting on our behalf. - Censured by mathsoc for his actions during the election.
Shannon Mann - attended mathsoc meeting on our behalf.
Robyn Stewart - attended mathsoc meeting on our behalf.

=1991=
===Winter===
Shannon Mann - attended mathsoc meeting on our behalf.
Ed Bourne - attended mathsoc meeting on our behalf.
===Spring===
Jason Knell - attended mathsoc meeting on our and PMC's behalf.
===Fall===
Wiktor Wiewiorowski - attended mathsoc meeting on our behalf this term.

=1995=
===Winter===
===Spring===
===Fall===
PR: Amy Brown (arbrown)
VP: Christina Norman (cbnorman)
TR: Steven Mills (samills)
SE: Allison Graham (akgraham)
SA: Gavin Peters

=1996=
===Winter===
PR: Nikita Borisov (nborisov)
VP: Joseph Deu Ngoc (dtdeungo)
TR: Stephen Mills (samills)
SE: Sharlene Schmeichel (saschmei)
SA: Dave Brown (dagbrown)
OF: Somsack Tsai (stsai)
LI: Devin Carless (dccarles)
FL: Allyson Graham (akgraham)
DE: Ian Goldberg (iagoldbe)

===Spring===
PR: Blake Winton (bwinton)
VP: Nick Harvey (njaharve)
TR: Nikita Borisov (nborisov)
SE: Viet-Trung Luu (vluu)
SA: Drew Hamilton (awhamilt)
OF: Jillian Arnott (jarnott)
LI: Ross Ridge (rridge)
FL: Devin Carless (dccarles)

=== Fall ===
PR: Shannon Mann (sjbmann)
VP: Joe "Frosh" Deu Ngoc (jtdeungo) resigned (heavy workload)
TR: Michal Van Biesbrouck (mlvanbie)
SE: Nikita Borisov (nborisov)
SA: Chris Rovers
OF: Dax Hutcheon (ddhutche)
LI: Aliz Csenki (acsenki)
FL: Aaron Chmielowiec (archmiel)
DE: Skuld (no uwuserid yet...)

=1997 =

===Winter===
PR:Dima Brodsky
VP: Nikita Borisov
TR: Stephen Mills
SE: ejones
SA: Alex Brodsky
OF: Chris Doherty
LI: Matt Corks
FL: Paul Prescod

=== Fall ===
PR: Chris Rovers (cdrovers)
VP: Viet-Trung Luu (vluu)
TR: Somsack Tsai (stsai)
SE: Matt Corks (mvcorks)

= 1998 =
=== Winter ===
PR: Suresh Naidu
VP: Viet-Trung Luu
TR: Tim Coleman
SE: Dax Hutcheon
Librarian: Dax Hutcheon
Flasher: Dax Hutcheon
Webmaster: Dax Hutcheon
SA:
OF: Aaron Chmielowiec

=== Spring ===

Position Name You might call them...
President roconnor Russell O'Connor
Vice-president trwcolem Tim Coleman
Treasurer knzarysk Karl Zaryski
Secretary (bwinton) (Blake Winton)
Sysadmin wbiggs Billy Biggs
Librarian snaidu Suresh Naidu
Flasher pechrysl Paul Chrysler
Office Manager dccarles Devin Carless
WWWW trwcolem Tim Coleman

=== Fall ===

President Joe Deu Ngoc jtdeungo
Vice-President Wai Ling Yee wlyee
Treasurer Fjord j2lynn
Secretary Matt Corks mvcorks
Sysadmin Andrew Hamilton awhamilt

World Wide Web Wench Dax Hutcheon ddhutche
Office Manager Richard Bell rlbell
Librarian Damian Gryski dgryski
Flasher Paul Chrysler pechrysl
Official Deity Ian Goldberg iagoldbe
Official Chairbeing Calum T. Dalek calum

=1999=

=== Winter ===

=== Spring ===

=== Fall ===

=2000=
=== Winter ===

=== Spring ===

=== Fall ===

=2001=
=== Winter ===

=== Spring ===

=== Fall ===

=2002=
http://www.mathnews.uwaterloo.ca/Issues/mn8902/cscflash.php
=== Winter ===
PR: Billy Biggs
VP: Stefanus Du Toit
TR: Melissa Basinger
SE: James Perry
SA: Barry Genova
Librarian: Ryan Golbeck
Webmaster: Jonathan Beverley
Office Manager: Sayan Li

=== Spring ===
PR: Alex Pop
VP: Melissa Basinger
TR: Siyan Li
SE: James A Morrison
SA: Jonathan Beverley
Webmaster: Stefanus Du Toit

=== Fall ===
PR: James A. Morrison
VP: Stefanus Du Toit
TR: James Perry
SE: Michael Biggs
SA: Ryan Golbeck
Librarian: Mark Sherry, Cassandra Schopf
Webmaster: Stefanus Du Toit

=2003=
=== Winter ===
PR: Kannan Vijayan
VP: Meg Darragh
TR: James Perry
SE: Wojtek Kosnik
SA: Stefanus Du Toit
Librarian: Simon Law
Webmaster: Julie Lavoie

=== Spring===
===Fall===
PR: sjdutoit
VP: m3darrag
TR: tmyklebu
SE: jeperry
SA: sfllaw
OF:

=2004=
===Winter===
PR: sfllaw
VP: fspacek
TR: ljain
SE: jlavoie
SA: tmyklebu
OF:

===Spring===
PR: dnmorton ?
VP: tloach
TR: mbiggs
SE: lanortha
SA:
OF:

===Fall ===
PR: jeperry
VP: mtsay
TR: msherry
SE: tmyklebu
SA: jlavoie
OF:

=2005=
===Winter===

PR: mtsay
VP: lanortha
TR: hkarau
SE: domorton
SA: tmyklebu
OF:

===Spring===

PR: msherry
VP: mdkess
TR: apiccon
SE: mbiggs
SA: tmyklebu
OF:

===Fall===

PR: Tim Loach
VP: Lesley Northam
TR: Caelyn McAulay
SE: The Professor
SA: hkarau
OF:

=2006=

===Winter===

PR: tmyklebu
VP: mdruker
TR:
SE:
SA:
OF:

===Spring===
PR: dtbartle
VP:
TR:
SE:
SA:
OF:

===Fall===

PR: mdkess
VP: mdsherry
TR: slmably
SE: cmcaulay
SA: woconnor

=2007=

===Winter===
PR: dtbartle
VP: dbelange
TR: cmcaulay
SE: dtenty
SA: hkarau
Webmaster: jnopporn
OF:

===Spring===
PR: gdcosta
VP: kmlarose
TR: kspaans
SE: elouie
SA: mspang
Librarian: daltenty
OF:

===Fall ===
PR: hkarau
VP: amccausl
TR: dchlobow
SE: sgcummin
SA:
OF:

=2008=
===Winter ===
PR: sgcummin
VP: m3lawren
TR: mtarkows
SE: ebering
SA: jmsaunde
OF:

===Summer ===
PR: b4taylor
VP: qxi
TR: m3lawren
SE:
SA:
OF:

===Fall ===
PR: Matthew Lawrence (m3lawren)
VP: Edgar Bering (ebering)
TR: Michael Gregson (mgregson)
SE: James Simpson (j2simpso) resigned for medical reasons, replaced by Dominik 'Domo' Chłobowski
SA: Kyle Spaans (kspaans)
OF:

=2009=
===Winter===
PR: Michael Gregson (mgregson)
VP: Edgar Bering (ebering)
TR: Brennan Taylor (b4taylor)
SE: James Simpson (j2simpso) resigned for business reasons, replaced by Rebecca Putinski (rjputins)
SA: Jacob Parker (j3parker)
OF: XinChi Yang / Sapphyre Gervais (x23yang / sagervai) (both)

===Spring ===
PR: Michael Spang (mspang)
VP: Jacob Parker (j3parker)
TR: Sapphyre Gervais (sagervai)
SE: Matthew McPherrin (mimcpher)
SA: Anthony Brennan (a2brenna)
OF:

===Fall===
PR: Jacob Parker (j3parker)
VP: Edgar Bering (ebering)
TR: Michael Spang (mspang)
SE: Brennan Taylor (b4taylor)
SA: Michael Ellis (m2ellis)
OF: Rebecca Putinski (rputinski)

=2010=
===Winter===
PR: Kyle Spaans (kspaans)
VP: Edgar Bering (ebering)
TR: Sapphyre Gervais (sagervai)
SE: Ajnu Jacob (ajacob)
SA: Matthew Thiffault (mthiffau)
OF: Jacob Parker (j3parker)

Keyed office staffers: j3camero,jdonland,m2ellis,mimcpher,nsasherr

===Spring===
PR: Jeff Cameron (j3camero)
VP: Brennan Taylor (b4taylor)
TR: Vardhan Mudunuru (vmudunur)
SE: Matthew Lawrence (m3lawren)
SA: Michael Ellis (m2ellis)
OF: Edgar Bering (ebering)

Past Executive

2010-08-20T01:35:19Z

Dtbartle: /* 2002 */

__NOTOC__
= Definitions =
#define PR President
#define VP Vice-president
#define TR Treasurer
#define SE Secretary
#define SA Sysadmin
#define OF Office Manager
#define LI Librarian
#define FL Flasher
#define DE Diety
#define WW Webmaster
#define SE-TR Secretary-Treasurer (Position was split)

=Founding 1967-1968=

Sponsor - J. Peter Sprung
PR: K. Rugger
VP: R. Jaques
SE-TR: G. Sutherland

Founding Members:
B. Kindree
R. Melen
V. Neglia
R. Charney
R. Truman
Glenn Berry
D. Meek

=1968-1969=

SE-TR: Glenn Berry

=1969-1970=

Unknown, only one letter found in the folder 'ACM History' addressed to Glenn Berry, which makes it likely that he was SE-TR once again. May be indicated in membership lists. The club appears to have died this academic year.

===A note on ACM affiliation===

The first attempt at joining the ACM was started with an informal inquiry Dec 5, 1967. This lead to a series of constitution edits (working towards affiliation) in Winter 1968. There was a break for the spring (no correspondence found, I presume we were waiting on a reply). In the fall records indicate that our constitution and chartering was rejected, further correspondence was sent in Fall 1968 by Glenn Berry. A new inquiry, seemingly unaware of the first was sent Dec 7, 1970

=1971=

James W. Welch is the only person sending letters for the club. Later signs them Vice-President. Continues this for most of the year. Either he was VP throughout 3 terms or we were on calendar year elections then.

=1972=

It appears we visited Western and Western visited us this year (there is some reference to a similar occurrence the year previous). Documents from 1973 indicate a termly exec structure, this probably goes back to 1972.

===Winter===

PR: Mike Campbell
SE: Doug Lacy

there is also stuff from James W. Welch without a position. Considering his work in the founding and that the letters are speaker invitations he is, with high probability, VP this term.

===Spring===

===Fall===

PR: Ian McIntosh

=1973=

Faculty Sponsor: M. Gentleman

===Winter===
===Spring===

Either secretary or VP: Jim Parry (sent the only letter on file, but it is a speaker invite, did not sign his position).

===Fall===

PR: Jim Parry
VP: Ray Walden
TR: Slavko Stemberger
SE: Mario Festival

=1974-1988=

James B- was president in Winter 88 (signature hard to read).
Tim Timar - cc'd on memos/mentioned on mathsoc minutes in 1987/88.
The Sysadmin and Office Manager positions seem to have been created somewhere in here. The 'Record Management Profile' that Robyn Stewart did as an assignment in 1991-1992 for some class at UBC
indicates the existence of both positions. As we acquired an HP-9000 in 1987 and this is likely our first computer (previously we only had a terminal?), the sysadmin was created then.

=1989=
===Winter===

PR: Kevin W. Smith ?? (derived from a signature, may be a bad reading)
VP: Angela Chambers
TR:
SE: Sean Goggin

===Spring===
===Fall===

PR: Sean Goggin

=1990=
===Winter===
===Spring===
===Fall===
PR: Wade Richards
Kivi Shapiro - attended mathsoc meeting on our behalf. - Censured by mathsoc for his actions during the election.
Shannon Mann - attended mathsoc meeting on our behalf.
Robyn Stewart - attended mathsoc meeting on our behalf.

=1991=
===Winter===
Shannon Mann - attended mathsoc meeting on our behalf.
Ed Bourne - attended mathsoc meeting on our behalf.
===Spring===
Jason Knell - attended mathsoc meeting on our and PMC's behalf.
===Fall===
Wiktor Wiewiorowski - attended mathsoc meeting on our behalf this term.

=1995=
===Winter===
===Spring===
===Fall===
PR: Amy Brown (arbrown)
VP: Christina Norman (cbnorman)
TR: Steven Mills (samills)
SE: Allison Graham (akgraham)
SA: Gavin Peters

=1996=
===Winter===
PR: Nikita Borisov (nborisov)
VP: Joseph Deu Ngoc (dtdeungo)
TR: Stephen Mills (samills)
SE: Sharlene Schmeichel (saschmei)
SA: Dave Brown (dagbrown)
OF: Somsack Tsai (stsai)
LI: Devin Carless (dccarles)
FL: Allyson Graham (akgraham)
DE: Ian Goldberg (iagoldbe)

===Spring===
PR: Blake Winton (bwinton)
VP: Nick Harvey (njaharve)
TR: Nikita Borisov (nborisov)
SE: Viet-Trung Luu (vluu)
SA: Drew Hamilton (awhamilt)
OF: Jillian Arnott (jarnott)
LI: Ross Ridge (rridge)
FL: Devin Carless (dccarles)

=== Fall ===
PR: Shannon Mann (sjbmann)
VP: Joe "Frosh" Deu Ngoc (jtdeungo) resigned (heavy workload)
TR: Michal Van Biesbrouck (mlvanbie)
SE: Nikita Borisov (nborisov)
SA: Chris Rovers
OF: Dax Hutcheon (ddhutche)
LI: Aliz Csenki (acsenki)
FL: Aaron Chmielowiec (archmiel)
DE: Skuld (no uwuserid yet...)

=1997 =

===Winter===
PR:Dima Brodsky
VP: Nikita Borisov
TR: Stephen Mills
SE: ejones
SA: Alex Brodsky
OF: Chris Doherty
LI: Matt Corks
FL: Paul Prescod

=== Fall ===
PR: Chris Rovers (cdrovers)
VP: Viet-Trung Luu (vluu)
TR: Somsack Tsai (stsai)
SE: Matt Corks (mvcorks)

= 1998 =
=== Winter ===
PR: Suresh Naidu
VP: Viet-Trung Luu
TR: Tim Coleman
SE: Dax Hutcheon
Librarian: Dax Hutcheon
Flasher: Dax Hutcheon
Webmaster: Dax Hutcheon
SA:
OF: Aaron Chmielowiec

=== Spring ===

Position Name You might call them...
President roconnor Russell O'Connor
Vice-president trwcolem Tim Coleman
Treasurer knzarysk Karl Zaryski
Secretary (bwinton) (Blake Winton)
Sysadmin wbiggs Billy Biggs
Librarian snaidu Suresh Naidu
Flasher pechrysl Paul Chrysler
Office Manager dccarles Devin Carless
WWWW trwcolem Tim Coleman

=== Fall ===

President Joe Deu Ngoc jtdeungo
Vice-President Wai Ling Yee wlyee
Treasurer Fjord j2lynn
Secretary Matt Corks mvcorks
Sysadmin Andrew Hamilton awhamilt

World Wide Web Wench Dax Hutcheon ddhutche
Office Manager Richard Bell rlbell
Librarian Damian Gryski dgryski
Flasher Paul Chrysler pechrysl
Official Deity Ian Goldberg iagoldbe
Official Chairbeing Calum T. Dalek calum

=1999=

=== Winter ===

=== Spring ===

=== Fall ===

=2000=
=== Winter ===

=== Spring ===

=== Fall ===

=2001=
=== Winter ===

=== Spring ===

=== Fall ===

=2002=
http://www.mathnews.uwaterloo.ca/Issues/mn8902/cscflash.php
=== Winter ===
PR: Billy Biggs
VP: Stefanus Du Toit
TR: Melissa Basinger
SE: James Perry
SA: Barry Genova
Librarian: Ryan Golbeck
Webmaster: Jonathan Beverley
Office Manager: Sayan Li

=== Spring ===
PR: Alex Pop
VP: Melissa Basinger
TR: Siyan Li
SE: James A Morrison
SA: Jonathan Beverley
Webmaster: Stefanus Du Toit

=== Fall ===
PR: James A. Morrison
VP: Stefanus Du Toit
TR: James Perry
SE: Michael Biggs
SA: Ryan Golbeck
Librarian: Mark Sherry, Cassandra Schopf
Webmaster: Stefanus Du Toit

=2003=
=== Winter ===
PR: Kannan Vijayan
VP: Meg Darragh
TR: James Perry
SE: Wojtek Kosnik
SA:
Librarian: Simon Law
Webmaster: Julie Lavoie

=== Spring===
===Fall===
PR: sjdutoit
VP: m3darrag
TR: tmyklebu
SE: jeperry
SA: sfllaw
OF:

=2004=
===Winter===
PR: sfllaw
VP: fspacek
TR: ljain
SE: jlavoie
SA: tmyklebu
OF:

===Spring===
PR: dnmorton ?
VP: tloach
TR: mbiggs
SE: lanortha
SA:
OF:

===Fall ===
PR: jeperry
VP: mtsay
TR: msherry
SE: tmyklebu
SA: jlavoie
OF:

=2005=
===Winter===

PR: mtsay
VP: lanortha
TR: hkarau
SE: domorton
SA: tmyklebu
OF:

===Spring===

PR: msherry
VP: mdkess
TR: apiccon
SE: mbiggs
SA: tmyklebu
OF:

===Fall===

PR: Tim Loach
VP: Lesley Northam
TR: Caelyn McAulay
SE: The Professor
SA: hkarau
OF:

=2006=

===Winter===

PR: tmyklebu
VP: mdruker
TR:
SE:
SA:
OF:

===Spring===
PR: dtbartle
VP:
TR:
SE:
SA:
OF:

===Fall===

PR: mdkess
VP: mdsherry
TR: slmably
SE: cmcaulay
SA: woconnor

=2007=

===Winter===
PR: dtbartle
VP: dbelange
TR: cmcaulay
SE: dtenty
SA: hkarau
Webmaster: jnopporn
OF:

===Spring===
PR: gdcosta
VP: kmlarose
TR: kspaans
SE: elouie
SA: mspang
Librarian: daltenty
OF:

===Fall ===
PR: hkarau
VP: amccausl
TR: dchlobow
SE: sgcummin
SA:
OF:

=2008=
===Winter ===
PR: sgcummin
VP: m3lawren
TR: mtarkows
SE: ebering
SA: jmsaunde
OF:

===Summer ===
PR: b4taylor
VP: qxi
TR: m3lawren
SE:
SA:
OF:

===Fall ===
PR: Matthew Lawrence (m3lawren)
VP: Edgar Bering (ebering)
TR: Michael Gregson (mgregson)
SE: James Simpson (j2simpso) resigned for medical reasons, replaced by Dominik 'Domo' Chłobowski
SA: Kyle Spaans (kspaans)
OF:

=2009=
===Winter===
PR: Michael Gregson (mgregson)
VP: Edgar Bering (ebering)
TR: Brennan Taylor (b4taylor)
SE: James Simpson (j2simpso) resigned for business reasons, replaced by Rebecca Putinski (rjputins)
SA: Jacob Parker (j3parker)
OF: XinChi Yang / Sapphyre Gervais (x23yang / sagervai) (both)

===Spring ===
PR: Michael Spang (mspang)
VP: Jacob Parker (j3parker)
TR: Sapphyre Gervais (sagervai)
SE: Matthew McPherrin (mimcpher)
SA: Anthony Brennan (a2brenna)
OF:

===Fall===
PR: Jacob Parker (j3parker)
VP: Edgar Bering (ebering)
TR: Michael Spang (mspang)
SE: Brennan Taylor (b4taylor)
SA: Michael Ellis (m2ellis)
OF: Rebecca Putinski (rputinski)

=2010=
===Winter===
PR: Kyle Spaans (kspaans)
VP: Edgar Bering (ebering)
TR: Sapphyre Gervais (sagervai)
SE: Ajnu Jacob (ajacob)
SA: Matthew Thiffault (mthiffau)
OF: Jacob Parker (j3parker)

Keyed office staffers: j3camero,jdonland,m2ellis,mimcpher,nsasherr

===Spring===
PR: Jeff Cameron (j3camero)
VP: Brennan Taylor (b4taylor)
TR: Vardhan Mudunuru (vmudunur)
SE: Matthew Lawrence (m3lawren)
SA: Michael Ellis (m2ellis)
OF: Edgar Bering (ebering)

Past Executive

2010-08-20T01:32:20Z

Dtbartle: Add fall 2002

__NOTOC__
= Definitions =
#define PR President
#define VP Vice-president
#define TR Treasurer
#define SE Secretary
#define SA Sysadmin
#define OF Office Manager
#define LI Librarian
#define FL Flasher
#define DE Diety
#define WW Webmaster
#define SE-TR Secretary-Treasurer (Position was split)

=Founding 1967-1968=

Sponsor - J. Peter Sprung
PR: K. Rugger
VP: R. Jaques
SE-TR: G. Sutherland

Founding Members:
B. Kindree
R. Melen
V. Neglia
R. Charney
R. Truman
Glenn Berry
D. Meek

=1968-1969=

SE-TR: Glenn Berry

=1969-1970=

Unknown, only one letter found in the folder 'ACM History' addressed to Glenn Berry, which makes it likely that he was SE-TR once again. May be indicated in membership lists. The club appears to have died this academic year.

===A note on ACM affiliation===

The first attempt at joining the ACM was started with an informal inquiry Dec 5, 1967. This lead to a series of constitution edits (working towards affiliation) in Winter 1968. There was a break for the spring (no correspondence found, I presume we were waiting on a reply). In the fall records indicate that our constitution and chartering was rejected, further correspondence was sent in Fall 1968 by Glenn Berry. A new inquiry, seemingly unaware of the first was sent Dec 7, 1970

=1971=

James W. Welch is the only person sending letters for the club. Later signs them Vice-President. Continues this for most of the year. Either he was VP throughout 3 terms or we were on calendar year elections then.

=1972=

It appears we visited Western and Western visited us this year (there is some reference to a similar occurrence the year previous). Documents from 1973 indicate a termly exec structure, this probably goes back to 1972.

===Winter===

PR: Mike Campbell
SE: Doug Lacy

there is also stuff from James W. Welch without a position. Considering his work in the founding and that the letters are speaker invitations he is, with high probability, VP this term.

===Spring===

===Fall===

PR: Ian McIntosh

=1973=

Faculty Sponsor: M. Gentleman

===Winter===
===Spring===

Either secretary or VP: Jim Parry (sent the only letter on file, but it is a speaker invite, did not sign his position).

===Fall===

PR: Jim Parry
VP: Ray Walden
TR: Slavko Stemberger
SE: Mario Festival

=1974-1988=

James B- was president in Winter 88 (signature hard to read).
Tim Timar - cc'd on memos/mentioned on mathsoc minutes in 1987/88.
The Sysadmin and Office Manager positions seem to have been created somewhere in here. The 'Record Management Profile' that Robyn Stewart did as an assignment in 1991-1992 for some class at UBC
indicates the existence of both positions. As we acquired an HP-9000 in 1987 and this is likely our first computer (previously we only had a terminal?), the sysadmin was created then.

=1989=
===Winter===

PR: Kevin W. Smith ?? (derived from a signature, may be a bad reading)
VP: Angela Chambers
TR:
SE: Sean Goggin

===Spring===
===Fall===

PR: Sean Goggin

=1990=
===Winter===
===Spring===
===Fall===
PR: Wade Richards
Kivi Shapiro - attended mathsoc meeting on our behalf. - Censured by mathsoc for his actions during the election.
Shannon Mann - attended mathsoc meeting on our behalf.
Robyn Stewart - attended mathsoc meeting on our behalf.

=1991=
===Winter===
Shannon Mann - attended mathsoc meeting on our behalf.
Ed Bourne - attended mathsoc meeting on our behalf.
===Spring===
Jason Knell - attended mathsoc meeting on our and PMC's behalf.
===Fall===
Wiktor Wiewiorowski - attended mathsoc meeting on our behalf this term.

=1995=
===Winter===
===Spring===
===Fall===
PR: Amy Brown (arbrown)
VP: Christina Norman (cbnorman)
TR: Steven Mills (samills)
SE: Allison Graham (akgraham)
SA: Gavin Peters

=1996=
===Winter===
PR: Nikita Borisov (nborisov)
VP: Joseph Deu Ngoc (dtdeungo)
TR: Stephen Mills (samills)
SE: Sharlene Schmeichel (saschmei)
SA: Dave Brown (dagbrown)
OF: Somsack Tsai (stsai)
LI: Devin Carless (dccarles)
FL: Allyson Graham (akgraham)
DE: Ian Goldberg (iagoldbe)

===Spring===
PR: Blake Winton (bwinton)
VP: Nick Harvey (njaharve)
TR: Nikita Borisov (nborisov)
SE: Viet-Trung Luu (vluu)
SA: Drew Hamilton (awhamilt)
OF: Jillian Arnott (jarnott)
LI: Ross Ridge (rridge)
FL: Devin Carless (dccarles)

=== Fall ===
PR: Shannon Mann (sjbmann)
VP: Joe "Frosh" Deu Ngoc (jtdeungo) resigned (heavy workload)
TR: Michal Van Biesbrouck (mlvanbie)
SE: Nikita Borisov (nborisov)
SA: Chris Rovers
OF: Dax Hutcheon (ddhutche)
LI: Aliz Csenki (acsenki)
FL: Aaron Chmielowiec (archmiel)
DE: Skuld (no uwuserid yet...)

=1997 =

===Winter===
PR:Dima Brodsky
VP: Nikita Borisov
TR: Stephen Mills
SE: ejones
SA: Alex Brodsky
OF: Chris Doherty
LI: Matt Corks
FL: Paul Prescod

=== Fall ===
PR: Chris Rovers (cdrovers)
VP: Viet-Trung Luu (vluu)
TR: Somsack Tsai (stsai)
SE: Matt Corks (mvcorks)

= 1998 =
=== Winter ===
PR: Suresh Naidu
VP: Viet-Trung Luu
TR: Tim Coleman
SE: Dax Hutcheon
Librarian: Dax Hutcheon
Flasher: Dax Hutcheon
Webmaster: Dax Hutcheon
SA:
OF: Aaron Chmielowiec

=== Spring ===

Position Name You might call them...
President roconnor Russell O'Connor
Vice-president trwcolem Tim Coleman
Treasurer knzarysk Karl Zaryski
Secretary (bwinton) (Blake Winton)
Sysadmin wbiggs Billy Biggs
Librarian snaidu Suresh Naidu
Flasher pechrysl Paul Chrysler
Office Manager dccarles Devin Carless
WWWW trwcolem Tim Coleman

=== Fall ===

President Joe Deu Ngoc jtdeungo
Vice-President Wai Ling Yee wlyee
Treasurer Fjord j2lynn
Secretary Matt Corks mvcorks
Sysadmin Andrew Hamilton awhamilt

World Wide Web Wench Dax Hutcheon ddhutche
Office Manager Richard Bell rlbell
Librarian Damian Gryski dgryski
Flasher Paul Chrysler pechrysl
Official Deity Ian Goldberg iagoldbe
Official Chairbeing Calum T. Dalek calum

=1999=

=== Winter ===

=== Spring ===

=== Fall ===

=2000=
=== Winter ===

=== Spring ===

=== Fall ===

=2001=
=== Winter ===

=== Spring ===

=== Fall ===

=2002=
http://www.mathnews.uwaterloo.ca/Issues/mn8902/cscflash.php
=== Winter ===

=== Spring ===

=== Fall ===
PR: James A. Morrison
VP: Stefanus Du Toit
SE: Michael Biggs
TR: James Perry
SA: Ryan Golbeck
Librarian: Mark Sherry, Cassandra Schopf
Webmaster: Stefanus Du Toit

=2003=
=== Winter ===
PR: Kannan Vijayan
VP: Meg Darragh
TR: James Perry
SE: Wojtek Kosnik
SA:
Librarian: Simon Law
Webmaster: Julie Lavoie

=== Spring===
===Fall===
PR: sjdutoit
VP: m3darrag
TR: tmyklebu
SE: jeperry
SA: sfllaw
OF:

=2004=
===Winter===
PR: sfllaw
VP: fspacek
TR: ljain
SE: jlavoie
SA: tmyklebu
OF:

===Spring===
PR: dnmorton ?
VP: tloach
TR: mbiggs
SE: lanortha
SA:
OF:

===Fall ===
PR: jeperry
VP: mtsay
TR: msherry
SE: tmyklebu
SA: jlavoie
OF:

=2005=
===Winter===

PR: mtsay
VP: lanortha
TR: hkarau
SE: domorton
SA: tmyklebu
OF:

===Spring===

PR: msherry
VP: mdkess
TR: apiccon
SE: mbiggs
SA: tmyklebu
OF:

===Fall===

PR: Tim Loach
VP: Lesley Northam
TR: Caelyn McAulay
SE: The Professor
SA: hkarau
OF:

=2006=

===Winter===

PR: tmyklebu
VP: mdruker
TR:
SE:
SA:
OF:

===Spring===
PR: dtbartle
VP:
TR:
SE:
SA:
OF:

===Fall===

PR: mdkess
VP: mdsherry
TR: slmably
SE: cmcaulay
SA: woconnor

=2007=

===Winter===
PR: dtbartle
VP: dbelange
TR: cmcaulay
SE: dtenty
SA: hkarau
Webmaster: jnopporn
OF:

===Spring===
PR: gdcosta
VP: kmlarose
TR: kspaans
SE: elouie
SA: mspang
Librarian: daltenty
OF:

===Fall ===
PR: hkarau
VP: amccausl
TR: dchlobow
SE: sgcummin
SA:
OF:

=2008=
===Winter ===
PR: sgcummin
VP: m3lawren
TR: mtarkows
SE: ebering
SA: jmsaunde
OF:

===Summer ===
PR: b4taylor
VP: qxi
TR: m3lawren
SE:
SA:
OF:

===Fall ===
PR: Matthew Lawrence (m3lawren)
VP: Edgar Bering (ebering)
TR: Michael Gregson (mgregson)
SE: James Simpson (j2simpso) resigned for medical reasons, replaced by Dominik 'Domo' Chłobowski
SA: Kyle Spaans (kspaans)
OF:

=2009=
===Winter===
PR: Michael Gregson (mgregson)
VP: Edgar Bering (ebering)
TR: Brennan Taylor (b4taylor)
SE: James Simpson (j2simpso) resigned for business reasons, replaced by Rebecca Putinski (rjputins)
SA: Jacob Parker (j3parker)
OF: XinChi Yang / Sapphyre Gervais (x23yang / sagervai) (both)

===Spring ===
PR: Michael Spang (mspang)
VP: Jacob Parker (j3parker)
TR: Sapphyre Gervais (sagervai)
SE: Matthew McPherrin (mimcpher)
SA: Anthony Brennan (a2brenna)
OF:

===Fall===
PR: Jacob Parker (j3parker)
VP: Edgar Bering (ebering)
TR: Michael Spang (mspang)
SE: Brennan Taylor (b4taylor)
SA: Michael Ellis (m2ellis)
OF: Rebecca Putinski (rputinski)

=2010=
===Winter===
PR: Kyle Spaans (kspaans)
VP: Edgar Bering (ebering)
TR: Sapphyre Gervais (sagervai)
SE: Ajnu Jacob (ajacob)
SA: Matthew Thiffault (mthiffau)
OF: Jacob Parker (j3parker)

Keyed office staffers: j3camero,jdonland,m2ellis,mimcpher,nsasherr

===Spring===
PR: Jeff Cameron (j3camero)
VP: Brennan Taylor (b4taylor)
TR: Vardhan Mudunuru (vmudunur)
SE: Matthew Lawrence (m3lawren)
SA: Michael Ellis (m2ellis)
OF: Edgar Bering (ebering)

Past Executive

2010-08-20T01:20:03Z

Dtbartle: Add dtbartle as president for spring 2006

__NOTOC__
= Definitions =
#define PR President
#define VP Vice-president
#define TR Treasurer
#define SE Secretary
#define SA Sysadmin
#define OF Office Manager

=Founding 1967-1968=

Sponsor - J. Peter Sprung
PR: K. Rugger
VP: R. Jaques
SE-TR: G. Sutherland

Founding Members:
B. Kindree
R. Melen
V. Neglia
R. Charney
R. Truman
Glenn Berry
D. Meek

=1968-1969=

SE-TR: Glenn Berry

=1969-1970=

Unknown, only one letter found in the folder 'ACM History' addressed to Glenn Berry, which makes it likely that he was SE-TR once again. May be indicated in membership lists. The club appears to have died this academic year.

===A note on ACM affiliation===

The first attempt at joining the ACM was started with an informal inquiry Dec 5, 1967. This lead to a series of constitution edits (working towards affiliation) in Winter 1968. There was a break for the spring (no correspondence found, I presume we were waiting on a reply). In the fall records indicate that our constitution and chartering was rejected, further correspondence was sent in Fall 1968 by Glenn Berry. A new inquiry, seemingly unaware of the first was sent Dec 7, 1970

=1971=

James W. Welch is the only person sending letters for the club. Later signs them Vice-President. Continues this for most of the year. Either he was VP throughout 3 terms or we were on calendar year elections then.

=1972=

It appears we visited Western and Western visited us this year (there is some reference to a similar occurrence the year previous). Documents from 1973 indicate a termly exec structure, this probably goes back to 1972.

===Winter===

PR: Mike Campbell
SE: Doug Lacy
there is also stuff from James W. Welch without a position. Considering his work in the founding and that the letters are speaker invitations he is, with high probability, VP this term.

===Spring===

===Fall===

PR: Ian McIntosh

=1973=

Faculty Sponsor: M. Gentleman

===Winter===
===Spring===

Either secretary or VP: Jim Parry (sent the only letter on file, but it is a speaker invite, did not sign his position).

===Fall===

PR: Jim Parry
VP: Ray Walden
TR: Slavko Stemberger
SE: Mario Festival

=1974-1988=

James B- was president in Winter 88 (signature hard to read)
Tim Timar - cc'd on memos/mentioned on mathsoc minutes in 1987/88.
The Sysadmin and Office Manager positions seem to have been created somewhere in here. The 'Record Management Profile' that Robyn Stewart did as an assignment in 1991-1992 for some class at UBC
indicates the existence of both positions. As we acquired an HP-9000 in 1987 and this is likely our first computer (previously we only had a terminal?), the sysadmin was created then.

=1989=
===Winter===

PR: Kevin W. Smith ?? (derived from a signature, may be a bad reading)
VP: Angela Chambers
TR:
SE: Sean Goggin

===Spring===
===Fall===

PR: Sean Goggin

=1990=
===Winter===
===Spring===
===Fall===
PR: Wade Richards
Kivi Shapiro - attended mathsoc meeting on our behalf. - Censured by mathsoc for his actions during the election.
Shannon Mann - attended mathsoc meeting on our behalf.
Robyn Stewart - attended mathsoc meeting on our behalf.

=1991=
===Winter===
Shannon Mann - attended mathsoc meeting on our behalf.
Ed Bourne - attended mathsoc meeting on our behalf.
===Spring===
Jason Knell - attended mathsoc meeting on our and PMC's behalf.
===Fall===
Wiktor Wiewiorowski - attended mathsoc meeting on our behalf this term.

=1995=
===Winter===
===Spring===
===Fall===
PR: Amy Brown (arbrown)
VP: Christina Norman (cbnorman)
TR: Steven Mills (samills)
SE: Allison Graham (akgraham)
SA: Gavin Peters

=1996=
===Winter===
PR: Nikita Borisov (nborisov)
VP: Joseph Deu Ngoc (dtdeungo)
TR: Stephen Mills (samills)
SE: Sharlene Schmeichel (saschmei)
SA: Dave Brown (dagbrown)
OF: Somsack Tsai (stsai)
LI: Devin Carless (dccarles)
FL: Allyson Graham (akgraham)
DE: Ian Goldberg (iagoldbe)

===Spring===
PR: Blake Winton (bwinton)
VP: Nick Harvey (njaharve)
TR: Nikita Borisov (nborisov)
SE: Viet-Trung Luu (vluu)
SA: Drew Hamilton (awhamilt)
OF: Jillian Arnott (jarnott)
LI: Ross Ridge (rridge)
FL: Devin Carless (dccarles)

=== Fall ===
PR: Shannon Mann (sjbmann)
VP: Joe "Frosh" Deu Ngoc (jtdeungo) resigned (heavy workload)
TR: Michal Van Biesbrouck (mlvanbie)
SE: Nikita Borisov (nborisov)
SA: Chris Rovers
OF: Dax Hutcheon (ddhutche)
LI: Aliz Csenki (acsenki)
FL: Aaron Chmielowiec (archmiel)
DE: Skuld (no uwuserid yet...)

=1997 =

===Winter===
PR:Dima Brodsky
VP: Nikita Borisov
TR: Stephen Mills
SE: ejones
SA: Alex Brodsky
OF: Chris Doherty
LI: Matt Corks
FL: Paul Prescod

=== Fall ===
PR: Chris Rovers (cdrovers)
VP: Viet-Trung Luu (vluu)
TR: Somsack Tsai (stsai)
SE: Matt Corks (mvcorks)

= 1998 =
=== Winter ===
PR: Suresh Naidu
VP: Viet-Trung Luu
TR: Tim Coleman
SE: Dax Hutcheon
Librarian: Dax Hutcheon
Flasher: Dax Hutcheon
Webmaster: Dax Hutcheon
SA:
OF: Aaron Chmielowiec

=== Spring ===

Position Name You might call them...
President roconnor Russell O'Connor
Vice-president trwcolem Tim Coleman
Treasurer knzarysk Karl Zaryski
Secretary (bwinton) (Blake Winton)
Sysadmin wbiggs Billy Biggs
Librarian snaidu Suresh Naidu
Flasher pechrysl Paul Chrysler
Office Manager dccarles Devin Carless
WWWW trwcolem Tim Coleman

=== Fall ===

President Joe Deu Ngoc jtdeungo
Vice-President Wai Ling Yee wlyee
Treasurer Fjord j2lynn
Secretary Matt Corks mvcorks
Sysadmin Andrew Hamilton awhamilt

World Wide Web Wench Dax Hutcheon ddhutche
Office Manager Richard Bell rlbell
Librarian Damian Gryski dgryski
Flasher Paul Chrysler pechrysl
Official Deity Ian Goldberg iagoldbe
Official Chairbeing Calum T. Dalek calum

=1999=

=== Winter ===

=== Spring ===

=== Fall ===

=2000=
=== Winter ===

=== Spring ===

=== Fall ===

=2001=
=== Winter ===

=== Spring ===

=== Fall ===

=2002=
http://www.mathnews.uwaterloo.ca/Issues/mn8902/cscflash.php
=== Winter ===

=== Spring ===

=== Fall ===

=2003=
=== Winter ===
PR: Kannan Vijayan
VP: Meg Darragh
TR: James Perry
SE: Wojtek Kosnik
SA:
Librarian: Simon Law
Webmaster: Julie Lavoie

=== Spring===
===Fall===
PR: sjdutoit
VP: m3darrag
TR: tmyklebu
SE: jeperry
SA: sfllaw
OF:

=2004=
===Winter===
PR: sfllaw
VP: fspacek
TR: ljain
SE: jlavoie
SA: tmyklebu
OF:

===Spring===
PR: dnmorton ?
VP: tloach
TR: mbiggs
SE: lanortha
SA:
OF:

===Fall ===
PR: jeperry
VP: mtsay
TR: msherry
SE: tmyklebu
SA: jlavoie
OF:

=2005=
===Winter===

PR: mtsay
VP: lanortha
TR: hkarau
SE: domorton
SA: tmyklebu
OF:

===Spring===

PR: msherry
VP: mdkess
TR: apiccon
SE: mbiggs
SA: tmyklebu
OF:

===Fall===

PR: Tim Loach
VP: Lesley Northam
TR: Caelyn McAulay
SE: The Professor
SA: hkarau
OF:

=2006=

===Winter===

PR: tmyklebu
VP: mdruker
TR:
SE:
SA:
OF:

===Spring===
PR: dtbartle
VP:
TR:
SE:
SA:
OF:

===Fall===

PR: mdkess
VP: mdsherry
TR: slmably
SE: cmcaulay
SA: woconnor

=2007=

===Winter===
PR: dtbartle
VP: dbelange
TR: cmcaulay
SE: dtenty
SA: hkarau
Webmaster: jnopporn
OF:

===Spring===
PR: gdcosta
VP: kmlarose
TR: kspaans
SE: elouie
SA: mspang
Librarian: daltenty
OF:

===Fall ===
PR: hkarau
VP: amccausl
TR: dchlobow
SE: sgcummin
SA:
OF:

=2008=
===Winter ===
PR: sgcummin
VP: m3lawren
TR: mtarkows
SE: ebering
SA: jmsaunde
OF:

===Summer ===
PR: b4taylor
VP: qxi
TR: m3lawren
SE:
SA:
OF:

===Fall ===
PR: Matthew Lawrence (m3lawren)
VP: Edgar Bering (ebering)
TR: Michael Gregson (mgregson)
SE: James Simpson (j2simpso) resigned for medical reasons, replaced by Dominik 'Domo' Chłobowski
SA: Kyle Spaans (kspaans)
OF:

=2009=
===Winter===
PR: Michael Gregson (mgregson)
VP: Edgar Bering (ebering)
TR: Brennan Taylor (b4taylor)
SE: James Simpson (j2simpso) resigned for business reasons, replaced by Rebecca Putinski (rjputins)
SA: Jacob Parker (j3parker)
OF: XinChi Yang / Sapphyre Gervais (x23yang / sagervai) (both)

===Spring ===
PR: Michael Spang (mspang)
VP: Jacob Parker (j3parker)
TR: Sapphyre Gervais (sagervai)
SE: Matthew McPherrin (mimcpher)
SA: Anthony Brennan (a2brenna)
OF:

===Fall===
PR: Jacob Parker (j3parker)
VP: Edgar Bering (ebering)
TR: Michael Spang (mspang)
SE: Brennan Taylor (b4taylor)
SA: Michael Ellis (m2ellis)
OF: Rebecca Putinski (rputinski)

=2010=
===Winter===
PR: Kyle Spaans (kspaans)
VP: Edgar Bering (ebering)
TR: Sapphyre Gervais (sagervai)
SE: Ajnu Jacob (ajacob)
SA: Matthew Thiffault (mthiffau)
OF: Jacob Parker (j3parker)

Keyed office staffers: j3camero,jdonland,m2ellis,mimcpher,nsasherr

===Spring===
PR: Jeff Cameron (j3camero)
VP: Brennan Taylor (b4taylor)
TR: Vardhan Mudunuru (vmudunur)
SE: Matthew Lawrence (m3lawren)
SA: Michael Ellis (m2ellis)
OF: Edgar Bering (ebering)

Mirror

2010-07-17T03:51:37Z

Dtbartle:

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on [[Machine_List#sodium-benzoate|sodium-benzoate]]. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 3.6 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 28 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 30 GiB || -
|-
| Blastwave || rsync://master.rsync.blastwave.org/blastwave/ http://www.blastwave.org/mirrors.php || 13 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 123 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 7.1 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 50 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 17 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 8.7 GiB || -
|-
| Damn Small Linux || rsync://ftp.heanet.ie/mirrors/damnsmalllinux.org/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://ftp3.nrc.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 506 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 39 GiB || Submitted a request to be added to the mirror list long ago with no response.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 76 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 6.1 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 48 GiB || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 45 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://debian-maintainers.org/unofficial/ http://www.debian-unofficial.org/mirrors.html || 465 MiB || -
|-
| Debian-volatile || rsync://volatile-master.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4.2 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 293 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/debian/ || 3.5 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://fedora-archives.ibiblio.org/fedora-enchilada/ http://mirrors.fedoraproject.org/publiclist/ || 667 GiB || -
|-
| Fedora-epel || rsync://fedora-archives.ibiblio.org/fedora-epel/ http://mirrors.fedoraproject.org/publiclist/ || 32 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ http://mirrorlist.freebsd.org/ || 1.8 TiB || We are ftp2.ca.freebsd.org.
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 606 MiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 165 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 95 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 22 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 85 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 140 GiB || -
|-
| Linux Mint (releases) || rsync://ftp.heanet.ie/pub/linuxmint.com/ http://www.linuxmint.com/mirrors.php || 35 GiB || -
|-
| Linux Mint (packages) || rsync://packages.linuxmint.com/packages/ http://www.linuxmint.com/mirrors.php || 5.0 GiB || -
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6.2 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 99 GiB || -
|-
| MySQL || rsync://mysql.he.net/mysql/ http://dev.mysql.com/downloads/mirrors.html || 212 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 16 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors || 129 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://stage.opensuse.org/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/all.html || 192 GiB || -
|-
| Slackware || rsync://slackware.cs.utah.edu/slackware/ http://slackware.com/getslack/ || 140 GiB || '''We have asked to get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 344 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 445 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 40 GiB || -
|-
| Ubuntu-releases || rsync://rsync.releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 40 GiB || Are the official Canadian mirror (''i.e.'', ca.releases.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5.6 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Proposed Archives to Mirror ==

* Mandriva
* OpenBSD (inc. OpenSSH)
* NetBSD
* PCLinuxOS
* OpenSSL
* RubyForge
* SourceForge
* MacPorts
* PLT Scheme (they don't ask for mirrors, but they currently offer download from half a dozen or so sites)
* VLC

== Implementation Details ==

The mirroring is done by one of three scripts. The latter two are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. [[#merlin|merlin]] is used to call one of these scripts. Most of the scripts and such used to maintain the mirror are available in the public [http://git.csclub.uwaterloo.ca/?p=public/mirror.git mirror.git] repository.

=== ftpsync ===

[http://ftp-master.debian.org/ftpsync.tar.gz ftpsync] is the official Debian mirror synchronization tool, and is used to rsync the Debian repository. It's located in ~mirror/debian. Its invocation takes a few steps (this is more or less how [[#merlin|merlin]] invokes it:

export BASEDIR=/home/mirror/debian
cd $BASEDIR
./bin/ftpsync sync:stage1
./bin/ftpsync sync:stage2

=== csc-sync-debian ===

This is used to sync debian-style repositories. Its usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. Its usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== merlin ===

The synchronization process is run by a Python script called "merlin", written by a2brenna, stored in ~mirror/merlin. The repository list, sync time, etc. is maintained in merlin.py.

=== HTTP ===

We use [[Apache]] as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 64
ThreadLimit 64
StartServers 2
MaxClients 4096
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

==== Index ====

An index of the archives we mirror is available at http://mirror.csclub.uwaterloo.ca/.
As of Winter 2010, it is now generated by a Python script in <tt>~mirror/mirror-index</tt>.

<tt>~mirror/mirror-index/make-index.py</tt> is scheduled in <tt>mirror</tt>'s crontab to be
run at 5:40 AM on the 14th and 28th of each month. The script can be run manually when needed
(for example, when an archive is removed) as follows:

sudo -u mirror /home/mirror/mirror-index/make-index.py

This causes an instance of <tt>du</tt> to be run which computes the size of each directory. This
list is then sorted alphabetically by directory name and returned to the Python script.
If any errors occur during this process, the script conservatively chooses to exit rather
than risk generating an index file that is incorrect.

<tt>make-index.py</tt> is configured by means of a [http://www.yaml.org/ YAML] file,
<tt>config.yaml</tt>, in the same directory. Its format is as follows:

docroot: /mirror/root
duflags: --human-readable --max-depth=1
output: /mirror/root/index.html

directories:
apache:
site: apache.org
url: <nowiki>http://www.apache.org/</nowiki>

archlinux:
site: archlinux.org
url: <nowiki>http://www.archlinux.org/</nowiki>

# (...)

The <tt>docroot</tt> is the directory which is to be scanned; this will probably
always be the mirror root from which Apache serves. <tt>duflags</tt> specifies
the flags to be passed to <tt>du</tt>. This is here so that it's easy to find
and alter. For instance, we could change <tt>--human-readable</tt> to <tt>--si</tt>
if we ever decided that, like hard disk manufacturers, we want sizes to appear larger
than they are. <tt>output</tt> defines the file to which the generated index will be
written.

Finally, <tt>directories</tt> specifies the list of directories to be listed.
No directories not listed here will be shown. If you add a new archive and it doesn't
appear, that's why. The format is fairly straightforward: simply name the directory
and provide a site (the display name in the "Project Site" column) and URL.

One caveat here is that YAML does not allow tabs for whitespace. Indent with
two spaces to remain consistent with the existing file format, please. Also note
that the directory name is case-sensitive, as is always the case on Unix.

Finally, the HTML index file is generated from <tt>index.mako</tt>, a
[http://www.makotemplates.org/ Mako] template (which is mostly HTML anyhow).
If you really can't figure out how it works, look up the Mako documentation.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

[[Category:Services]]
[[Category:Software]]

Mirror

2010-07-16T17:08:51Z

Dtbartle:

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on [[Machine_List#sodium-benzoate|sodium-benzoate]]. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 3.6 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 28 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 30 GiB || -
|-
| Blastwave || rsync://master.rsync.blastwave.org/blastwave/ http://www.blastwave.org/mirrors.php || 13 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 123 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 7.1 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 50 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 17 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 8.7 GiB || -
|-
| Damn Small Linux || rsync://ftp.heanet.ie/mirrors/damnsmalllinux.org/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://ftp3.nrc.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 506 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 39 GiB || Submitted a request to be added to the mirror list long ago with no response.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 76 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 6.1 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 48 GiB || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 45 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://debian-maintainers.org/unofficial/ http://www.debian-unofficial.org/mirrors.html || 465 MiB || -
|-
| Debian-volatile || rsync://volatile-master.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4.2 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 293 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/debian/ || 3.5 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://fedora-archives.ibiblio.org/fedora-enchilada/ http://mirrors.fedoraproject.org/publiclist/ || 667 GiB || -
|-
| Fedora-epel || rsync://fedora-archives.ibiblio.org/fedora-epel/ http://mirrors.fedoraproject.org/publiclist/ || 32 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ http://mirrorlist.freebsd.org/ || 1.2 TiB || We are ftp2.ca.freebsd.org.
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 606 MiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 165 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 95 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 22 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 85 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 140 GiB || -
|-
| Linux Mint (releases) || rsync://ftp.heanet.ie/pub/linuxmint.com/ http://www.linuxmint.com/mirrors.php || 35 GiB || -
|-
| Linux Mint (packages) || rsync://packages.linuxmint.com/packages/ http://www.linuxmint.com/mirrors.php || 5.0 GiB || -
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6.2 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 99 GiB || -
|-
| MySQL || rsync://mysql.he.net/mysql/ http://dev.mysql.com/downloads/mirrors.html || 212 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 16 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors || 129 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://stage.opensuse.org/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/all.html || 192 GiB || -
|-
| Slackware || rsync://slackware.cs.utah.edu/slackware/ http://slackware.com/getslack/ || 140 GiB || '''We have asked to get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 344 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 445 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 40 GiB || -
|-
| Ubuntu-releases || rsync://rsync.releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 40 GiB || Are the official Canadian mirror (''i.e.'', ca.releases.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5.6 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Proposed Archives to Mirror ==

* Mandriva
* OpenBSD (inc. OpenSSH)
* NetBSD
* PCLinuxOS
* OpenSSL
* RubyForge
* SourceForge
* MacPorts
* PLT Scheme (they don't ask for mirrors, but they currently offer download from half a dozen or so sites)
* VLC

== Implementation Details ==

The mirroring is done by one of three scripts. The latter two are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. [[#merlin|merlin]] is used to call one of these scripts. Most of the scripts and such used to maintain the mirror are available in the public [http://git.csclub.uwaterloo.ca/?p=public/mirror.git mirror.git] repository.

=== ftpsync ===

[http://ftp-master.debian.org/ftpsync.tar.gz ftpsync] is the official Debian mirror synchronization tool, and is used to rsync the Debian repository. It's located in ~mirror/debian. Its invocation takes a few steps (this is more or less how [[#merlin|merlin]] invokes it:

export BASEDIR=/home/mirror/debian
cd $BASEDIR
./bin/ftpsync sync:stage1
./bin/ftpsync sync:stage2

=== csc-sync-debian ===

This is used to sync debian-style repositories. Its usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. Its usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== merlin ===

The synchronization process is run by a Python script called "merlin", written by a2brenna, stored in ~mirror/merlin. The repository list, sync time, etc. is maintained in merlin.py.

=== HTTP ===

We use [[Apache]] as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 64
ThreadLimit 64
StartServers 2
MaxClients 4096
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

==== Index ====

An index of the archives we mirror is available at http://mirror.csclub.uwaterloo.ca/.
As of Winter 2010, it is now generated by a Python script in <tt>~mirror/mirror-index</tt>.

<tt>~mirror/mirror-index/make-index.py</tt> is scheduled in <tt>mirror</tt>'s crontab to be
run at 5:40 AM on the 14th and 28th of each month. The script can be run manually when needed
(for example, when an archive is removed) as follows:

sudo -u mirror /home/mirror/mirror-index/make-index.py

This causes an instance of <tt>du</tt> to be run which computes the size of each directory. This
list is then sorted alphabetically by directory name and returned to the Python script.
If any errors occur during this process, the script conservatively chooses to exit rather
than risk generating an index file that is incorrect.

<tt>make-index.py</tt> is configured by means of a [http://www.yaml.org/ YAML] file,
<tt>config.yaml</tt>, in the same directory. Its format is as follows:

docroot: /mirror/root
duflags: --human-readable --max-depth=1
output: /mirror/root/index.html

directories:
apache:
site: apache.org
url: <nowiki>http://www.apache.org/</nowiki>

archlinux:
site: archlinux.org
url: <nowiki>http://www.archlinux.org/</nowiki>

# (...)

The <tt>docroot</tt> is the directory which is to be scanned; this will probably
always be the mirror root from which Apache serves. <tt>duflags</tt> specifies
the flags to be passed to <tt>du</tt>. This is here so that it's easy to find
and alter. For instance, we could change <tt>--human-readable</tt> to <tt>--si</tt>
if we ever decided that, like hard disk manufacturers, we want sizes to appear larger
than they are. <tt>output</tt> defines the file to which the generated index will be
written.

Finally, <tt>directories</tt> specifies the list of directories to be listed.
No directories not listed here will be shown. If you add a new archive and it doesn't
appear, that's why. The format is fairly straightforward: simply name the directory
and provide a site (the display name in the "Project Site" column) and URL.

One caveat here is that YAML does not allow tabs for whitespace. Indent with
two spaces to remain consistent with the existing file format, please. Also note
that the directory name is case-sensitive, as is always the case on Unix.

Finally, the HTML index file is generated from <tt>index.mako</tt>, a
[http://www.makotemplates.org/ Mako] template (which is mostly HTML anyhow).
If you really can't figure out how it works, look up the Mako documentation.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

[[Category:Services]]
[[Category:Software]]

Mirror

2010-07-11T16:05:47Z

Dtbartle:

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on [[Machine_List#sodium-benzoate|sodium-benzoate]]. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 3.6 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 28 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 30 GiB || -
|-
| Blastwave || rsync://master.rsync.blastwave.org/blastwave/ http://www.blastwave.org/mirrors.php || 13 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 123 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 7.1 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 50 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 17 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 8.7 GiB || -
|-
| Damn Small Linux || rsync://ftp.heanet.ie/mirrors/damnsmalllinux.org/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://ftp3.nrc.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 506 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 39 GiB || Submitted a request to be added to the mirror list long ago with no response.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 76 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 6.1 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 48 GiB || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 45 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://debian-maintainers.org/unofficial/ http://www.debian-unofficial.org/mirrors.html || 465 MiB || -
|-
| Debian-volatile || rsync://volatile-master.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4.2 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 293 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/debian/ || 3.5 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://fedora-archives.ibiblio.org/fedora-enchilada/ http://mirrors.fedoraproject.org/publiclist/ || 667 GiB || -
|-
| Fedora-epel || rsync://fedora-archives.ibiblio.org/fedora-epel/ http://mirrors.fedoraproject.org/publiclist/ || 32 GiB || -
|-
| <s>FreeBSD</s> || rsync://ftp1.ca.freebsd.org/ http://mirrorlist.freebsd.org/ || 1.2 TiB || Removed to free up space. '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 606 MiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 165 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 95 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 22 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 85 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 140 GiB || -
|-
| Linux Mint (releases) || rsync://ftp.heanet.ie/pub/linuxmint.com/ http://www.linuxmint.com/mirrors.php || 35 GiB || -
|-
| Linux Mint (packages) || rsync://packages.linuxmint.com/packages/ http://www.linuxmint.com/mirrors.php || 5.0 GiB || -
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6.2 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 99 GiB || -
|-
| MySQL || rsync://mysql.he.net/mysql/ http://dev.mysql.com/downloads/mirrors.html || 212 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 16 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors || 129 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://stage.opensuse.org/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/all.html || 192 GiB || -
|-
| Slackware || rsync://slackware.cs.utah.edu/slackware/ http://slackware.com/getslack/ || 140 GiB || '''We have asked to get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 344 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 445 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 40 GiB || -
|-
| Ubuntu-releases || rsync://rsync.releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 40 GiB || Are the official Canadian mirror (''i.e.'', ca.releases.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5.6 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Proposed Archives to Mirror ==

* Mandriva
* OpenBSD (inc. OpenSSH)
* NetBSD
* PCLinuxOS
* OpenSSL
* RubyForge
* SourceForge
* MacPorts
* PLT Scheme (they don't ask for mirrors, but they currently offer download from half a dozen or so sites)
* VLC

== Implementation Details ==

The mirroring is done by one of three scripts. The latter two are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. [[#merlin|merlin]] is used to call one of these scripts. Most of the scripts and such used to maintain the mirror are available in the public [http://git.csclub.uwaterloo.ca/?p=public/mirror.git mirror.git] repository.

=== ftpsync ===

[http://ftp-master.debian.org/ftpsync.tar.gz ftpsync] is the official Debian mirror synchronization tool, and is used to rsync the Debian repository. It's located in ~mirror/debian. Its invocation takes a few steps (this is more or less how [[#merlin|merlin]] invokes it:

export BASEDIR=/home/mirror/debian
cd $BASEDIR
./bin/ftpsync sync:stage1
./bin/ftpsync sync:stage2

=== csc-sync-debian ===

This is used to sync debian-style repositories. Its usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. Its usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== merlin ===

The synchronization process is run by a Python script called "merlin", written by a2brenna, stored in ~mirror/merlin. The repository list, sync time, etc. is maintained in merlin.py.

=== HTTP ===

We use [[Apache]] as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 64
ThreadLimit 64
StartServers 2
MaxClients 4096
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

==== Index ====

An index of the archives we mirror is available at http://mirror.csclub.uwaterloo.ca/.
As of Winter 2010, it is now generated by a Python script in <tt>~mirror/mirror-index</tt>.

<tt>~mirror/mirror-index/make-index.py</tt> is scheduled in <tt>mirror</tt>'s crontab to be
run at 5:40 AM on the 14th and 28th of each month. The script can be run manually when needed
(for example, when an archive is removed) as follows:

sudo -u mirror /home/mirror/mirror-index/make-index.py

This causes an instance of <tt>du</tt> to be run which computes the size of each directory. This
list is then sorted alphabetically by directory name and returned to the Python script.
If any errors occur during this process, the script conservatively chooses to exit rather
than risk generating an index file that is incorrect.

<tt>make-index.py</tt> is configured by means of a [http://www.yaml.org/ YAML] file,
<tt>config.yaml</tt>, in the same directory. Its format is as follows:

docroot: /mirror/root
duflags: --human-readable --max-depth=1
output: /mirror/root/index.html

directories:
apache:
site: apache.org
url: <nowiki>http://www.apache.org/</nowiki>

archlinux:
site: archlinux.org
url: <nowiki>http://www.archlinux.org/</nowiki>

# (...)

The <tt>docroot</tt> is the directory which is to be scanned; this will probably
always be the mirror root from which Apache serves. <tt>duflags</tt> specifies
the flags to be passed to <tt>du</tt>. This is here so that it's easy to find
and alter. For instance, we could change <tt>--human-readable</tt> to <tt>--si</tt>
if we ever decided that, like hard disk manufacturers, we want sizes to appear larger
than they are. <tt>output</tt> defines the file to which the generated index will be
written.

Finally, <tt>directories</tt> specifies the list of directories to be listed.
No directories not listed here will be shown. If you add a new archive and it doesn't
appear, that's why. The format is fairly straightforward: simply name the directory
and provide a site (the display name in the "Project Site" column) and URL.

One caveat here is that YAML does not allow tabs for whitespace. Indent with
two spaces to remain consistent with the existing file format, please. Also note
that the directory name is case-sensitive, as is always the case on Unix.

Finally, the HTML index file is generated from <tt>index.mako</tt>, a
[http://www.makotemplates.org/ Mako] template (which is mostly HTML anyhow).
If you really can't figure out how it works, look up the Mako documentation.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

[[Category:Services]]
[[Category:Software]]

Mirror

2010-07-11T15:43:44Z

Dtbartle:

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on [[Machine_List#sodium-benzoate|sodium-benzoate]]. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 3.6 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 28 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 30 GiB || -
|-
| Blastwave || rsync://master.rsync.blastwave.org/blastwave/ http://www.blastwave.org/mirrors.php || 13 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 123 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 7.1 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 50 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 17 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 8.7 GiB || -
|-
| Damn Small Linux || rsync://ftp.heanet.ie/mirrors/damnsmalllinux.org/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://ftp3.nrc.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 506 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 39 GiB || Submitted a request to be added to the mirror list long ago with no response.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 76 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 6.1 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 48 GiB || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 45 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://debian-maintainers.org/unofficial/ http://www.debian-unofficial.org/mirrors.html || 465 MiB || -
|-
| Debian-volatile || rsync://volatile-master.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4.2 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 293 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/debian/ || 3.5 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://fedora-archives.ibiblio.org/fedora-enchilada/ http://mirrors.fedoraproject.org/publiclist/ || 910 GiB || -
|-
| <s>FreeBSD</s> || rsync://ftp1.ca.freebsd.org/ http://mirrorlist.freebsd.org/ || 1.2 TiB || Removed to free up space. '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 606 MiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 165 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 95 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 22 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 85 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 140 GiB || -
|-
| Linux Mint (releases) || rsync://ftp.heanet.ie/pub/linuxmint.com/ http://www.linuxmint.com/mirrors.php || 35 GiB || -
|-
| Linux Mint (packages) || rsync://packages.linuxmint.com/packages/ http://www.linuxmint.com/mirrors.php || 5.0 GiB || -
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6.2 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 99 GiB || -
|-
| MySQL || rsync://mysql.he.net/mysql/ http://dev.mysql.com/downloads/mirrors.html || 212 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 16 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors || 129 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://stage.opensuse.org/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/all.html || 192 GiB || -
|-
| Slackware || rsync://slackware.cs.utah.edu/slackware/ http://slackware.com/getslack/ || 140 GiB || '''We have asked to get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 344 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 445 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 40 GiB || -
|-
| Ubuntu-releases || rsync://rsync.releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 40 GiB || Are the official Canadian mirror (''i.e.'', ca.releases.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5.6 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Proposed Archives to Mirror ==

* Mandriva
* OpenBSD (inc. OpenSSH)
* NetBSD
* PCLinuxOS
* OpenSSL
* RubyForge
* SourceForge
* MacPorts
* PLT Scheme (they don't ask for mirrors, but they currently offer download from half a dozen or so sites)
* VLC

== Implementation Details ==

The mirroring is done by one of three scripts. The latter two are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. [[#merlin|merlin]] is used to call one of these scripts. Most of the scripts and such used to maintain the mirror are available in the public [http://git.csclub.uwaterloo.ca/?p=public/mirror.git mirror.git] repository.

=== ftpsync ===

[http://ftp-master.debian.org/ftpsync.tar.gz ftpsync] is the official Debian mirror synchronization tool, and is used to rsync the Debian repository. It's located in ~mirror/debian. Its invocation takes a few steps (this is more or less how [[#merlin|merlin]] invokes it:

export BASEDIR=/home/mirror/debian
cd $BASEDIR
./bin/ftpsync sync:stage1
./bin/ftpsync sync:stage2

=== csc-sync-debian ===

This is used to sync debian-style repositories. Its usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. Its usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== merlin ===

The synchronization process is run by a Python script called "merlin", written by a2brenna, stored in ~mirror/merlin. The repository list, sync time, etc. is maintained in merlin.py.

=== HTTP ===

We use [[Apache]] as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 64
ThreadLimit 64
StartServers 2
MaxClients 4096
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

==== Index ====

An index of the archives we mirror is available at http://mirror.csclub.uwaterloo.ca/.
As of Winter 2010, it is now generated by a Python script in <tt>~mirror/mirror-index</tt>.

<tt>~mirror/mirror-index/make-index.py</tt> is scheduled in <tt>mirror</tt>'s crontab to be
run at 5:40 AM on the 14th and 28th of each month. The script can be run manually when needed
(for example, when an archive is removed) as follows:

sudo -u mirror /home/mirror/mirror-index/make-index.py

This causes an instance of <tt>du</tt> to be run which computes the size of each directory. This
list is then sorted alphabetically by directory name and returned to the Python script.
If any errors occur during this process, the script conservatively chooses to exit rather
than risk generating an index file that is incorrect.

<tt>make-index.py</tt> is configured by means of a [http://www.yaml.org/ YAML] file,
<tt>config.yaml</tt>, in the same directory. Its format is as follows:

docroot: /mirror/root
duflags: --human-readable --max-depth=1
output: /mirror/root/index.html

directories:
apache:
site: apache.org
url: <nowiki>http://www.apache.org/</nowiki>

archlinux:
site: archlinux.org
url: <nowiki>http://www.archlinux.org/</nowiki>

# (...)

The <tt>docroot</tt> is the directory which is to be scanned; this will probably
always be the mirror root from which Apache serves. <tt>duflags</tt> specifies
the flags to be passed to <tt>du</tt>. This is here so that it's easy to find
and alter. For instance, we could change <tt>--human-readable</tt> to <tt>--si</tt>
if we ever decided that, like hard disk manufacturers, we want sizes to appear larger
than they are. <tt>output</tt> defines the file to which the generated index will be
written.

Finally, <tt>directories</tt> specifies the list of directories to be listed.
No directories not listed here will be shown. If you add a new archive and it doesn't
appear, that's why. The format is fairly straightforward: simply name the directory
and provide a site (the display name in the "Project Site" column) and URL.

One caveat here is that YAML does not allow tabs for whitespace. Indent with
two spaces to remain consistent with the existing file format, please. Also note
that the directory name is case-sensitive, as is always the case on Unix.

Finally, the HTML index file is generated from <tt>index.mako</tt>, a
[http://www.makotemplates.org/ Mako] template (which is mostly HTML anyhow).
If you really can't figure out how it works, look up the Mako documentation.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

[[Category:Services]]
[[Category:Software]]

Mirror

2010-07-11T15:43:32Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on [[Machine_List#sodium-benzoate|sodium-benzoate]]. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 3.6 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 28 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 30 GiB || -
|-
| Blastwave || rsync://master.rsync.blastwave.org/blastwave/ http://www.blastwave.org/mirrors.php || 13 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 123 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 7.1 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 50 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 17 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 8.7 GiB || -
|-
| Damn Small Linux || rsync://ftp.heanet.ie/mirrors/damnsmalllinux.org/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://ftp3.nrc.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 506 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 39 GiB || Submitted a request to be added to the mirror list long ago with no response.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 76 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 6.1 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 48 GiB || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 45 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://debian-maintainers.org/unofficial/ http://www.debian-unofficial.org/mirrors.html || 465 MiB || -
|-
| Debian-volatile || rsync://volatile-master.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4.2 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 293 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/debian/ || 3.5 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://fedora-archives.ibiblio.org/fedora-enchilada/ http://mirrors.fedoraproject.org/publiclist/ || 910 GiB || -
|-
| <s>FreeBSD</s> || rsync://ftp1.ca.freebsd.org/ http://mirrorlist.freebsd.org/ || 1.2 TiB || Removed to free up space. '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 606 MiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 165 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 95 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 22 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 85 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 140 GiB || -
|-
| Linux Mint (releases) || rsync://ftp.heanet.ie/pub/linuxmint.com/ http://www.linuxmint.com/mirrors.php || 35 GiB || -
|-
| Linux Mint (packages) || rsync://packages.linuxmint.com/packages/ http://www.linuxmint.com/mirrors.php || 5.0 GiB || -
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6.2 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 99 GiB || -
|-
| MySQL || rsync://mysql.he.net/mysql/ http://dev.mysql.com/downloads/mirrors.html || 212 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 16 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors || 129 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://stage.opensuse.org/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/all.html || 192 GiB || -
|-
| Slackware || rsync://slackware.cs.utah.edu/slackware/ http://slackware.com/getslack/ || 140 GiB || '''We have asked to get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 344 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 445 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 40 GiB || -
|-
| Ubuntu-releases || rsync://rsync.releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 40 GiB || Are the official Canadian mirror (''i.e.'', ca.releases.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5.6 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Proposed Archives to Mirror ==

* Fedora
* Mandriva
* OpenBSD (inc. OpenSSH)
* NetBSD
* PCLinuxOS
* OpenSSL
* RubyForge
* SourceForge
* MacPorts
* PLT Scheme (they don't ask for mirrors, but they currently offer download from half a dozen or so sites)
* VLC

== Implementation Details ==

The mirroring is done by one of three scripts. The latter two are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. [[#merlin|merlin]] is used to call one of these scripts. Most of the scripts and such used to maintain the mirror are available in the public [http://git.csclub.uwaterloo.ca/?p=public/mirror.git mirror.git] repository.

=== ftpsync ===

[http://ftp-master.debian.org/ftpsync.tar.gz ftpsync] is the official Debian mirror synchronization tool, and is used to rsync the Debian repository. It's located in ~mirror/debian. Its invocation takes a few steps (this is more or less how [[#merlin|merlin]] invokes it:

export BASEDIR=/home/mirror/debian
cd $BASEDIR
./bin/ftpsync sync:stage1
./bin/ftpsync sync:stage2

=== csc-sync-debian ===

This is used to sync debian-style repositories. Its usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. Its usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== merlin ===

The synchronization process is run by a Python script called "merlin", written by a2brenna, stored in ~mirror/merlin. The repository list, sync time, etc. is maintained in merlin.py.

=== HTTP ===

We use [[Apache]] as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 64
ThreadLimit 64
StartServers 2
MaxClients 4096
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

==== Index ====

An index of the archives we mirror is available at http://mirror.csclub.uwaterloo.ca/.
As of Winter 2010, it is now generated by a Python script in <tt>~mirror/mirror-index</tt>.

<tt>~mirror/mirror-index/make-index.py</tt> is scheduled in <tt>mirror</tt>'s crontab to be
run at 5:40 AM on the 14th and 28th of each month. The script can be run manually when needed
(for example, when an archive is removed) as follows:

sudo -u mirror /home/mirror/mirror-index/make-index.py

This causes an instance of <tt>du</tt> to be run which computes the size of each directory. This
list is then sorted alphabetically by directory name and returned to the Python script.
If any errors occur during this process, the script conservatively chooses to exit rather
than risk generating an index file that is incorrect.

<tt>make-index.py</tt> is configured by means of a [http://www.yaml.org/ YAML] file,
<tt>config.yaml</tt>, in the same directory. Its format is as follows:

docroot: /mirror/root
duflags: --human-readable --max-depth=1
output: /mirror/root/index.html

directories:
apache:
site: apache.org
url: <nowiki>http://www.apache.org/</nowiki>

archlinux:
site: archlinux.org
url: <nowiki>http://www.archlinux.org/</nowiki>

# (...)

The <tt>docroot</tt> is the directory which is to be scanned; this will probably
always be the mirror root from which Apache serves. <tt>duflags</tt> specifies
the flags to be passed to <tt>du</tt>. This is here so that it's easy to find
and alter. For instance, we could change <tt>--human-readable</tt> to <tt>--si</tt>
if we ever decided that, like hard disk manufacturers, we want sizes to appear larger
than they are. <tt>output</tt> defines the file to which the generated index will be
written.

Finally, <tt>directories</tt> specifies the list of directories to be listed.
No directories not listed here will be shown. If you add a new archive and it doesn't
appear, that's why. The format is fairly straightforward: simply name the directory
and provide a site (the display name in the "Project Site" column) and URL.

One caveat here is that YAML does not allow tabs for whitespace. Indent with
two spaces to remain consistent with the existing file format, please. Also note
that the directory name is case-sensitive, as is always the case on Unix.

Finally, the HTML index file is generated from <tt>index.mako</tt>, a
[http://www.makotemplates.org/ Mako] template (which is mostly HTML anyhow).
If you really can't figure out how it works, look up the Mako documentation.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

[[Category:Services]]
[[Category:Software]]

Mirror

2009-11-27T20:37:13Z

Dtbartle:

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on [[Machine_List#sodium-benzoate|sodium-benzoate]]. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.4 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 23 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 75 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 11 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 131 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6.8 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 41 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 11 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 446 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 30 GiB || Submitted a request to be added to the mirror list long ago with no response.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 70 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5.3 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 36 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 970 MiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 3.3 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 226 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3.7 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager/site/647 rsync://ftp.muug.mb.ca/pub/fedora/linux/core/ || 548 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 673 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 581 MiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 159 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 86 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 20 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 69 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 130 GiB || -
|-
| Linux Mint (releases) || rsync://ftp.heanet.ie/pub/linuxmint.com/ http://www.linuxmint.com/mirrors.php || 23 GiB || -
|-
| Linux Mint (packages) || rsync://packages.linuxmint.com/packages/ http://www.linuxmint.com/mirrors.php || 4.1 GiB || -
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6.0 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 105 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 148 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 102 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://stage.opensuse.org/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/all.html || 161 GiB || -
|-
| Slackware || - || 139 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 286 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 394 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 36 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 39 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5.3 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|-
| wine-budgetdedicated || http://wine.budgetdedicated.com/apt/ || 151 MiB || -
|}

== Proposed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (209 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-22T09:04:14Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list long ago with no response.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager/site/647 rsync://archive.linux.duke.edu/fedora-enchilada/ rsync://archive.linux.duke.edu/fedora-epel/ || 721 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint (releases) || rsync://ftp.heanet.ie/pub/linuxmint.com/ http://www.linuxmint.com/mirrors.php || 21 GiB || -
|-
| Linux Mint (packages) || rsync://packages.linuxmint.com/packages/ http://www.linuxmint.com/mirrors.php || 4 GiB || -
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://stage.opensuse.org/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/all.html || 160 GiB || -
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-22T09:03:55Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list long ago with no response.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager/site/647 rsync://archive.linux.duke.edu/fedora-enchilada/ rsync://archive.linux.duke.edu/fedora-epel/ || 721 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint (releases) || rsync://ftp.heanet.ie/pub/linuxmint.com/ http://www.linuxmint.com/mirrors.php || 21 GiB || '''Sent a request to be added.'''
|-
| Linux Mint (packages) || rsync://packages.linuxmint.com/packages/ http://www.linuxmint.com/mirrors.php || 4 GiB || '''Sent a request to be added.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://stage.opensuse.org/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/all.html || 160 GiB || -
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-20T09:16:24Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list long ago with no response.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager/site/647 rsync://archive.linux.duke.edu/fedora-enchilada/ rsync://archive.linux.duke.edu/fedora-epel/ || 721 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint (releases) || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''Sent a request to be added.'''
|-
| Linux Mint (packages) || rsync://packages.linuxmint.com/packages/ || 4 GiB || '''Sent a request to be added.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://stage.opensuse.org/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/all.html || 160 GiB || -
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-20T08:49:51Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list long ago with no response.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager/site/647 rsync://archive.linux.duke.edu/fedora-enchilada/ rsync://archive.linux.duke.edu/fedora-epel/ || 721 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint (releases) || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''Sent a request to be added.'''
|-
| Linux Mint (packages) || rsync://packages.linuxmint.com/packages/ || 4 GiB || '''Sent a request to be added.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/all.html || 160 GiB || -
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-19T06:52:54Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list long ago with no response.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager/site/647 rsync://archive.linux.duke.edu/fedora-enchilada/ rsync://archive.linux.duke.edu/fedora-epel/ || 721 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint (releases) || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''Sent a request to be added.'''
|-
| Linux Mint (packages) || rsync://packages.linuxmint.com/packages/ || 4 GiB || '''Sent a request to be added.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/all.html || 160 GiB || '''Sent a request to be added.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-19T06:36:43Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list long ago with no response.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager/site/647 rsync://archive.linux.duke.edu/fedora-enchilada/ rsync://archive.linux.duke.edu/fedora-epel/ || 721 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint (releases) || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''Sent a request to be added.'''
|-
| Linux Mint (packages) || rsync://packages.linuxmint.com/packages/ || 4 GiB || '''Sent a request to be added.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/all.html || 160 GiB || '''We should get added as an official mirror.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-19T06:36:30Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list long ago with no response.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager/site/647 rsync://archive.linux.duke.edu/fedora-enchilada/ rsync://archive.linux.duke.edu/fedora-epel/ || 721 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint (releases) || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''Sent a request to be added.'''
|-
| Linux Mint (packages) || rsync://packages.linuxmint.com/packages/ || 4 GiB || '''Sent a request to be added.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We should get added as an official mirror.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-19T06:29:03Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list long ago with no response.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager/site/647 rsync://archive.linux.duke.edu/fedora-enchilada/ rsync://archive.linux.duke.edu/fedora-epel/ || 721 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint (releases) || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''Sent a request to be added.'''
|-
| Linux Mint (packages) || rsync://packages.linuxmint.com/packages/ || 4 GiB || '''Sent a request to be added.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-19T06:18:18Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list long ago with no response.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager/site/647 rsync://archive.linux.duke.edu/fedora-enchilada/ rsync://archive.linux.duke.edu/fedora-epel/ || 721 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint (releases) || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| Linux Mint (packages) || rsync://packages.linuxmint.com/packages/ || 4 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-19T06:15:02Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list long ago with no response.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager/site/647 rsync://archive.linux.duke.edu/fedora-enchilada/ rsync://archive.linux.duke.edu/fedora-epel/ || 721 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint (releases) || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| Linux Mint (packages) || rsync://packages.linuxmint.com/packages/ || ?? GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-19T06:07:54Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list long ago with no response.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager/site/647 rsync://archive.linux.duke.edu/fedora-enchilada/ rsync://archive.linux.duke.edu/fedora-epel/ || 721 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-19T06:04:16Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''Sent a request to be added.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager/site/647 rsync://archive.linux.duke.edu/fedora-enchilada/ rsync://archive.linux.duke.edu/fedora-epel/ || 721 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-18T09:27:21Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager/site/647 rsync://archive.linux.duke.edu/fedora-enchilada/ rsync://archive.linux.duke.edu/fedora-epel/ || 721 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-18T09:24:43Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || http://mirrors.fedoraproject.org/publiclist https://admin.fedoraproject.org/mirrormanager rsync://archive.linux.duke.edu/fedora-enchilada/ rsync://archive.linux.duke.edu/fedora-epel/ || 721 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-18T09:12:32Z

Dtbartle:

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || http://mirrors.fedoraproject.org/publiclist rsync://archive.linux.duke.edu/fedora-enchilada/ rsync://archive.linux.duke.edu/fedora-epel/ || 721 GiB || -
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-18T09:10:14Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || http://mirrors.fedoraproject.org/publiclist rsync://archive.linux.duke.edu/fedora-enchilada/ rsync://archive.linux.duke.edu/fedora-epel/ || 721 GiB || '''Sent a request to be listed.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-18T08:25:54Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 721 GiB || '''Sent a request to be listed.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T12:18:52Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 721 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T11:42:22Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.4 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 721 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || 24 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T09:22:14Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 721 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-ports-releases || rsync://cdimage.ubuntu.com/cdimage/ubuntu-ports/releases/ || ? GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T09:09:46Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 721 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-ports || rsync://ports.ubuntu.com/ubuntu-ports/ || 345 GiB || -
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 38 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 18 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T09:02:02Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 721 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || '''We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.'''
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T09:01:48Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 721 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || We used to be the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); when we get more bandwidth get us added back.
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T09:00:47Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 721 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 253 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com)
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T08:51:53Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 721 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 722 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 146 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 81 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 19 GiB || -
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 83 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 121 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 6 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 127 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 144 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 15 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 112 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 160 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 118 GiB || '''We should get added as an official mirror.'''
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 261 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com)
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T08:25:08Z

Dtbartle:

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca. We also have a [http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 bandwidth graph] you can look at.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 721 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 718 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 92 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 65 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 14 GiB || -
|-
| Gobuntu-releases || rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ || 7 GiB || Canonical does not currently list mirrors.
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 65 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 82 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 5 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 95 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 76 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 8 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 136 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 230 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 27 GiB || -
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 261 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com)
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T08:24:39Z

Dtbartle: /* Monitoring */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 721 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 718 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 92 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 65 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 14 GiB || -
|-
| Gobuntu-releases || rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ || 7 GiB || Canonical does not currently list mirrors.
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 65 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 82 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 5 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 95 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 76 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 8 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 136 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 230 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 27 GiB || -
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 261 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com)
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Monitoring ==
[http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440 Bandwidth graph]

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T08:24:14Z

Dtbartle:

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 721 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 718 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 92 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 65 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 14 GiB || -
|-
| Gobuntu-releases || rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ || 7 GiB || Canonical does not currently list mirrors.
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 65 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 82 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 5 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 95 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 76 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 8 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 136 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 230 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 27 GiB || -
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 261 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com)
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Monitoring ==
[http://cacti.csclub.uwaterloo.ca/graph_image.php?action=view&local_graph_id=1560&rra_id=1&graph_height=120&graph_width=440]

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T08:17:00Z

Dtbartle: /* Propsed Archives to Mirror */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 721 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 718 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 92 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 65 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 14 GiB || -
|-
| Gobuntu-releases || rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ || 7 GiB || Canonical does not currently list mirrors.
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 65 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 82 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 5 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 95 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 76 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 8 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 136 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 230 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 27 GiB || -
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 261 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com)
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T08:16:47Z

Dtbartle: /* Propsed Archives to Mirror */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 721 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 718 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 92 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 65 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 14 GiB || -
|-
| Gobuntu-releases || rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ || 7 GiB || Canonical does not currently list mirrors.
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 65 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 82 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 5 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 95 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 76 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 8 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 136 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 230 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 27 GiB || -
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 261 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com)
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* openSUSE (306 GiB)
* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)
* PCLinuxOS

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T08:15:11Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 721 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 718 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 92 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 65 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 14 GiB || -
|-
| Gobuntu-releases || rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ || 7 GiB || Canonical does not currently list mirrors.
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 65 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 82 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 5 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 95 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 76 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 8 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 136 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://mirrors.opensuse.org/list/11.1.html || 230 GiB || '''We are no longer on the download list - find out why.'''
|-
| Slackware || - || 27 GiB || -
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 261 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com)
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* openSUSE (306 GiB)
* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T08:12:28Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 234 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 721 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 718 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 92 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 65 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 14 GiB || -
|-
| Gobuntu-releases || rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ || 7 GiB || Canonical does not currently list mirrors.
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 65 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 82 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 5 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 95 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 76 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 8 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 136 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://download.opensuse.org/distribution/11.0/iso/dvd/MD5SUMS?mirrorlist || 230 GiB || -
|-
| Slackware || - || 27 GiB || -
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 261 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com)
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* openSUSE (306 GiB)
* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T08:09:42Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 138 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 873 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 718 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 92 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 65 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 14 GiB || -
|-
| Gobuntu-releases || rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ || 7 GiB || Canonical does not currently list mirrors.
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 65 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 82 GiB || -
|-
| Linux Mint || rsync://ftp.heanet.ie/pub/linuxmint.com/ || 21 GiB || '''We should get added as an official mirror.'''
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 5 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 95 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 76 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 8 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 136 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://download.opensuse.org/distribution/11.0/iso/dvd/MD5SUMS?mirrorlist || 230 GiB || -
|-
| Slackware || - || 27 GiB || -
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 261 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com)
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* openSUSE (306 GiB)
* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T08:03:31Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-ports || rsync://ftp.debian-ports.org/debian/ || 124G || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 29 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 138 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 873 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 718 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 92 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 65 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 14 GiB || -
|-
| Gobuntu-releases || rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ || 7 GiB || Canonical does not currently list mirrors.
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 65 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 82 GiB || -
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 5 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 95 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 76 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 8 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 136 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://download.opensuse.org/distribution/11.0/iso/dvd/MD5SUMS?mirrorlist || 230 GiB || -
|-
| Slackware || - || 27 GiB || -
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 261 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com)
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* openSUSE (306 GiB)
* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T08:00:30Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 410 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 46 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 138 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 873 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 718 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 92 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 65 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 14 GiB || -
|-
| Gobuntu-releases || rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ || 7 GiB || Canonical does not currently list mirrors.
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 65 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 82 GiB || -
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 5 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 95 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 76 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 8 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 136 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://download.opensuse.org/distribution/11.0/iso/dvd/MD5SUMS?mirrorlist || 230 GiB || -
|-
| Slackware || - || 27 GiB || -
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 261 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com)
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* openSUSE (306 GiB)
* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T07:57:38Z

Dtbartle: /* Propsed Archives to Mirror */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 316 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 46 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 138 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 873 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 718 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 92 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 65 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 14 GiB || -
|-
| Gobuntu-releases || rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ || 7 GiB || Canonical does not currently list mirrors.
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 65 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 82 GiB || -
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 5 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 95 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 76 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 8 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 136 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://download.opensuse.org/distribution/11.0/iso/dvd/MD5SUMS?mirrorlist || 230 GiB || -
|-
| Slackware || - || 27 GiB || -
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 261 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com)
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* openSUSE (306 GiB)
* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* NetBSD (340 GiB)

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T07:56:51Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 316 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 28 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-cd || rsync://cdimage.debian.org/debian-cd/ || 68 GiB || Mirror only the first CD and DVD, and all small CD's (netinst, business-card, etc...).
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 5 GiB || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 46 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 138 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 873 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 718 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 92 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 65 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 14 GiB || -
|-
| Gobuntu-releases || rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ || 7 GiB || Canonical does not currently list mirrors.
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 65 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 82 GiB || -
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 5 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 95 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 76 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 8 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 136 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://download.opensuse.org/distribution/11.0/iso/dvd/MD5SUMS?mirrorlist || 230 GiB || -
|-
| Slackware || - || 27 GiB || -
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 261 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com)
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* openSUSE (306 GiB)
* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* FreeBSD (500 GiB)
* NetBSD (340 GiB)

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T07:52:19Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 16 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 10 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 18 GiB || '''We should get added as an official mirror.''' Maybe contact the dude listed here: http://www.damnsmalllinux.org/donate.html.
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 316 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 36 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 2 GiB || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 46 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 138 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 873 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 718 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 92 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 65 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 14 GiB || -
|-
| Gobuntu-releases || rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ || 7 GiB || Canonical does not currently list mirrors.
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 65 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 82 GiB || -
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 5 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 95 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 76 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 8 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 136 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://download.opensuse.org/distribution/11.0/iso/dvd/MD5SUMS?mirrorlist || 230 GiB || -
|-
| Slackware || - || 27 GiB || -
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 261 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com)
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* openSUSE (306 GiB)
* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* FreeBSD (500 GiB)
* NetBSD (340 GiB)

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T07:50:21Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 6 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 10 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 5 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 17 GiB || -
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 316 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 36 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 2 GiB || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 46 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 138 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 873 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 718 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 92 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 65 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 14 GiB || -
|-
| Gobuntu-releases || rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ || 7 GiB || Canonical does not currently list mirrors.
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 65 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 82 GiB || -
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 5 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 95 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 76 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 8 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 136 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://download.opensuse.org/distribution/11.0/iso/dvd/MD5SUMS?mirrorlist || 230 GiB || -
|-
| Slackware || - || 27 GiB || -
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 261 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com)
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* openSUSE (306 GiB)
* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* FreeBSD (500 GiB)
* NetBSD (340 GiB)

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.

Mirror

2009-08-13T07:50:03Z

Dtbartle: /* Archives Mirrored */

We currently run a public mirror ([http://mirror.csclub.uwaterloo.ca/ mirror.csclub.uwaterloo.ca]) on sodium-benzoate. We are listed on the ResNet [http://noc.uwaterloo.ca/cn/Stats/resReport "don't count" list] so downloading from our mirror will not count against one's ResNet quota. Requests to mirror a particular distribution or archive should be made to syscom@csclub.uwaterloo.ca.

== Archives Mirrored ==

'''Total Size:''' 4.5 TiB

{| class="wikitable"
!style="width: 12em"| Archive ||style="width: 31em"| Upstream/Listing ||style="width: 5em"| Size || Notes
|-
| Apache || rsync://rsync.us.apache.org/apache-dist/ http://www.apache.org/mirrors/ || 20 GiB || -
|-
| Arch Linux || rsync://mirror.rit.edu/archlinux http://www.archlinux.org/download/ http://wiki.archlinux.org/index.php/Mirrors#Mirror_List || 41 GiB || -
|-
| Blastwave || rsync://www.ibiblio.org/sun-packages/csw/ http://www.blastwave.org/mirrors.php || 10 GiB || -
|-
| CentOS || rsync://us-msync.centos.org/CentOS/ http://mirror-status.centos.org/ || 125 GiB || -
|-
| CPAN || rsync://rsync.nic.funet.fi/CPAN/ http://mirror.cpan.org/ || 5 GiB || -
|-
| CRAN || rsync://cran.r-project.org/CRAN/ || 36 GiB || '''We should get added as an official mirror.'''
|-
| CTAN || rsync://carroll.aset.psu.edu/ctan/ http://www.dante.de/cgi-bin/ctan/list.cgi http://www.dante.de/mirmon/ || 10 GiB || -
|-
| Cygwin || rsync://cygwin.com/cygwin-ftp/ http://www.cygwin.com/mirrors.html || 5 GiB || -
|-
| Damn Small Linux || rsync://ftp.belnet.be/packages/damnsmalllinux/ || 17 GiB || -
|-
| Debian || rsync://gulus.usherbrooke.ca/debian/ http://www.debian.org/mirror/list http://www.de.debian.org/dmc/today/ || 316 GiB || Requested to be added to the ftp.ca.debian.org rotation; will be added if the need arises..
|-
| Debian-backports || rsync://www.backports.org/backports.org/ http://www.backports.org/debian/README.mirrors.html || 36 GiB || Submitted a request to be added to the mirror list.
|-
| Debian-multimedia || rsync://www.debian-multimedia.org/debian/ http://www.debian-multimedia.org/debian-m.php || 2 GiB || -
|-
| Debian-security || rsync://security.debian.org/debian-security/ || 46 GiB || Debian does not currently list debian-security mirrors, and encourages users to use security.debian.org exclusively.
|-
| Debian-unofficial || rsync://ftp.debian-unofficial.org/debian/ http://www.debian-unofficial.org/mirrors.html || 1 GiB || -
|-
| Debian-volatile || rsync://volatile.debian.org/debian-volatile/ http://www.debian.org/volatile/volatile-mirrors || 4 GiB || -
|-
| Eclipse || rsync://download.eclipse.org/eclipseMirror http://www.eclipse.org/downloads/download.php?file=/ || 138 GiB || -
|-
| Emdebian || rsync://www.emdebian.org/emdebian/ || 3 GiB || Emdebian does not currently list mirrors.
|-
| Fedora || rsync://archive.linux.duke.edu/fedora-enchilada/ || 873 GiB || '''We should get added as an official mirror.'''
|-
| FreeBSD || rsync://ftp1.ca.freebsd.org/ || 718 GiB || '''We should get added as an official mirror.'''
|-
| Gentoo (portage) || rsync://rsync1.us.gentoo.org/gentoo-portage/ || 1 GiB || We are rsync4.ca.gentoo.org.
|-
| Gentoo (sources) || rsync://masterdistfiles.gentoo.org/gentoo/ http://www.gentoo.org/main/en/mirrors.xml http://mirrorstats.gentoo.org/ || 92 GiB || See ~sysadmin/passwords/gentoo for rsync password.
|-
| GNOME || rsync://ftp.gnome.org/gnome/ || 65 GiB || -
|-
| GNU || rsync://ftp.ibiblio.org/pub/gnu/ftp/gnu/ http://www.gnu.org/order/ftp.html || 14 GiB || -
|-
| Gobuntu-releases || rsync://cdimage.ubuntu.com/cdimage/gobuntu/releases/ || 7 GiB || Canonical does not currently list mirrors.
|-
| KDE || rsync://master.kde.org/kdeftp/ http://www.kde.org/mirrors/ || 65 GiB || -
|-
| kernel.org || rsync://kernel.org/pub/linux/ rsync://kernel.org/pub/software/ http://kernel.org/mirrors/countries/html/CA.html || 82 GiB || -
|-
| mozdev || rsync://rsync.mozdev.org/mozdev/ http://mirrors.mozdev.org/index.html || 5 GiB || Currently in the North American rotation, but could request to be added to the global rotation.
|-
| mozilla.org || rsync://releases-rsync.mozilla.org/mozilla-releases/ http://www.mozilla.org/mirrors.html || 95 GiB || -
|-
| MySQL || rsync://mysql.mirrors.pair.com/mysql/ http://dev.mysql.com/downloads/mirrors.html || 76 GiB || -
|-
| non-GNU || rsync://dl.sv.gnu.org/releases/ http://dl.sv.gnu.org/releases/00_MIRRORS.html http://download.savannah.gnu.org/mirmon/ || 8 GiB || -
|-
| Openoffice (extended set) || rsync://rsync.services.openoffice.org/openoffice-extended/ http://distribution.openoffice.org/mirrors/#mirrors http://www.ooodev.org/mirmon/ || 136 GiB || -
|-
| OpenSUSE (opensuse-full) || rsync://suse.mirrors.tds.net/opensuse-full/opensuse/ http://download.opensuse.org/distribution/11.0/iso/dvd/MD5SUMS?mirrorlist || 230 GiB || -
|-
| Slackware || - || 27 GiB || -
|-
| Ubuntu || rsync://archives.ubuntu.com/ubuntu/ https://launchpad.net/ubuntu/+archivemirrors || 261 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com)
|-
| Ubuntu-releases || rsync://releases.ubuntu.com/releases/ https://launchpad.net/ubuntu/+cdmirrors http://www.ubuntu.com/getubuntu/download http://www.kubuntu.org/download.php http://www.edubuntu.org/Download || 37 GiB || Are the official Canadian mirror (''i.e.'', ca.archive.ubuntu.com); ubuntu-releases includes Ubuntu, Kubuntu, and Edubuntu.
|-
| xorg.freedesktop.org || rsync://xorg.freedesktop.org/xorg-archive/ http://www.x.org/wiki/Releases/Download || 5 GiB || -
|-
| Xubuntu-releases || rsync://cdimage.ubuntu.com/cdimage/xubuntu/releases/ http://www.xubuntu.org/get || 20 GiB || -
|}

== Propsed Archives to Mirror ==

* openSUSE (306 GiB)
* Mandriva (774 GiB)
* OpenBSD (54 GiB)
* FreeBSD (500 GiB)
* NetBSD (340 GiB)

== Implementation Details ==

The mirroring is done by one of two scripts. Both are based on [http://www.debian.org/mirror/anonftpsync anonftpsync]. Various cronjobs are scheduled to call one of these scripts.

=== csc-sync-debian ===

This is used to sync debian-style repositories. It's usage is:
csc-sync-debian local_dir rsync_host rsync_dir [trace_host [trace_dir]]

If trace_host is specified, then $rsync_dir/project/trace/$trace_host is checked to see if it has changed. If it has, a normal debian-style (two-pass) rsync is done.

=== csc-sync-standard ===

This is used to sync a tree in a general way. Like anonftpsync, it supports locking and logging. It's usage is:

csc-sync-standard local_dir rsync_host rsync_dir

=== Crontab ===

All cronjobs are listed in mirror's crontab. If csc-sync-debian is used, the cronjob is typically run bi-hourly. When using csc-sync-standard, the frequency of the cronjob is typically 12 hours. The crontab currently looks like this:

# m h dom mon dow command

#
# bi-hourly
#
5 */2 * * * ~/bin/csc-sync-debian debian gulus.usherbrooke.ca debian ftp-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-multimedia www.debian-multimedia.org debian marillat.net
5 */2 * * * ~/bin/csc-sync-debian debian-backports www.backports.org backports.org www.backports.org
5 */2 * * * ~/bin/csc-sync-debian debian-volatile volatile.debian.org debian-volatile volatile-master.debian.net
5 */2 * * * ~/bin/csc-sync-debian debian-security security.debian.org debian-security security-master.debian.org
5 */2 * * * ~/bin/csc-sync-debian debian-unofficial ftp.debian-unofficial.org debian ftp-master.debian-unofficial.org
5 */2 * * * ~/bin/csc-sync-debian ubuntu archive.ubuntu.com ubuntu drescher.canonical.com
5 */2 * * * ~/bin/csc-sync-standard ubuntu-releases releases.ubuntu.com releases
5 */2 * * * ~/bin/csc-sync-standard xubuntu-releases cdimage.ubuntu.com cdimage/xubuntu/releases/
5 */2 * * * ~/bin/csc-sync-standard kubuntu-kde4 cdimage.ubuntu.com cdimage/kubuntu-kde4/releases/

#
# daily
#
5 3,15 * * * ~/bin/csc-sync-debian emdebian www.emdebian.org emdebian
5 3,15 * * * ~/bin/csc-sync-standard nexenta nexenta.org repository
5 3,15 * * * ~/bin/csc-sync-standard nexenta-releases nexenta.org releases
5 3,15 * * * ~/bin/csc-sync-standard CPAN rsync.nic.funet.fi CPAN
5 3,15 * * * ~/bin/csc-sync-standard CRAN cran.r-project.org CRAN
5 3,15 * * * ~/bin/csc-sync-standard CTAN carroll.aset.psu.edu ctan
5 3,15 * * * ~/bin/csc-sync-standard openoffice ftp.ussg.iu.edu openoffice
5 4,16 * * * ~/bin/csc-sync-standard cygwin cygwin.com cygwin-ftp
5 4,16 * * * ~/bin/csc-sync-standard gnu ftp.ibiblio.org pub/gnu/ftp/gnu/
5 4,16 * * * ~/bin/csc-sync-standard nongnu dl.sv.gnu.org releases
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/linux kernel.org all/linux/
5 4,16 * * * ~/bin/csc-sync-standard kernel.org/software kernel.org all/software/
5 4,16 * * * ~/bin/csc-sync-standard apache rsync.us.apache.org apache-dist
5 4.16 * * * ~/bin/csc-sync-standard eclipse download.eclipse.org eclipseMirror
5 5,17 * * * ~/bin/csc-sync-standard mysql mysql.he.net mysql
5 5,17 * * * ~/bin/csc-sync-standard kde master.kde.org kdeftp
5 5,17 * * * ~/bin/csc-sync-standard mozdev rsync.mozdev.org mozdev
5 5,17 * * * ~/bin/csc-sync-standard blastwave www.ibiblio.org sun-packages/csw/
5 5,17 * * * ~/bin/csc-sync-standard archlinux mirror.rit.edu archlinux
5 5,17 * * * ~/bin/csc-sync-standard debian-ports ftp.debian-ports.org debian
5 5,17 * * * ~/bin/csc-sync-debian-cd
5 6,18 * * * ~/bin/csc-sync-standard x.org xorg.freedesktop.org xorg-archive
5 6,18 * * * ~/bin/csc-sync-standard gnome ftp.gnome.org gnome
5 6,18 * * * ~/bin/csc-sync-standard centos us-msync.centos.org CentOS
5 6,18 * * * ~/bin/csc-sync-standard opensuse suse.mirrors.tds.net opensuse-full/opensuse/
5 6,18 * * * ~/bin/csc-sync-standard damnsmalllinux ftp.belnet.be packages/damnsmalllinux/

#
# other
#
29 */4 * * * RSYNC_USER=gentoo RSYNC_PASSWORD=******** ~/bin/csc-sync-standard gentoo-distfiles masterdistfiles.gentoo.org gentoo
15,45 * * * * ~/bin/csc-sync-standard gentoo-portage rsync1.us.gentoo.org gentoo-portage
5,35 * * * * ~/bin/csc-sync-standard mozilla.org releases-rsync.mozilla.org mozilla-releases

=== HTTP ===

We use Apache as our web server. Here's a snippet of the worker configuration:

<IfModule mpm_worker_module>
ServerLimit 32
ThreadLimit 64
StartServers 2
MaxClients 2048
MinSpareThreads 16
MaxSpareThreads 48
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

We use the bwbar application to display current bandwidth in the footer of mirror pages.

We use mod_bw to ensure every connection is at least 100 KiB/s.

=== FTP ===

We use proftpd (standalone daemon) as our ftp server. To increase performance we disable DNS lookups in proftpd.conf:

UseReverseDNS off
IdentLookups off

We also limit the amount of CPU/memory resources used (e.g. to minimize [http://en.wikipedia.org/wiki/Globbing Globbing] resources):

RLimitCPU session 10
RLimitMemory session 4096K

We allow a maximum of 200 concurrent ftp sessions:

MaxInstances 500
MaxClients 500

=== Rsync ===

We use rsyncd (standalone daemon). We disable compression and checksumming in rsyncd.conf:

dont compress = *
refuse options = c delete

For ftp and rsync, the contents of /mirror/root/include/motd.msg are displayed when users connect.