The basic steps for setting up a NetBSD FTP Mirror are: Prepare disk space. See "How much disk space is needed fora FTP mirror" entry below for information on amount of space required. Retrieve a copy of the NetBSD FTP site contents Set things up to synchronize the contents of your mirror with ftp.NetBSD.org. Send in the information about your mirror to mirrors@NetBSD.org. See "Contact information format for FTP Mirrors" for details on that. The amount of disk space needed fluctuates due to things such as pre-built packages being added, binary snapshots being deprecated, and new formal releases of NetBSD. At this time (September 2004) a complete mirror of NetBSD needs about 150 GB and a new release of NetBSD adds between 10 and 15 GB (including binary packages). There are several ways that you can retrieve and sync the files. Depending on your personal preference, your network connection, the packages installed on your systems etc. you may choose one over the other. Either way, it's probably a good idea to first run the relevant command by hand and then, when you have confirmed that everything is working as it should, create a script to be run by &man.cron.8;. Below you can find some examples of how to keep your local copy in sync. Simply replace "<FTPROOT>" with the actual root of your ftp server. Using net/rsync: $ rsync -v -Wartz --delete rsync://rsync.NetBSD.org/NetBSD <FTPROOT>/pub/NetBSD/ Using net/wget: $ cd <FTPROOT> $ wget -v -xnH -m ftp://ftp.NetBSD.org/pub/NetBSD/ Using net/mirror: $ cat > NetBSD.org <<EOF package=netbsd comment=NetBSD site=ftp.NetBSD.org remote_dir=/pub/NetBSD local_dir=<FTPROOT>/pub/NetBSD do_deletes=true algorithm=1 EOF $ mirror NetBSD.org As a NetBSD FTP mirror, it's important that you synchronize your copy of the NetBSD FTP tree regularly. Otherwise, old and/or obsolete files may confuse users. Also, it might happen that due to an e.g. security issue, file removal (or another administrative task) may be requested from the NetBSD Project. If your FTP mirror experiences any significant system outage, you should notify mirrors@NetBSD.org as soon as possible so that we can temporarily redirect your CNAME (if there is one) to a nearby alternative. Also, you may wish to send a note about the outage to the netbsd-users mailing list. If the CNAME for your mirror has been redirected, be sure to let us know when the situation is rectified, so we can revert the change. Contact information needs to be submitted to mirrors@NetBSD.org. This information gets added to an internal mirrors database, from which the mirror-related documentation gets generated. The "contact*" fields are never made public. Contact information for NetBSD FTP mirrors should be sent in the following format: mirror *machine name* country *location (country) of host* location *more specific location (e.g, ``University of Foo'')* ftp ftp://ftp.foo-univ.ac.jp/pub/NetBSD/ ftp-host ftp.foo-univ.ac.jp ftp-releng ftp://ftp.foo-univ.ac.jp/pub/NetBSD-daily/ organisation *organisation name* contactperson *contact person's name* contactemail *contact person's electronic mail address* contactkeyid *contact person's pgp key id* contactphone *contact person's telephone number* contactaddress *contact person's postal address* An example submission might look like this. mirror netbsd.foo.com country Japan location Department of Computer Science, Foo University ftp ftp://ftp.foo-univ.ac.jp/pub/NetBSD/ ftp-host ftp.foo-univ.ac.jp organisation Foo University contactperson Foo Bar contactemail Bar@foo-univ.ac.jp contactkeyid GA23FG12 contactphone +81 3 xxxx yyyy contactaddress 9876 Foo City, Tokyo 1234567 Japan If the contact information for your mirror ever needs to be changed, please feel free to let us know at mirrors@NetBSD.org. If you would like a ftp.country-code (e.g. ftp.jp) CNAME for your mirror, please ask mirrors@NetBSD.org. We do not normally do subdomain delegation, so you cannot run a nameserver for "country-code.NetBSD.org". Please let us know at mirrors@NetBSD.org so that we can remove your site from our list and redirect the DNS name to an alternative site. BitTorrent is a tool for distributing files. It's extremely easy to use - downloads are started by clicking on hyperlinks. Whenever more than one person is downloading at once they send pieces of the file(s) to each other, thus relieving the central server's bandwidth burden. Even with many simultaneous downloads, the upload burden on the central server remains quite small, since each new downloader introduces new upload capacity. The basic steps for setting up a NetBSD BitTorrent Mirror are: Install net/bittorrent Fetch the .torrent files from our FTP site. The files are in the /pub/NetBSD/iso/<release> subdirectory, so if you already are mirroring this directory, you're all set. For more information see this page. Start BitTorrent: cd /ftp/pub/NetBSD/iso/<release>; (btlaunchmany.py .) >/dev/null 2>/dev/null &. If you didn't download the ISO images already, they will be downloaded now. You can use the --max_upload_rate arg and --max_download_rate arg options to limit the transfer rates. Contact information needs to be submitted to mirrors@NetBSD.org. This information gets added to an internal mirrors database, from which the mirror-related documentation gets generated. The "contact*" fields are never made public. Contact information for NetBSD BitTorrent mirrors should be sent in the following format: mirror *machine name* country *location (country) of host* location *more specific location (e.g, ``University of Foo'')* bittorrent *BitTorrent URL / URL with additional Information* ipv6 *yes or no, if your server supports IPv6* organisation *organisation name* contactperson *contact person's name* contactemail *contact person's electronic mail address* contactkeyid *contact person's pgp key id* contactphone *contact person's telephone number* contactaddress *contact person's postal address* An example submission might look like this. mirror ftp.NetBSD.org country USA location Silicon Valley, California bittorrent http://www.NetBSD.org/mirrors/torrents/info.html ipv6 yes organisation The NetBSD Foundation, Inc. contactperson Admins contactemail <email-here>@NetBSD.org contactkeyid AF226A4C contactphone 555-123-1234 contactaddress Somwhere in Silicon Valley, California The following is an overview of the recommended procedure for setting up a NetBSD WWW mirror: For setting up a mirror with this procedure, you'll need cvs, ssh, and wget. Both cvs and ssh ship with NetBSD versions after 1.5, and wget is available from the NetBSD packages collection (see net/net/wget).Of course you'll also need a web server, such as apache. Retrieve a copy of the NetBSD WWW site. See "Retrieving a copy of the NetBSD WWW site" for instructions on how to do so with anoncvs. Set things up to synchronize the contents of your mirror with www.NetBSD.org. See "Syncing the NetBSD WWW site contents" for an example of how to do so using a cron job. Send in the information about your mirror to mirrors@NetBSD.org. See "Contact information format for WWW Mirrors" for details on that. The amount of diskspace needed for a complete copy of the htdocs repository is currently (2004-03) around 85 MB. The total amount should not vary too much, but will obviously grow with time. To retrieve a copy of www.NetBSD.org via anoncvs, you should do: # su - nbwww # cd /path/to/htdocs # env CVS_RSH=ssh CVSROOT=anoncvs@anoncvs.NetBSD.org:/cvsroot \ cvs checkout -P htdocs Note: "/path/to/htdocs" should be where you want the mirror of www.NetBSD.org to reside. The directory should be owned by an e.g. "nbwww" user. The best way to do this is to set up a cron job for your e.g. nbwww user to update the web site at a regular interval using anoncvs and wget (more on the wget usage below): # crontab -u nbwww -e SHELL=/bin/sh PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/pkg/bin MAILTO=whoever@gets.the.mirror.mail CVS_RSH=ssh CVSROOT=anoncvs@anoncvs.NetBSD.org:/cvsroot 0 1 * * * cd /path/to/htdocs && cvs update -dP && wget -nv -xnH -N -Fi mirrors/fetch.html && wget -nv -xnH -N -r -np http://www.NetBSD.org/Gnats/ Note: In the suggested contents, you'll want to change "MAILTO" to reflect who should get the mirror-related mail. Also, "/path/to/htdocs" should be changed to where your www.NetBSD.org mirror resides. The fetch.html file links to the handful of documents on www.NetBSD.org that are generated by www.NetBSD.org. These files get generated daily from content in the doc module of the NetBSD CVS repository. In the example cron tab above, you'll see "wget -nv -xnH -NFi mirrors/fetch.html". This means that daily, through wget, these www.NetBSD.org created files will be updated in your copy of the web site contents. It is much preferred that NetBSD WWW mirror sites run a separate virtual web server for a www.NetBSD.org mirror on a separate IP address. This is so that clients using HTTP/1.0 still get to the correct web site. If that's not possible in your set up, then the other solution for HTTP/1.0 clients would be putting the NetBSD mirror as the primary name-based virtual host. Contact information needs to be submitted to mirrors@NetBSD.org. This information gets added to an internal mirrors database, from which the mirror-related documentation gets generated. The "contact*" fields are never made public. Contact information for NetBSD WWW mirrors should be sent in the following format: mirror *machine name* country *location (country) of host* location *more specific location (e.g, ``University of Foo'')* www *WWW URL* www-host *external WWW name* organisation *organisation name* contactperson *contact person's name* contactemail *contact person's electronic mail address* contactkeyid *contact person's pgp key id* contactphone *contact person's telephone number* contactaddress *contact person's postal address* An example submission might look like this. mirror netbsd.foo-univ.ac.jp country Japan location Department of Computer Science, Foo University www http://netbsd.foo-univ.ac.jp/ www-host netbsd.foo-univ.ac.jp organisation Foo University contactperson Foo Bar contactemail Bar@foo-univ.ac.jp contactkeyid GA23FG12 contactphone +81 3 xxxx yyyy contactaddress 9876 Foo City, Tokyo 1234567 Japan If the contact information for your mirror ever needs to be changed, please feel free to let us know at mirrors@NetBSD.org. If you would like a www.country-code (e.g. www.jp) CNAME for your mirror, please ask mirrors@NetBSD.org. We do not normally do subdomain delegation, so you cannot run a nameserver for "country-code.NetBSD.org". Please let us know at mirrors@NetBSD.org so that we can remove your site from our list and redirect the DNS name to an alternative site. The following is an overview of the recommended procedure for setting up a NetBSD CVSweb mirror: For setting up a mirror with this procedure, you'll need cvs, cvsup, and the cvsweb CGI script. NetBSD 1.6 and above ship with CVS, both cvsup and the cvsweb CGI script are available from the NetBSD packages collection (see devel/cvsup and www/cvsweb). Of course you'll also need a web server, such as www/apache. Retrieve a copy of the NetBSD CVS tree. See "Retrieving a copy of the NetBSD CVS tree" for instructions on how to do so with cvsup. Set things up to synchronize the contents of your mirror with the cvsup server. See "Syncing the NetBSD CVS tree contents" for an example of how to do so using a cron job. Send in the information about your mirror to mirrors@NetBSD.org. See "Contact information format for CVSweb Mirrors" for details on that. The amount of disk space needed is rather large due to the fact that you are providing an interface to the entire CVS repository. Currently (2004-03), the total disk space required for all sources in CVS is around 2.5 GB. To retrieve a copy of the CVS tree via cvsup, you should do: # su - cvsweb # mkdir -p /home/cvsweb/netbsd # cd /home/cvsweb Note: "/home/cvsweb/netbsd" should be where you want the mirror of the tree to reside. The directory should be owned by an e.g. "cvsweb" user. Create an "supfile" using your editor (For examples, see netbsd-supfile). Please take a look at the list of available public cvsup servers, you'll find the list here. # cvsup -g -L 2 netbsd-supfile The best way to do this is to set up a cron job for your e.g. cvsweb user to update the tree at a regular interval using cvsup: # crontab -u cvsweb -e SHELL=/bin/sh PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/pkg/bin MAILTO=whoever@gets.the.mirror.mail 0 1 * * * cvsup -g /home/cvsweb/netbsd-supfile Note: In the suggested contents, you'll want to change "MAILTO" to reflect who should get the mirror-related mail. Take a look at the cvsweb.conf and be sure to edit the following options: @CVSrepositories = ( 'netbsd' => ['NetBSD', '/home/cvsweb/netbsd'], ); $mancgi = "http://netbsd.gw.com/cgi-bin/man-cgi?%s+%s+NetBSD-current"; Place cvsweb.cgi in your web servers' cgi-bin directory and be sure to edit it to point to your cvsweb.conf. It is much preferred that NetBSD CVSweb mirror sites run a separate virtual web server for a CVSweb mirror on a separate IP address. This is so that clients using HTTP/1.0 still get to the correct web site. If that's not possible in your set up, then the other solution for HTTP/1.0 clients would be putting the NetBSD mirror as the primary name-based virtual host. Contact information needs to be submitted to mirrors@NetBSD.org. This information gets added to an internal mirrors database, from which the mirror-related documentation gets generated. The "contact*" fields are never made public. Contact information for NetBSD CVSweb mirrors should be sent in the following format: mirror *machine name* country *location (country) of host* location *more specific location (e.g, ``University of Foo'')* cvsweb *CVSweb URL* cvsweb-host *external CVSweb name* organisation *organisation name* contactperson *contact person's name* contactemail *contact person's electronic mail address* contactkeyid *contact person's pgp key id* contactphone *contact person's telephone number* contactaddress *contact person's postal address* An example submission might look like this. mirror netbsd.foo-univ.ac.jp country Japan location Department of Computer Science, Foo University cvsweb http://netbsd.foo-univ.ac.jp/cgi-bin/cvsweb.cgi/ cvsweb-host cvsweb.netbsd.foo-univ.ac.jp organisation Foo University contactperson Foo Bar contactemail Bar@foo-univ.ac.jp contactkeyid GA23FG12 contactphone +81 3 xxxx yyyy contactaddress 9876 Foo City, Tokyo 1234567 Japan If the contact information for your mirror ever needs to be changed, please feel free to let us know at mirrors@NetBSD.org. If you would like a cvsweb.country-code (e.g. cvsweb.jp) CNAME for your mirror, please ask mirrors@NetBSD.org. We do not normally do subdomain delegation, so you cannot run a nameserver for "country-code.NetBSD.org". Please let us know at mirrors@NetBSD.org so that we can remove your site from our list and redirect the DNS name to an alternative site.