The NetBSD Project

CVS log for htdocs/robots.txt

[BACK] Up to [] / htdocs

Request diff between arbitrary revisions

Default branch: MAIN

Revision 1.3 / (download) - annotate - [select for diffs], Tue Dec 24 02:57:14 2013 UTC (5 years, 8 months ago) by mspo
Branch: MAIN
Changes since 1.2: +1 -0 lines
Diff to previous 1.2 (unified)

also disallow donations/

Revision 1.2 / (download) - annotate - [select for diffs], Tue Jan 15 15:28:45 2008 UTC (11 years, 8 months ago) by david
Branch: MAIN
Changes since 1.1: +1 -0 lines
Diff to previous 1.1 (unified)

Ask Google not to index xml (source) files, because returning them as
search results is bad for users.

Note - a wildcard in the disallow statement is not valid robots.txt
usage, but Google supports it anyway.

Revision 1.1 / (download) - annotate - [select for diffs], Thu May 22 18:30:16 2003 UTC (16 years, 4 months ago) by grant
Branch: MAIN
CVS Tags: hrs-xmlcleanup-base, hrs-xmlcleanup

attempt to ask robots to not crawl /cgi-bin/.

This form allows you to request diff's between any two revisions of a file. You may select a symbolic revision name using the selection box or you may type in a numeric name using the type-in text box.

CVSweb <>