Skip to main content.
home | support | download

Back to List Archive

Re: robots.txt File

From: Dave Stevens <dstevens(at)not-real.roaddog.com>
Date: Thu Apr 26 2001 - 05:12:39 GMT
>I recently received a FILE NOT FOUND error on the server log for a
>robots.txt file.
>
>Any clues to why SWISH-e is looking for this file?


A robots.txt file is a file you put on your site to control the behavior of
Web spiders, or robots.  For more info on robots.txt files work, surf
http://www.robotstxt.org/ .  Well behaved spiders look for the file to see
if it's OK to index the site.

FWIW, I'd suggest upgrading to 2.x if at all possible.  The difference is
astounding, in terms of speed and stability.

Dave
Received on Thu Apr 26 05:14:32 2001