On Thu, May 06, 2004 at 11:19:13AM -0700, Mark Jordan wrote:
> Can anyone tell me how to configure spider.pl so that it doesn't
> follow broken links?
It has to follow them to find out they are broken.
> Am I on the right track, is there a better way to make spider.pl not
> follow broken links, or at least not index 404 not found response
Are you 404 pages indeed returning a status of 404?
Can you provide any URLs to test?
Received on Thu May 6 11:27:22 2004