On Thu, May 06, 2004 at 11:19:13AM -0700, Mark Jordan wrote:
> Hi,
> 
> Can anyone tell me how to configure spider.pl so that it doesn't
> follow broken links?

It has to follow them to find out they are broken.

> Am I on the right track, is there a better way to make spider.pl not
> follow broken links, or at least not index 404 not found response
> pages?

Are you 404 pages indeed returning a status of 404?

Can you provide any URLs to test?

Thanks,

-- 
Bill Moseley
moseley@hank.org