Skip to main content.
home | support | download

Back to List Archive

Re: Checking for 404 responses in header

From: Bill Moseley <moseley(at)not-real.hank.org>
Date: Thu May 06 2004 - 18:27:22 GMT
On Thu, May 06, 2004 at 11:19:13AM -0700, Mark Jordan wrote:
> Hi,
> 
> Can anyone tell me how to configure spider.pl so that it doesn't
> follow broken links?

It has to follow them to find out they are broken.

> Am I on the right track, is there a better way to make spider.pl not
> follow broken links, or at least not index 404 not found response
> pages?

Are you 404 pages indeed returning a status of 404?

Can you provide any URLs to test?

Thanks,

-- 
Bill Moseley
moseley@hank.org
Received on Thu May 6 11:27:22 2004