Skip to main content.
home | support | download

Back to List Archive

Re: tracking the completeness of spidering?

From: Bill Conlon <bill(at)>
Date: Thu Dec 16 2004 - 21:08:20 GMT
You probably don't need the headers, but you can find out whats 
happening  by placing


in your @servers


On Thursday, December 16, 2004, at 11:25  AM, Richard Morin wrote:

> Some pages seem to be missing from my indexes, so I'm
> wondering if the spidering process is failing in some
> silent manner.
> I have considered adding some code to spider.config
> (e.g., to tally visited pages and report the results),
> but before I go there, I thought I should ask if there
> is an easier way.  Clues?  Suggestions?
> -r
Received on Thu Dec 16 13:08:26 2004