You probably don't need the headers, but you can find out whats 
happening  by placing

         debug => DEBUG_URL | DEBUG_SKIPPED | DEBUG_FAILED | 
DEBUG_HEADERS,

in your @servers

bill

On Thursday, December 16, 2004, at 11:25  AM, Richard Morin wrote:

> Some pages seem to be missing from my indexes, so I'm
> wondering if the spidering process is failing in some
> silent manner.
>
> I have considered adding some code to spider.config
> (e.g., to tally visited pages and report the results),
> but before I go there, I thought I should ask if there
> is an easier way.  Clues?  Suggestions?
>
> -r
>