On Sep 15, 2004, at 1:35 PM, Bill Moseley wrote:

> On Wed, Sep 15, 2004 at 01:28:23PM -0700, Richard Morin wrote:
>> Clearly, some program found "Sutton" somewhere and wasn't happy,
>> but this isn't enough information to allow the user to debug
>> anything.  Could we have a more comprehensive error message?
>
> No, sorry.  That's not a message generated from any of the code we
> control -- rather from (I suspect) the module that parsers the
> robots.txt file.  Do you have an invalid line in your robots.txt file
> with the word "Sutton"?

The spidering script certainly knows where it's looking, at
any given time.  Does the module not return a status code?  Sigh.

In any case, your question about my robots.txt file presumes
that I have a robots.txt file (as far as I know, I do not).
If I had the URL, I'd know where to start looking...

-r