>On Wed, Sep 15, 2004 at 01:37:50PM -0700, Richard Morin wrote:
>> If I had the URL, I'd know where to start looking...

At 01:57 PM 9/15/2004, Bill Moseley wrote:
>Well that's easy: it's http://<yourserver>/robots.txt

It's not my discussion, but let me make an educated guess...
Richard is saying he thinks SWISH should print the URL of
the file it was attempting to spider when the problem occurred.
I tend to agree with him. 

That's why he said this:

At 01:43 PM 9/15/2004, Richard Morin wrote:
>The spidering script certainly knows where it's looking, at
>any given time.  Does the module not return a status code?  Sigh.

In the commercial software I've written I found it useful to have 
every nested level print something and return an error code as a 
low-level error ripples up to the calling tool aborting. That 
REALLY helps with debugging, and it REALLY cuts down on the number 
of support calls I get. People need to know what file was being 
worked on, what was being done, etc. I even have a verbosity level
that cuts down on warnings and stroking for large jobs. Nothing
new about that, eh?

Not a complaint. Just clarification. Perhaps an enhancement request.

Steve