On Wed, Sep 15, 2004 at 01:37:50PM -0700, Richard Morin wrote:
> The spidering script certainly knows where it's looking, at
> any given time. Does the module not return a status code? Sigh.
perldoc LWP::RobotUA for how it works. Like I said, it's not
spider.pl that's fetching that file. It would be nice if that module
gave a better message -- you could bring that up on the LWP list as
it only makes sense if you understand the context of the message.
> In any case, your question about my robots.txt file presumes
> that I have a robots.txt file (as far as I know, I do not).
> If I had the URL, I'd know where to start looking...
Well that's easy: it's http://<yourserver>/robots.txt
Unsubscribe from or help with the swish-e list:
Help with Swish-e:
Received on Wed Sep 15 13:59:44 2004