Sorry, can't find this in the docs or discussion list anyplace:
I need to set up robots.txt to disallow everybody except SWISH from
spidering my server. I'm using spider.pl called through S-prog. What is
the user-agent name for the SWISH spider?
Here's what I thought robots.txt would look like -- but this makes SWISH
abort as soon as it starts spidering:
User-agent: spider.pl
Disallow:
User-agent: *
Disallow: /
What's the appropriate "name" for spider.pl when run via S-prog?
Thanks --
David VanHook
Received on Thu Jul 24 19:21:12 2003