Skip to main content.
home | support | download

Back to List Archive

Re: Disallow in Robots.txt

From: Bill Moseley <moseley(at)not-real.hank.org>
Date: Tue Jan 09 2007 - 13:34:22 GMT
On Tue, Jan 09, 2007 at 02:19:51AM -0800, James wrote:
> Brad,
> 
> It appears that you were right, in part, after all.
> 
> changed: $server->{agent} ||= 'swish-e spider 2.2 http://swish-e.org/';
> to: $server->{agent} ||= 'MyTitle http://mywebsite.com/';
> 
> changed: $ua->agent( "SwishSpider http://swish-e.org" );
> To: $ua->agent( "MyTitle" );
> 
> It took a bit of poking around, but I think I got it.  I'll begin testing on
> it to see that it actually works as expected.


Only a bit:

    swish-e.org -> documentation -> spider.pl -> configuration options

But doesn't get you very much info:

    # agent

    This optional key sets the name of the spider.

But it does basically answer your question.

    "is there a way to declare what bot I am?"

Which is:

    agent   => 'MyAgent',


And, if you don't top-post the list archives will be more friendly to others.

-- 
Bill Moseley
moseley@hank.org

Unsubscribe from or help with the swish-e list: 
   http://swish-e.org/Discussion/

Help with Swish-e:
   http://swish-e.org/current/docs
   swish-e@sunsite.berkeley.edu
Received on Tue Jan 9 05:34:27 2007