Skip to main content.
home | support | download

Back to List Archive

Re: spidering, delay_sec and keep_alive

From: J Robinson <jrobinson852(at)not-real.yahoo.com>
Date: Tue Sep 27 2005 - 16:53:19 GMT
--- Bill Moseley <moseley@hank.org> wrote:

> On Tue, Sep 27, 2005 at 06:54:01AM -0700, J Robinson
> wrote:
> > Hello Swisheans:
> > 
> > I've noticed that spider.pl in swish-e won't use
> > $server->{delay_sec} if the spider uses keepalive
> > connections to the web server.
> > 
> > I'd prefer to have spider.pl respect the delay_sec
> > setting even if using keepalive connections.
> 
> Why?

Because I don't want to slam the server, even if I am
using keepalives to minimize the impact. Retrieving
pages off the server could still cause significant
load (ie, dynamic pages). If I wanted to hit pages as
fast as possible, I'd set delay_sec to 0! :)



		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com
Received on Tue Sep 27 09:53:25 2005