The current spider.pl documents say:

=item keep_alive

..

"When a connection is not closed the spider does not
wait the "delay_sec" time when making the next
request.  In other words, there is no delay in
requesting documents while the connection is open."

Seems to me that the delay_sec should be respected
even if the connection is keep-alive. Just because a
connection to a server is kept alive doesn't mean that
fetching pages doesn't cause a load on the server!

Perhaps a separate keep_alive_delay_sec could be used?
Or just have spider.pl keep using the delay_sec
anyway?

Interested to hear what others think!
jrobinson


		
__________________________________ 
Do you Yahoo!? 
All your favorites on one personal page – Try My Yahoo!
http://my.yahoo.com