I am running swish-e 2.4.3 on a redhat linux box. I am using the
included spider.pl script to spider my website.
My problem: When i enable the keep_alive directive of the spider program
and set the delay_sec to 5, the spider fetches the pages at blazing
speed ignoring the delay_sec directive, and after going through around
5000 pages it then catches up on all the delay, it stops fetching any
more pages and just keeps sleeping for 5 seconds each. After a long wait
it continues from where it left off.
I am not sure if anyone else is facing similar problems, and if anyone
has a work around to using the keep_alive directive and also obey the
delay_sec directive. Any help is appreciated.
Thanks in advance,
Received on Tue Jul 5 09:14:54 2005