Skip to main content.
home | support | download

Back to List Archive

Delay in spidering?

From: François Tissandier <francois.tissandier(at)>
Date: Thu Dec 08 2005 - 11:48:18 GMT
Hello everybody

I have a probably simple problem to solve:

I'm spidering my content too quickly, I don't want my server to be at
100%. The delay_sec is not the solution as i'm using the keep_alive
function. I read that I could enter a delay in the test_url function,
but as my perl level is close to 0, I didn't find the solution. I tried
a "sleep(0.5)", but it's not working. I need a very small delay, under a
second, probably 0.2s or 0.1s. But it seems that sleep doesn't take any
delay under a second... 

Thanks in advance if you have some solution for this problem.

Received on Thu Dec 8 03:48:39 2005