Skip to main content.
home | support | download

Back to List Archive

RE: Server or documet limit on spider.pl

From: <redna(at)not-real.euskalerria.org>
Date: Fri Jan 09 2004 - 11:24:06 GMT
I've updated spider.pl from a just downloaded version and the problem has
been fixed.

The matter was that when spider.pl was making the summary for a server(a
normal server) it crashed.
Now it's allright.

Thanks.



>Come to think of it, you are probably right. I have not seen the issue in
a
>long time, and in the meanwhile I am sure I have updated LWP to the newest
>version. Anyway, my main point is that the spider definitely is not
limited.
>It is actually quite versatile.
>
>Thanks,
>
>Aaron Bazar
>http://www.whynotgetfit.com
>
>-----Original Message-----
>From: swish-e@sunsite.berkeley.edu
>[mailto:swish-e@sunsite.berkeley.edu]On Behalf Of Bill Moseley
>Sent: Wednesday, January 07, 2004 1:25 PM
>To: Multiple recipients of list
>Subject: [SWISH-E] RE: Server or documet limit on spider.pl
>
>
>On Wed, Jan 07, 2004 at 10:14:02AM -0800, Aaron Bazar wrote:
>> FYI,
>>
>> I have used spider.pl to grab over 100,000 documents over a period of 16
>> hours. I have, at other times, run into some issues where spider.pl
starts
>> using up a huge chunk of memory. I never was able to determine why it
>> happened sometimes and not others.
>
>Maybe a different issue, but there was a problem a while back where the
>spider with some versions of LWP didn't free memory.  This was discussed
>on both the swish-e list and the LWP list.  Was it possible you were
>using different machines or different versions of LWP?
>
>
>--
>Bill Moseley
>moseley@hank.org
>
>
_________________________________________________________
Txat euskalduna >>> http://www.euskalerria.org/solasgunea
Received on Fri Jan 9 11:24:18 2004