Skip to main content.
home | support | download

Back to List Archive

RE: Server or documet limit on spider.pl

From: Aaron Bazar <aaronb(at)not-real.spamcop.net>
Date: Wed Jan 07 2004 - 18:40:56 GMT
Come to think of it, you are probably right. I have not seen the issue in a
long time, and in the meanwhile I am sure I have updated LWP to the newest
version. Anyway, my main point is that the spider definitely is not limited.
It is actually quite versatile.

Thanks,

Aaron Bazar
http://www.whynotgetfit.com

-----Original Message-----
From: swish-e@sunsite.berkeley.edu
[mailto:swish-e@sunsite.berkeley.edu]On Behalf Of Bill Moseley
Sent: Wednesday, January 07, 2004 1:25 PM
To: Multiple recipients of list
Subject: [SWISH-E] RE: Server or documet limit on spider.pl


On Wed, Jan 07, 2004 at 10:14:02AM -0800, Aaron Bazar wrote:
> FYI,
>
> I have used spider.pl to grab over 100,000 documents over a period of 16
> hours. I have, at other times, run into some issues where spider.pl starts
> using up a huge chunk of memory. I never was able to determine why it
> happened sometimes and not others.

Maybe a different issue, but there was a problem a while back where the
spider with some versions of LWP didn't free memory.  This was discussed
on both the swish-e list and the LWP list.  Was it possible you were
using different machines or different versions of LWP?


--
Bill Moseley
moseley@hank.org
Received on Wed Jan 7 18:41:06 2004