Skip to main content.
home | support | download

Back to List Archive

Re: Segmentation fault with latest code on

From: Bill Moseley <moseley(at)not-real.hank.org>
Date: Tue Sep 04 2001 - 12:40:33 GMT
At 01:54 AM 9/4/2001 -0700, Gavin Walker wrote:
>Thanks for that tip.  I tried the version from Aug 30, 2001 but
>hit a problem that the swish-e process consumed over 350MB of RAM
>and it had only been running for about 20 seconds.  This meant that
>it was continually swapping on my workstation. which has 256MB of RAM.

I had a similar case.  There's been some optimization to the indexing, and
it really does same memory and makes indexing faster.  But, some files made
it eat memory.  

For example, on my machine with 128M I used to be able to index my /usr/doc
(about 27,000 files) in about 15 minutes with just a little swapping.

With the new indexing process it would go along fine, but then my memory
would jump up to about 200M, and I'd swap to death.  Jose found the
problem, which was a large file with the same word repeated thousands of
times, IIRC.  Jose fixed that, and now I can index my /usr/doc in about 70
to 80M and in something like four minutes.  But I thought that was fixed
before Aug 30.

So, perhaps you have some file that swish isn't dealing with correctly.  If
you run the Aug 30 version with -v set to see file names as they are
indexed, does it stop at some file (when your memory usage jumps)?

>I used your tip again and reverted to an earlier version "Jun 17, 2001"
>(I have this code running elsewhere) and this runs and only uses about
>11MB of RAM to index.  Indexing takes about 7 minutes.

How many files and what's their average size?



Bill Moseley
mailto:moseley@hank.org
Received on Tue Sep 4 12:44:17 2001