Skip to main content.
home | support | download

Back to List Archive

Re: [SWISH-E:45] Re: Max (reasonable) size of index?

From: Roy Tennant <rtennant(at)not-real.library.berkeley.EDU>
Date: Sat Oct 25 1997 - 03:12:04 GMT
Yes, memory is an issue, as is time. Some of our indexes take hours to 
complete (hey, isn't that why it gets dark at night?). I'm not sure of 
the amount of RAM usage, but it must be considerable. One machine we are 
using has 1 GB of RAM, so if we had a lot less I'm sure it would be 
something to watch more closely.

On Fri, 24 Oct 1997, WWW server manager wrote:

> Roy Tennant wrote:
> > On Fri, 24 Oct 1997, Michael A. Tilp wrote:
> > > 
> > > 	Anyone have any guesses as to the maximum size of a SWISH index? I've
> > > seen it used on sites of 5000+ pages (the old version); I'm just wondering
> > > how far that could go. Call me a bit afeared of beginning a large index and
> > > then watching it choke a few months down the line ;)
> > 
> > Our largest index here so far is in the 15 MB range, and probably in the 
> > neighborhood of 20,000 files. When we are through with our transition we 
> > will be creating indexes over 20 MB in size. So far no problems. This is 
> > on a DEC Alpha and a Sun SPARCCenter.
> One thing to watch, though, is memory use during indexing - building an 8MB
> index from around 6500 documents here takes about 32MB of memory at its peak 
> (and 17 minutes on a SPARC 10/51, an "old" and hence slow system by current
> standards). If you don't have enough memory to avoid it (and/or the rest of
> the system) being slowed by paging/swapping, whether or not it is *possible*
> to build large indexes may be only half the story!
>                                 John Line
> -- 
> University of Cambridge WWW manager account (usually John Line)
> Send general WWW-related enquiries to
Received on Fri Oct 24 20:16:43 1997