Skip to main content.
home | support | download

Back to List Archive

Re: [SWISH-E:44] Re: Max (reasonable) size of index?

From: WWW server manager <webadm(at)not-real.info.cam.ac.uk>
Date: Fri Oct 24 1997 - 23:00:26 GMT
Roy Tennant wrote:
> On Fri, 24 Oct 1997, Michael A. Tilp wrote:
> > 
> > 	Anyone have any guesses as to the maximum size of a SWISH index? I've
> > seen it used on sites of 5000+ pages (the old version); I'm just wondering
> > how far that could go. Call me a bit afeared of beginning a large index and
> > then watching it choke a few months down the line ;)
> 
> Our largest index here so far is in the 15 MB range, and probably in the 
> neighborhood of 20,000 files. When we are through with our transition we 
> will be creating indexes over 20 MB in size. So far no problems. This is 
> on a DEC Alpha and a Sun SPARCCenter.

One thing to watch, though, is memory use during indexing - building an 8MB
index from around 6500 documents here takes about 32MB of memory at its peak 
(and 17 minutes on a SPARC 10/51, an "old" and hence slow system by current
standards). If you don't have enough memory to avoid it (and/or the rest of
the system) being slowed by paging/swapping, whether or not it is *possible*
to build large indexes may be only half the story!

                                John Line
-- 
University of Cambridge WWW manager account (usually John Line)
Send general WWW-related enquiries to webmaster@ucs.cam.ac.uk
Received on Fri Oct 24 16:08:11 1997