Skip to main content.
home | support | download

Back to List Archive

Re: Incremental updating

From: Bill Moseley <moseley(at)not-real.hank.org>
Date: Tue Jun 08 2004 - 20:01:27 GMT
On Tue, Jun 08, 2004 at 12:47:28PM -0700, adivey1@cox.net wrote:
> Is it possible to do some kind of incremental update (via spider or
> file system)? If not, how often would you reccommend indexing a large
> (1,000,000+ hits a month) site?

Indexing what? The content?  Depends how current it needs to be.
There's not really any incremental indexing available where you need to
update an existing index that includes more than just additions to the
index.


> Also, this is a problem for me doing a large number of sites at once.
> We're trying to develop a search engine for our entire program which
> has around 100 domains.

Do you need to search across domains in a single search?
If not index them separately.

The only options for "incremental" is to split the index into smaller
chunks and search them -f index1 index2 index3 or to merge them -M
index1 index2 index3 index.out.

But you would need to try both to see which, if either, works for your
data set.


> I started it last night and when I got in in
> the morning, it had frozen, swish-e was using 630MB of RAM, and the
> computer was barely usable.

Did you use the -e option?

-- 
Bill Moseley
moseley@hank.org
Received on Tue Jun 8 20:01:29 2004