Skip to main content.
home | support | download

Back to List Archive

Re: Can Swish-e index 20 million urls? What is the

From: Peter Karman <peter(at)not-real.peknet.com>
Date: Fri Sep 30 2005 - 02:10:59 GMT
Thanks, James. Can you tell us whether that was speed/memory for indexing or 
searching? (I'm assuming indexing, but just wanted to make sure.) Also, are you 
indexing via filesystem or spider (-S prog)?

James Hemphill scribbled on 9/28/05 10:59 PM:

> 
> For what it's worth. At Biblio.com we have around 25 million records 
> indexed using swish-e. The only kludge involved is that we do have to 
> break the index files into diffrent files. We found that putting 2 
> million listings in each index file was the optimum speed/memory usage 
> point for swish.
> 
> James Hemphill
> 
> On Wed, 28 Sep 2005, Peter Karman wrote:
> 
>> Don't know if you've received replies offlist...
>>
>>
>>> I'm a new user. I want to index 20 millions urls from one server. Is it
>>> possible with Swish-e?
>>
>>
>> Swish-e is intended for collections of a million docs or less. That 
>> said, some
>> folks on this list have many more indexed successfully.
>>
>> If you try Swish-e with 20 million docs, please let us know how it 
>> goes for you.
>>
>>

-- 
Peter Karman  .  http://peknet.com/  .  peter(at)not-real.peknet.com
Received on Thu Sep 29 19:11:18 2005