Skip to main content.
home | support | download

Back to List Archive

Re: New index proccess in development version

From: Bill Moseley <moseley(at)not-real.hank.org>
Date: Wed Aug 29 2001 - 14:49:32 GMT
Jose,

When I was indexing I had similar upper memory usage between the different
versions -- just that the newer version got to the upper memory limit and
stayed there.  But maybe that depends on the type of documents that are
being indexed?  In other words, you think it's possible that Chris has
files that are making swish use more memory than in the previous version?

Is it really true that BSD doesn't free the memory?  I just assumed that
the reported number was not reduced after freeing memory.


At 07:33 AM 08/29/01 -0700, jmruiz@boe.es wrote:
>
>Hi Chris,
>
>I will need more information. Can you give me the exact error
>message?
>Which is the amount of memory being using by swish
>(July 24th and Yesterday's)?
>Have you noticed anything strange in memory usage?
>Wich is the longest file size?
>
>By the way, Bill Mosseley told me that swish does not return the
>freed memory to the OS until proccess ends in FreeBSD.
>This seems to be the allocation library being used. Since
>MacOS X is based on BSD,  it can have a similar problem. This
>situation does not occur in linux or win32.
>
>cu
>jose
>
>On 29 Aug 2001, at 8:18, Chris Blackstone wrote:
>
>> I downloaded the daily for yesterday, made it and indexed my server
>> 
>> With the July 24th snapshot, my site indexes in about an hour
>> Mac OS X 10.0.4
>> 7156 Docs (745 of those PDF files)
>> 
>> With yesterday's dev snapshot, swish-e crashed with an out of memory
>> error. I don't remember the exact error, but I can rerun swish-e if
>> that info will be helpful.
>> 
>> jmruiz@boe.es wrote:
>> 
>> > Hi all,
>> >
>> > I have changed the way memory is being used in the latest
>> > development versions. Now, it should use less memory. Also,
>> > the index files must be a little smaller because of a better
>> > compression schema. The -e (economic option) index option
>> > should also use less memory than in previous versions.
>> >
>> > I would like to hear from you if this is true in all cases. If there
>> > is any particular case in which the performance have been degraded
>> > or the memory usage has increased, let me know.
>> >
>> > I would also like to know if there are some people indexing large
>> > collections of data (more than 100000 documents). If so, I would
>> > like to know the OS you are using, index time, search time, memory
>> > usage, etc...
>> >
>> > Thanks in advance.
>> >
>> > cu
>> > Jose Ruiz
>> 
>> --
>> #!chris blackstone
>> #!web services coordinator
>> 
>> Arlington Public Schools
>> 1426 N. Quincy St.
>> Arlington, VA 22207
>> Phone:  703.228.6185
>> Fax:    703.875.9491
>> Pager:  703.612.3042
>> http://www.arlington.k12.va.us
>> 
>> 
>
>
>

Bill Moseley
mailto:moseley@hank.org
Received on Wed Aug 29 14:53:23 2001