Skip to main content.
home | support | download

Back to List Archive

out of memory writing word data

From: Sewell K H (ISeLS) <khsewell(at)not-real.glam.ac.uk>
Date: Wed Nov 23 2005 - 10:41:19 GMT
Hi,

SWISH-E 2.4.3
FreeBSD 5.4-RELEASE
The machine has 2GB RAM. 4GB swap. Lots of disk space.

I'm indexing about 200,000 files, average size about 20k.

Using the prog method with this config (swishconfig.conf):

IndexDir /usr/local/www/rsc/rsc-gather.pl
MetaNames swishtitle authors journalref year
UndefinedMetaTags ignore
PropertyNames arttitle journal year authors
StoreDescription HTML* <pre>
TranslateCharacters :ascii7:
IndexFile /usr/local/www/rsc/index.swish-e
ConvertHTMLEntities yes

Using this command as root:

/usr/local/bin/swish-e -e -S prog -c /usr/local/www/rsc/swishconfig.conf

I get this:

..
Removing very common words...
no words removed.
Writing main index...
Sorting words ...
Sorting 4,598,287 words alphabetically
Writing header ...
Writing index entries ...
    Writing word text: Complete
    Writing word hash: Complete
    Writing word data: ...err: Ran out of memory (could not allocate 
262144 more bytes)!

My ulimit is:

core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) 524288
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 11095
pipe size          (512 bytes, -p) 1
stack size            (kbytes, -s) 65536
cpu time             (seconds, -t) unlimited
max user processes            (-u) 5547
virtual memory        (kbytes, -v) unlimited

I'd be very grateful for advice.

Thanks,

Kevin
University of Glamorgan
Received on Wed Nov 23 02:41:35 2005