Skip to main content.
home | support | download

Back to List Archive

Re: Out of memory during "writing word data"

From: Peter Karman <peter(at)not-real.peknet.com>
Date: Tue Oct 17 2006 - 18:33:56 GMT
brad miele scribbled on 10/17/06 1:16 PM:
> Hi,
> 
> In the last few days, our largest index has started to fail during the 
> "writing word data" part of the process with an out of memory error. My 
> concern here is, of course, that we have hit some sort of upper level of 
> indexing capability. 

the error msg should say something about the # of bytes it could not allocate. 
That would be helpful to know.


> 
> OS: FreeBSD 5.4 STABLE, SMP, 2 x 2.3Ghz xeon, 2.5G ram
> Swish-e version: 2.4.3 (will upgrade as a first step if this might help)
> Indexing description: directory of xml files, total filecount in the 
> directory is 985707, looks like it started breaking at 984309. We aren't 
> using the btree/incrimental on this index, and are using -e, which had 
> helped us get past an out of memory issue that used to occur earlier in 
> the file parsing process.

You don't appear to be indexing as much as some folks (you haven't hit the magic 
million mark, in other words). So I don't think you've hit a limit inherent in 
swish-e.

I would try 2.4.4 just to see if something was fixed. iirc, there were some out 
of memory issues addressed ... sometime in the past (this is what happens when 
so much time passes between releases...).

if that doesn't fix it, and you really aren't out of memory, it's likely time to 
get out gdb and trace exactly where the fault occurs.

-- 
Peter Karman  .  http://peknet.com/  .  peter(at)not-real.peknet.com
Received on Tue Oct 17 11:33:57 2006