Skip to main content.
home | support | download

Back to List Archive

Re: Again: Indexing laaarge ammounts of data (more

From: Hup Chen <hup(at)>
Date: Thu Sep 25 2003 - 00:55:54 GMT

> 64-bit files.  However, you'd need to use a 64-bit filesystem like XFS
> (SGI or Linux) to have files larger than 2 GB.  Linux ext2/ext3, for
> example, can only support 2 GB files (maybe 4 GB?).  (Also note that
> most NFS implementations are limited to 32-bit.)

  I am afraid this is not true.  Linux ext2/ext3 can support 16GB-64TB
files.  You can get more information at here:

> never tested with large file support and anything that needed more than
> 32 bits. I suspect that signed integer overflow might be the tricky
> part.

  I was trying to merge a 1.9GB index1 with a 800MB index2 under my Linux
ext3 with LFS support, but it failed at "writing word hash", and left me a
2.6GB prop.temp.  Seem like it was some kind of overflow.

Writing index entries ...
  Writing word text: Complete
  Writing word hash: ...err: Ran out of memory (could not allocate
99880956 more bytes)!

> While you are in there you could convert to UTF-8, too.

Sorry if this is a stupid question, how to convert it to UTF-8?

Technical Manager

Search.  Compare.  Save.  (new book search)  (used book search)  (CDs)  (DVD & VHS)  (magazine subscriptions)
Received on Thu Sep 25 00:56:07 2003