Skip to main content.
home | support | download

Back to List Archive

Re: Again: Indexing laaarge ammounts of data (more

From: Hup Chen <hup(at)not-real.addall.com>
Date: Thu Sep 25 2003 - 00:55:54 GMT
Hi,

> 64-bit files.  However, you'd need to use a 64-bit filesystem like XFS
> (SGI or Linux) to have files larger than 2 GB.  Linux ext2/ext3, for
> example, can only support 2 GB files (maybe 4 GB?).  (Also note that
> most NFS implementations are limited to 32-bit.)

  I am afraid this is not true.  Linux ext2/ext3 can support 16GB-64TB
files.  You can get more information at here:
http://www.suse.de/~aj/linux_lfs.html



> never tested with large file support and anything that needed more than
> 32 bits. I suspect that signed integer overflow might be the tricky
> part.

  I was trying to merge a 1.9GB index1 with a 800MB index2 under my Linux
ext3 with LFS support, but it failed at "writing word hash", and left me a
2.6GB prop.temp.  Seem like it was some kind of overflow.


Writing index entries ...
  Writing word text: Complete
  Writing word hash: ...err: Ran out of memory (could not allocate
99880956 more bytes)!




> While you are in there you could convert to UTF-8, too.

Sorry if this is a stupid question, how to convert it to UTF-8?


---
Hup
Technical Manager
AddALL.com

Search.  Compare.  Save.

http://www.addall.com  (new book search)
http://used.addall.com/  (used book search)
http://www.amusicarea.com  (CDs)
http://www.amoviearea.com  (DVD & VHS)
http://www.amagarea.com  (magazine subscriptions)
Received on Thu Sep 25 00:56:07 2003