Skip to main content.
home | support | download

Back to List Archive

advantages and disadvantages of indexing via the spider

From: Eric Lease Morgan <emorgan(at)not-real.nd.edu>
Date: Mon Feb 16 2004 - 16:15:07 GMT
What are the advantages and disadvantages of indexing via the the 
spider?

I want to index the content of many electronic serials, specifically, 
all or part of the serials listed the Directory of Open Access 
Journals:

   http://www.doaj.org/

I suppose I could use spider.pl to crawl the remote files and index 
them. I could also use something like wget to create mirrors of the 
files and index them that way.

What are the advantages and disadvantages of either approach? If I use 
the spider, the I don't need nearly as much local disk space. If I do 
the mirroring thing, then I have local copies and I save on network 
bandwidth.

-- 
Eric Lease Morgan
Head, Digital Access and Information Architecture Department
University Libraries of Notre Dame

(574) 631-8604
Received on Mon Feb 16 08:15:07 2004