Skip to main content.
home | support | download

Back to List Archive

Re: [SWISH-E:185] Re: More?

From: Simon Wilkinson <sxw(at)not-real.tardis.ed.ac.uk>
Date: Fri Mar 06 1998 - 17:29:11 GMT
> > was assuming a Sun SPARC dual-150, but that's not written in stone... Any
> > suggestions on what system runs Swish the best? ;)

One final point thats not been made yet. 

Bear in mind before setting a hacked version of SWISH-E or SWISH++ loose on
the world, the need to be a responsible robot. In addition to hacking in
HTTP fetching you'll need to add support for robots.txt parsing, decent
delays between server fetches, and possibly robots META tag parsing.

There may be other system which will do some of this for you. For instance
Harvest is currently being developed to include a SWISH backend. Please don't,
however, just add a HTTP client to SWISH and go crawling the web.

Cheers,

Simon.
Received on Fri Mar 6 09:38:14 1998