Skip to main content.
home | support | download

Back to List Archive

Re: More?

From: Hadean Dragon <hadean(at)not-real.ican.net>
Date: Fri Mar 06 1998 - 23:41:01 GMT
>Bear in mind before setting a hacked version of SWISH-E or SWISH++ loose on
>the world, the need to be a responsible robot. In addition to hacking in
>HTTP fetching you'll need to add support for robots.txt parsing, decent
>delays between server fetches, and possibly robots META tag parsing.


Actually, I've already made sure that the robot would check for robots.txt
and I've done some messing around with delays and whatnots...  I understand
totally that it's very inapropriate to let loose a wild robot on the net
without any care in the world about other people's systems...  (This goes
back to the Golden Rule I gues... "Do unto others...")

>There may be other system which will do some of this for you. For instance
>Harvest is currently being developed to include a SWISH backend. Please
don't,
>however, just add a HTTP client to SWISH and go crawling the web.


I've heard of Harvest... And also another one, but I can't remember the name
of it right now... How are they?
Received on Fri Mar 6 15:50:04 1998