On Thu, 2003-04-03 at 19:00, Bill Moseley wrote:
> The "Delay" is set to 60 seconds.  That seems way too long for the average
> user.  I'd think 5 seconds would be fine.

Web Robot guidelines (circa 1993) states no more than one document per
minute; preferably one document per 5 minutes.  Maybe that's a little
conservative given today's bandwidth and server hardware.  But, it is a
safe value.

The assumption, I think, is that someone is going to be running SWISH-E
against their own website.  I think by default we should assume that the
user isn't in control of the server they're spidering.

> MaxDepth is set to 5.  That only seems like a way to not index documents
> you thought should be indexed.  I'd think zero (do not limit by depth)
> would be best).

So long as the robot can't get stuck in a loop that should be fine.

> Those are just the defaults, they can still be overridden in your
> configuration file.

Which is why we might want to err on the conservative side.  :-)

-- 
 David Norris
  http://www.webaugur.com/dave/
  ICQ - 412039