On Thu, 2003-04-03 at 19:00, Bill Moseley wrote:
> The "Delay" is set to 60 seconds. That seems way too long for the average
> user. I'd think 5 seconds would be fine.
Web Robot guidelines (circa 1993) states no more than one document per
minute; preferably one document per 5 minutes. Maybe that's a little
conservative given today's bandwidth and server hardware. But, it is a
The assumption, I think, is that someone is going to be running SWISH-E
against their own website. I think by default we should assume that the
user isn't in control of the server they're spidering.
> MaxDepth is set to 5. That only seems like a way to not index documents
> you thought should be indexed. I'd think zero (do not limit by depth)
> would be best).
So long as the robot can't get stuck in a loop that should be fine.
> Those are just the defaults, they can still be overridden in your
> configuration file.
Which is why we might want to err on the conservative side. :-)
ICQ - 412039
Received on Fri Apr 4 01:24:30 2003