I am looking for a way to stop the spider from passing a certain URL.
I want to index a subtree of my site, but it contains a link to the main
index file, and from there it spiders out to the whole site again. In
short: I want to exclude the main index file. Any ideas?
I stumbled on something that could be a bug: the spider did not follow
links in my file as long as it's header contained the tag:
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
(the line was included by some html generator)
Sounds weird isn't it?
Received on Sun Mar 21 00:57:13 1999