Skip to main content.
home | support | download

Back to List Archive

how to stop the swish spider

From: Saskia Beeser <saskia(at)>
Date: Sun Mar 21 1999 - 09:02:58 GMT

I am looking for a way to stop the spider from passing a certain URL. 
I want to index a subtree of my site, but it contains a link to the main
index file, and from there it spiders out to the whole site again. In
short: I want to exclude the main index file. Any ideas?
                          Saskia Beeser
                          Computation Center 
                          Ben-Gurion University
I stumbled on something that could be a bug: the spider did not follow
links in my file as long as it's header contained the tag:
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
(the line was included by some html generator)
Sounds weird isn't it? 
Received on Sun Mar 21 00:57:13 1999