Skip to main content.
home | support | download

Back to List Archive

Re: Exclusions for HTTP indexing method?

From: Ron Samuel Klatchko <rsk(at)not-real.brightmail.com>
Date: Tue Jan 11 2000 - 17:57:17 GMT
"Halasz, Steve" wrote:
> I think that Swish-e is brilliant. However it doesn't seem to allow
> exclusion of certain directories or files when indexing with the HTTP
> method. I'm hoping for some way to do exclusions along the lines of
> 
> FileRules pathname /some/directory
> 
> but for HTTP.

You can use robot exclusion rules for this.  Create a file called
robots.txt in your web servers document root (or add to it if it's
already there):

User-agent: swishspider
Disallow: /some/directory

moo
------------------------------------------------------------
           Ron Samuel Klatchko - Software Jester
            Brightmail Inc - rsk@brightmail.com
Received on Tue Jan 11 13:10:22 2000