Skip to main content.
home | support | download

Back to List Archive

Re: Trouble with obeyRobotsNoIndex

From: Bill Moseley <moseley(at)>
Date: Wed Oct 16 2002 - 17:46:58 GMT
At 09:25 AM 10/16/02 -0700, Markus Strickler wrote:
>I'm currently encountering problems with swish-e not generating an index
>if I have obeyRobotsNoIndex set to yes and use the HTML2 parser.
>With swish-e-2.1-dev-25-2002-03-22 which worked for several months
>without any problems indexing just stops during writing word data.
>So I tried the latest release for windows (2.2.1) which stops during
>indexing after a few hundred files with "Indexing failed".

There was a bug in the code that handled removing files (when that no index
meta tag is found swish has to back-out the additions to the index up to
that point for the current file).  But that should have been fixed.  Maybe
there's still another problem.

>The directory I try to index contains around 700 files where most
>contain a robots no index meta tag.

Are you spidering these files -- that is following links within the files
to determine what to index, or just indexing a bunch of files in a directory?

If most are not to be indexed then I'd be tempted to figure out a way to
only process with swish the files that really need to be indexed.

As a general rule, if you (or anyone posting with a problem) can provide a
working example of the problem (e.g. a small set of input files, a small
config file, and the cut-n-pasted output showing how it doesn't work right)
then the chances of getting a quick fix are greatly improved.

>Has anybody else had similar problems? I've tried this on three machines
>all running windows 2000 without success.

It might be helpful if you tested on a unix-based system to see if it's a
specific windows issue.

Bill Moseley
Received on Wed Oct 16 17:50:45 2002