Skip to main content.
home | support | download

Back to List Archive

Re: duplicate entries in DB after regex performed on URLs?

From: Bill Moseley <moseley(at)>
Date: Wed Dec 07 2005 - 15:19:57 GMT
On Wed, Dec 07, 2005 at 10:12:09AM -0500, Chad Day wrote:
> swish.conf:
> $ cat swish.conf 
> # Example configuration file
> SwishProgParameters spider.conf
> #ReplaceRules regex /\?PHPSESSID.*$//i
> #ReplaceRules regex /&PHPSESSID.*$//i
> # Tell Swish-e what to index (same as -i switch above)
> IndexDir /usr/local/lib/swish-e/
> IndexFile /usr/local/apache/htdocs/website.index 
> IndexOnly .php .txt .html .htm .pdf .xml .htm .shtml

IndexOnly has no effect here.

> # Index the PDF files
> FileFilter .pdf /usr/X11R6/bin/pdftotext '"%p" -' is filtering -- this will attempt to filter already filtered

> # Tell Swish-e that .txt files are to use the text parser.
> IndexContents TXT* .txt .pdf
> IndexContents XML* .xml
> IndexContents HTML* .htm .html .shtml .php 

The spider should tell swish what type the content is, so
IndexContents should not be needed.

It will take a few hours before I can look at the "cookie" issue.

Bill Moseley

Unsubscribe from or help with the swish-e list:

Help with Swish-e:
Received on Wed Dec 7 07:19:58 2005