> I believe someone posted a patch recently that did exactly this.
> 
Nope, that patch is for the same problem, but it is in 
http.c and I've applied it.

The problem is that the perl helper 'swishspider' fails the test on 
the "text/html" string and NEVER PASSES the page to swish-e for 
parsing. The result being that all the links on the page are skipped.

I have not tried it with the patch to http.c removed, but looking at 
the code, it seems to me it would be a second point of failure.

> At 01:19 PM 6/15/99 -0700, Michael wrote:
> >I'd like to be able to use the FileRules when indexing http sites. Is 
> >there a simple method to enable these checks for spidering??
> >
Michael@bizsystems.com