Skip to main content.
home | support | download

Back to List Archive

Re: anyone using swish-e to index a news spool?

From: Dave Stevens <dstevens(at)not-real.roaddog.com>
Date: Mon Feb 05 2001 - 08:29:14 GMT
Bas Meijer writes:

>I have used Mhonarc to build the webfront for a newsspool, Swish-e is
>indexing it fine and is running for three years now!

Once I was able to generate a dynamic Twister URL that would return valid
results, it worked fine.  The dynamic URL was the issue, not Swish-e.  Once
I got the URL tuned, spider depths, etc, it works very well.  I spider with
a URL created specifically for the spider, to display each message in the
spool, in cronological order, not threaded, with no nav structure on the
template. That way, I can set the spidering depth to "1",  it doesn't
transverse links it's already visited, or try to follow thread structures,
and doesn't try to spider the nav links.

Using the normal URL for the standard interface, took about 9 mins to index
236 posts.  Due to the dynamic URL structure of the standard page, most of
the spidering was over links already traversed.  After I made a spider only
template and URL, it took just under 4 mins, using 1 sec intervals.

Thanks for the reply.

Dave
Received on Mon Feb 5 08:32:55 2001