Skip to main content.
home | support | download

Back to List Archive

Re: spider a database

From: Michael Porcaro <music(at)>
Date: Sun Nov 06 2005 - 07:34:24 GMT
After spidering my whole site for about 3 or 4 hours, I got this error:

err: Property Compression Error.  zlib compress2 returned: -4  Prop len:
106 compress buf size: 1107 compress level:-1

I have a 73 meg index file, but the file has a .temp extension still.
What went wrong?  Do I have to respider my whole site again?  Obviously
I need some type of program installed?  Do I need to install zlib

-----Original Message-----
[] On Behalf Of Bill Moseley
Sent: Saturday, November 05, 2005 6:18 PM
To: Multiple recipients of list
Subject: [SWISH-E] Re: spider a database

On Sat, Nov 05, 2005 at 11:09:43AM -0800, Michael Porcaro wrote:
> Ok I think I am understanding now. I was confused because I didn't
> realize there are 2 different configuration files.  One for parameters
> which is much simpler (swish.conf) and another for, which
> requires perl knowledge (a perl config file).  So there are 2 config
> files, am I correct on this?

Kind of.  The spider's job is to fetch documents form websites.  So,
as you might expect, there's a config file to tell the spider what
urls to spider, which to skip, and maybe how to filter non-text
documents.  The spider output the files it fetches in a format that's
read by swish.

Swish-e's job is to take documents and parse out the words and index
them.  So there's a config file for controlling how swish deals with
its input.

If you want to think of both of those activities as one thing with two
config files, that's up to you.

> Finally, where is this custom config perl file supposed to go?  Under
> what directory?  I tried running it in my cgi-bin (local website) but
> didn't work.

Interesting.  So, why do you think the configuration file is a cgi

Bill Moseley

Unsubscribe from or help with the swish-e list:

Help with Swish-e:
Received on Sat Nov 5 23:34:31 2005