Skip to main content.
home | support | download

Back to List Archive

Re: Callback Functions For Indexing

From: William M Conlon <bill(at)not-real.tothept.com>
Date: Fri Jan 27 2006 - 18:24:18 GMT
How about using:

	debug => DEBUG_URL | DEBUG_SKIPPED | DEBUG_FAILED | DEBUG_HEADERS,

  within your @servers

On Jan 27, 2006, at 9:45 AM, andy rosbrook wrote:

> Well i just want to know after each URL in spider.config weather the
> spidering was a success or a failure. I know i could just check for a
> complete index.swish-e but this doesnt allow me to capture any error
> messages.
>
> Ill take a look at grabbin STDERR though, thanks.
>
>
>> From: Bill Moseley <moseley@hank.org>
>> Reply-To: moseley@hank.org
>> To: Multiple recipients of list <swish-e@sunsite3.berkeley.edu>
>> Subject: [SWISH-E] Re: Callback Functions For Indexing
>> Date: Fri, 27 Jan 2006 07:21:45 -0800 (PST)
>>
>> On Fri, Jan 27, 2006 at 06:30:56AM -0800, andy rosbrook wrote:
>>> Is there anyway to use a callback function to catch errors when
>> spidering
>>> websites with spider.pl?
>>>
>>> I am currently spidering only a few small sites at a time and  
>>> need a way
>> of
>>> knowing weather the spider successfully indexed the site or not,  
>>> is this
>>> possible? if so is there a way of grabbing the error message into  
>>> perl?
>>
>> Not sure what you mean.  You want to know if any file returned a
>> non-200 status?  Or if swish-e indexed any words?
>>
>>
>> IPC::Open3 will capture stderr and stdout.
>>
>> --
>> Bill Moseley
>> moseley@hank.org
>>
>> Unsubscribe from or help with the swish-e list:
>>    http://swish-e.org/Discussion/
>>
>> Help with Swish-e:
>>    http://swish-e.org/current/docs
>>    swish-e@sunsite.berkeley.edu
>>
>
> _________________________________________________________________
> Are you using the latest version of MSN Messenger? Download MSN  
> Messenger
> 7.5 today! http://messenger.msn.co.uk
>

Bill

William M. Conlon, P.E., Ph.D.
To the Point
345 California Avenue Suite 2
Palo Alto, CA 94306
    vox:  650.327.2175 (direct)
    fax:  650.329.8335
mobile:  650.906.9929
e-mail:  mailto:bill@tothept.com
    web:  http://www.tothept.com
Received on Fri Jan 27 10:24:18 2006