Skip to main content.
home | support | download

Back to List Archive

Re: random crashing of spider.pl!?

From: Bill Moseley <moseley(at)not-real.hank.org>
Date: Mon Jan 17 2005 - 16:54:08 GMT
On Mon, Jan 17, 2005 at 08:17:56AM -0800, Justin Tang wrote:
> Hi all:
>   I think I figured out what happened, but I don't know how to solve it.  I
> think what happens is that the spider is put to sleep when it can't connect
> to the site(seems like it's asking me for a user name and password, but I
> already set crident_time as undef), and I forked the spider out as a zombie
> program, so when it sleeps the process is killed.  Is there any way around
> the spider being put to sleep?  Here is a copy of the setting I have in my
> config file.

I'm not following what you are saying about forking the spider out as
a zombie.  Are you saying you are running the spider in the
background?

I assume you are NOT running on Windows.  On windows there's no
alarm() function (IIRC) so it would just sit there and wait, I
suppose.

If the spider is running in the background then I'd think that any
password request would just timeout and continue.  But, maybe there's
an issue if there's no controlling terminal.  I have not checked that.

Maybe you should add a few debugging lines in the
get_basic_credentials() subroutine and see if it's stopping there, and
where.

> I've been stuck on this for so long... If anyone can help me out of it, I
> would be so grateful...

General ideas:

How are you starting the spider?

Can you make it happen if you specify just one file in the spider
config (the protected file)?

Set a SIGUSER to do a backtrace if the spider is hanging. 

Add more debugging statements to see where things fail.



-- 
Bill Moseley
moseley@hank.org

Unsubscribe from or help with the swish-e list: 
   http://swish-e.org/Discussion/

Help with Swish-e:
   http://swish-e.org/current/docs
   swish-e@sunsite.berkeley.edu
Received on Mon Jan 17 08:54:08 2005