Skip to main content.
home | support | download

Back to List Archive

Re: connection bug (and fix)

From: Bill Moseley <moseley(at)>
Date: Mon Oct 07 2002 - 13:44:01 GMT
On Mon, 7 Oct 2002, Trond Nilsen wrote:

> Thus, when running the spider over a large number of sites, a backlog of 
> unclosed connections builds up, which eventually prevents new connections from 
> being opened (at least, in the case of Win32).
> To fix this, I've just removed the user agent from the server hash once 
> spidering of each server is complete, letting it close as control falls off 
> the end of that block of code.
> That is, I've added a new line
>      $server->{ua} = undef;
> at line 263 of ''.

Good catch.

I just never spidered that many different sites before to have it
an issue, but you are right that it would leave a bunch of CLOSE_WAIT
connections around.

I'll add that to


Bill Moseley
Received on Mon Oct 7 13:48:12 2002