On Mon, 7 Oct 2002, Trond Nilsen wrote:

> Thus, when running the spider over a large number of sites, a backlog of 
> unclosed connections builds up, which eventually prevents new connections from 
> being opened (at least, in the case of Win32).
> 
> To fix this, I've just removed the user agent from the server hash once 
> spidering of each server is complete, letting it close as control falls off 
> the end of that block of code.
> 
> That is, I've added a new line
> 
>      $server->{ua} = undef;
> 
> at line 263 of 'spider.pl'.

Good catch.

I just never spidered that many different sites before to have it
an issue, but you are right that it would leave a bunch of CLOSE_WAIT
connections around.

I'll add that to spider.pl.

Thanks,

-- 
Bill Moseley moseley@hank.org