On Wed, Apr 07, 2004 at 05:27:04PM -0700, Mark Greenaway wrote:
> I wanted a list of sites containing info on their main page
> 
> I then want to index these sites
> 
> I have it working using individual servers configured in the config.pl
> and combine them all when defining @servers
> 
> This is very clumsy, particularly as the number of sites increases.

Well, one advantage of having a perl code as a config file is you can
programmatically create the config.  So your spider config could do
something like this untested bit of code:

open URLS, "/path/to/list/of/urls" or die $!;
while ( my $url = <URLS> ) {
    push @servers, build_config( $url );
}

were build_config() returns a hash ref of your parameters.

Would that work?


-- 
Bill Moseley
moseley@hank.org