Skip to main content.
home | support | download

Back to List Archive

Follow up on non-spidering spider

From: Bruce Bowler <bbowler(at)>
Date: Wed Jan 27 1999 - 20:29:07 GMT
In case anyone else runs into this, yesterday I posted a problem about a
spider that didn't find any links on my index page even though there were
plenty there.  I eventually (with the help of Giulia Hill and Ron Klatchko)
figured out that some meta tags in my header were causing the problem.  The
solution was to modify the spider perl code so that the line that looked like

    if( $response->header("content-type") eq "text/html" ) {

ended up looking like 

    if( substr($response->header("content-type"),0,9) eq "text/html" ) {

Now everything seems to work like a champ...

Thanks Ron and Giulia!

Bruce Bowler                             207.633.9600 (voice)
Research Associate                       207.633.9641 (fax)
Bigelow Laboratory for Ocean Sciences
West Boothbay Harbor ME  04575 
Received on Wed Jan 27 12:29:04 1999