Skip to main content.
home | support | download

Back to List Archive

Re: HTTP spidering - zero results

From: David Norris <dave(at)>
Date: Mon Jun 12 2000 - 10:27:46 GMT
Angel Parn wrote:
>     -i

My guess would be that the Content Type header is not exactly what the
spider expects.  You can grab the HTTP headers with Lynx:

lynx -head -dump http://someurl/here/

If that is the problem then you may need to modify the spider.  I think
there are patches for this in the patches index.  There should be a few
theads in the archive regarding this, as well.

,David Norris
  Dave's Web -
  Dave's Weather -
  ICQ Universal Internet Number - 412039
  E-Mail -
Received on Mon Jun 12 03:25:56 2000