Skip to main content.
home | support | download

Back to List Archive

Re: [swish-e] Passing username and password when spidering restricted websites

From: Peter Karman <peter(at)not-real.peknet.com>
Date: Wed Jun 16 2010 - 01:30:59 GMT
Troy Wical wrote on 6/15/10 9:09 AM:
> Had my down time, now getting back into this again. This time it's for the workplace. We have several internal documentation sites, and search all of them individually can be a pain. So I decided to spider all of them and make them all searchable via swish.cgi.  I have it working fairly well so far, but am having a hard time spidering sites that require authentication.  All the sites are being indexed individually, and this is the basic conf that I am using:
> 
> ###############################
> 
> IndexDir spider.pl
> SwishProgParameters default http://restricted-website.com/dir/index.php 
> IndexFile /path/to/indexes/restricted-website.index
> StoreDescription HTML* <body> 200000
> 

Instead of "default" above you need to create a spider config file and put
"credentials" in it:

http://swish-e.org/docs/spider.html#credentials

-- 
Peter Karman  .  http://peknet.com/  .  peter(at)not-real.peknet.com
_______________________________________________
Users mailing list
Users@lists.swish-e.org
http://lists.swish-e.org/listinfo/users
Received on Tue Jun 15 21:31:01 2010