Skip to main content.
home | support | download

Back to List Archive

RE: HTTP Crawler

From: Bill Moseley <moseley(at)>
Date: Thu May 02 2002 - 16:50:56 GMT
At 09:27 AM 05/02/02 -0700, Hsiao Ketung Contr 61 CS/SCBN wrote:
>User-Agent: *
>Disallow: /somedirectory/
>Disallow: /somedirectory/
>What does robots.txt does and 
>what's your suggestion ?

Google is your friend.

If you were to use -S prog with you can tell it to ignore
robots.txt.  But, I'd suggest you try to get -S http method working first
before trying to tackle the -S prog / setup with swish.

Bill Moseley
Received on Thu May 2 16:51:00 2002