On Wed, Jun 25, 2003 at 11:02:34AM -0500, Cleveland@mail.winnefox.org wrote:
> > I assume you mean you are running IIS on the Windows machine?
> Unfortunately, yes.
> > Do you see the request for robots.txt in the IIS access log?
> I did find this:
> 2003-06-25 14:22:53 184.108.40.206 - W3SVC22 FORGE 220.127.116.11 80
> GET /robots.txt - 200 0 279 213 0 HTTP/1.1 www.oshkoshpubliclibrary.org
> swish-e+spider+2.2+http://swish-e.org/ - -
> > Maybe you need a line feed after that line?
> I added one in there, but it still indexes that pdf file.
Ah, yes. remove your blank lines. A blank line separates sections
based on the user agent.
Received on Wed Jun 25 17:34:18 2003