--- Bill Conlon <bill@tothept.com> wrote: > I have a robots.txt file: > Is the robots.txt file located at /robots.txt ? Maybe the spider can't get to it because of authentication (I can't get to http://beowulf3.tothept.com/robots.txt ) ===== Greg Fenton greg_fenton@yahoo.com __________________________________ Do you Yahoo!? New and Improved Yahoo! Mail - 100MB free storage! http://promotions.yahoo.com/new_mailReceived on Tue Jul 27 21:33:53 2004