Skip to main content.
home | support | download

Back to List Archive

Re: Getting the right files indexed the right way

From: Rob de Santos AFANA <rdesantos(at)not-real.afana.com>
Date: Thu Jan 29 2004 - 08:05:54 GMT
Bill and Dave,

> Think twice about placing anything in robots.txt that isn't linked
from 
> anywhere.  I have used fake entries in robots.txt and then seen 404 
> request for those entries.  Not everyone plays by the rules.

Oh how well I know and that's why I was trying to work thru this. 

I took Dave's suggestion to just let the system spider the modified link
that is used to access Gallery from the web.  This actually worked
nicely since the .htaccess file used by Gallery automatically redirected
the spider to the correct locations and didn't allow indexing of any
protected files.   

Everything is working fine and I am now fine tuning swish.cgi.  Thanks
for everyone's help. 

Regards, 

-Rob de Santos
-Columbus, Ohio USA
Chairman of the Board,
Australian Football Association of North America (AFANA)
ph: 1-888-4AFANA1 (North America) (1-888-423-2621)
ph: 1-614-338-0002 (outside NA)  
e-mail: rdesantos(at)not-real.afana.com   web: <http://www.afana.com>
Received on Thu Jan 29 00:05:58 2004