Skip to main content.
home | support | download

Back to List Archive

indexing dynamic sites and robots.txt

From: Michael <michael(at)>
Date: Thu Apr 11 2002 - 21:39:22 GMT
Is it possible to index dynamic sites that use URL's of the form
a new page is represented by

but I believe that swish-e thinks

has already been indexed and move on without looking at the new page.
Is this what happens??


User-agent *

seems to disallow everthing that swish sees. I should allow 
everything per the spec


User-agent *
Disallowed: \

would disallow everything.

Received on Thu Apr 11 21:40:46 2002