Skip to main content.
home | support | download

Back to List Archive

indexing dynamic sites and robots.txt

From: Michael <michael(at)not-real.insulin-pumpers.org>
Date: Thu Apr 11 2002 - 21:39:22 GMT
Is it possible to index dynamic sites that use URL's of the form

http://somewhere.com/page.cfm?object=1234
a new page is represented by
http://somewhere.com/page.cfm?object=1235

but I believe that swish-e thinks 

http://somewhere.com/page.cfm

has already been indexed and move on without looking at the new page.
Is this what happens??

Robots

User-agent *
Disallowed:

seems to disallow everthing that swish sees. I should allow 
everything per the spec

While 

User-agent *
Disallowed: \

would disallow everything.

Michael

Michael@Insulin-Pumpers.org
Received on Thu Apr 11 21:40:46 2002