Bill Moseley wrote:
> >  <META HTTP-EQUIV="Refresh" CONTENT="1;
> >URL=http://www.prophezine.com/search/database/archives.html">
> Ouch!  You have two one second browser redirects just to get to that one page.

Hmm, (excuse my pet peeve ;) I'd agree with Bill that this is an
administration problem more than a spider problem...  Although, I would
never use, recommend, nor even hint at the existence of a META redirect
(gritting teeth as I type this) many folks do for various (good, bad,
indifferent) reasons.

> Might I recommend that you get rid of the META refresh tags and use
> Apache's redirect features.

(prophezine.com does appear to be running a semi-recent Apache 
 according to 'lynx -head -dump http://www.prophezine.com/')

>   Redirect /search/database/index.html
> http://www.prophezine.com/search/database/archives.shtml

I would not quite do exactly that...  Apache's default redirect status
is Temporary (HTTP 302?).  You want to have Apache tell the browser that
this is a permanent (HTTP 301) change (in the event of a miracle and the
browser actually follows specs by auto-updating it's bookmarks).  

The Apache syntax:
  Redirect [status code] url-path url

For example:
  Redirect Permanent /search/database/index.html
http://www.prophezine.com/search/database/archives.shtml

or

  Redirect 301 /search/database/index.html
http://www.prophezine.com/search/database/archives.shtml

And, then patching the spider somehow since this may be an issue with
folks indexing servers where they have no control.

-- 
,David Norris
  Open Server Architecture Project - http://www.opensa.org/
  Dave's Web - http://www.webaugur.com/dave/
  ICQ Universal Internet Number - 412039
  E-Mail - dave@webaugur.com