Skip to main content.
home | support | download

Back to List Archive

Re: Re: spider generates 'Wrong method or server' errors

From: Dirk-WIllem van Gulik <Dirk.vanGulik(at)not-real.jrc.it>
Date: Thu Dec 24 1998 - 10:47:48 GMT
David Norris wrote:

> > EquivalentServer A B C
> > tells the spider to treat servers A, B and C as single logical server.
>
> Just out of curiousity, would one be able to do this:
>
> EquivalentServer http://myserver/ http://myserver/too

Actually, if you look at www.apache.org -> search you'll see that we
do something similar there; i.e. if you reach the page from one of the
many mirors; the hits will point you back to the mirror site with
the main apache site as a second. This way you can provide a (centrally
indexed) service, i.e. the mirrors do not have to run CGI whilst
still allowing the people to keep using the mirrors.

The reason to menstion this is that it it implemented purely with the
REFERER field from the HTTP request; i.e. you know where the
search page is in your structure, or in the mirrored structure and you
map accordingly; i.e. ($hissearchroot = referer) =` s:[^/]+$::g; $url =`
s/$mysearchroot/$hissearchroot/g. I.e keep the problem away from
the indexing.

Dw

> BTW, I think I have narrowed the spidering problem down to a few lines of
> code.  I will have more info later.  It has to do with the screwy command
> interpreter on Windows 9x.  NT would likely work fine.  To run the PERL
> script on 9x, one would have to call "start swishspider.pl...", instead of
> "swishspider.pl..."
>
> ,David Norris
>
> World Wide Web - http://www.geocities.com/CapeCanaveral/Lab/1652/
> Illusionary Web - http://illusionary.dyn.ml.org/ <-- 02:00 - 10:00 GMT
> Video/Audio Phone - callto:illusionary.dyn.ml.org
> Page via mail - 412039@pager.mirabilis.com
> ICQ Universal Internet Number - 412039
> E-Mail - kg9ae@geocities.com
Received on Thu Dec 24 03:48:50 1998