Skip to main content.
home | support | download

Back to List Archive

RE: Spidering on Windows

From: David Norris <kg9ae(at)not-real.geocities.com>
Date: Fri Oct 30 1998 - 15:48:52 GMT
I have got it to work just fine on Windows 98.  Thanks to Mark Gaulin for
moving the forest out of the way of the trees.  I hope I have commandline[]
correct, I didn't check to see what all was being done there.

Here is what I changed in http.c (starts around line 380 or so):

#ifdef _WIN32
	char* spiderprog = "swishspider.pl";
	char commandline[] = "perl %s%s %s/swishspider@%ld \"%s\"";
#else
	char* spiderprog = "swishspider";
	char commandline[] = "%s%s %s/swishspider@%ld '%s'";
#endif

Now, this assumes that perl.exe is in the path.  Which, is what perl 5
suggests to you when install finishes.  So, this should work on any Win32
OS.

It works fine.  However, I am having a weird little problem.  I don't think
it is related to my changes.  The problem is 'URL disallowed by server' when
it encounters a link to another location on my server.  It seems that the
only URL allowed is http://myserver , even http://myserver/ is not allowed,
neither is http://myserver/index.html

The output of SWISH-E HTTP is at the end of this message.  Here is my
config:

IndexDir "http://illusionary.dyn.ml.org"
IndexFile E:/swishe123.idx
IndexName "HTTP Test"
IndexDescription "This is an index to test Spider Support"
IndexPointer "http://illusionary.dyn.ml.org/index.html"
IndexAdmin "David Norris, (kg9ae@geocities.com)"
MetaNames first author
IndexReport 3
FollowSymLinks no
IgnoreLastChar
IgnoreFirstChar
IgnoreLimit 50 1000
IndexComments 0
# HTTP Only
MaxDepth 5
Delay 5
TmpDir C:/tmp
SpiderDirectory ../
EquivalentServer http://illusionary.dyn.ml.org http://localhost

,David Norris

World Wide Web - http://www.geocities.com/CapeCanaveral/Lab/1652/
Illusionary Web - http://illusionary.dyn.ml.org/ <-- 02:00 - 10:00 GMT
Video/Audio Phone - callto:illusionary.dyn.ml.org
Page via mail - 412039@pager.mirabilis.com
ICQ Universal Internet Number - 412039
E-Mail - kg9ae@geocities.com

retrieving http://illusionary.dyn.ml.org (0)...
 (352 words)
Skipping http://illusionary.dyn.ml.org/:  URL disallowed by server.
Skipping http://illusionary.dyn.ml.org/source.php3?page_url=/index.html:
URL disallowed by server.
Skipping http://illusionary.dyn.ml.org/search.php3:  URL disallowed by
server.
Skipping http://illusionary.dyn.ml.org/music.php3:  URL disallowed by
server.
Skipping http://illusionary.dyn.ml.org/music:  URL disallowed by server.
Skipping http://illusionary.dyn.ml.org/Library/:  URL disallowed by server.
Skipping http://illusionary.dyn.ml.org/search.library.php3:  URL disallowed
by server.
Skipping ftp://illusionary.dyn.ml.org/:  Wrong method or server.
Skipping http://illusionary.dyn.ml.org/pub/:  URL disallowed by server.
Skipping http://illusionary.dyn.ml.org/server-status:  URL disallowed by
server.
Skipping http://illusionary.dyn.ml.org/status.html:  URL disallowed by
server.
Skipping http://illusionary.dyn.ml.org/logs.html:  URL disallowed by server.
Skipping http://illusionary.dyn.ml.org/usage:  URL disallowed by server.
Skipping http://illusionary.dyn.ml.org/opera.html:  URL disallowed by
server.
Skipping http://illusionary.dyn.ml.org/library/?N=A:  URL disallowed by
server.
Skipping http://illusionary.dyn.ml.org/library/web%20technology/?N=A:  URL
disallowed by server.
Skipping
http://illusionary.dyn.ml.org/library/web%20technology/html%20reference%20ma
terial/html%204%20final/cover.html:  URL disallowed by server.
Skipping
http://illusionary.dyn.ml.org/library/web%20technology/html%20reference%20ma
terial/html%204%20final/?N=A:  URL disallowed by server.
Skipping
http://illusionary.dyn.ml.org/library/web%20technology/cascading%20style%20s
heets/css%202.0/cover.html:  URL disallowed by server.
Skipping
http://illusionary.dyn.ml.org/library/web%20technology/cascading%20style%20s
heets/css%201.0/rec-css1.htm:  URL disallowed by server.
Skipping
http://illusionary.dyn.ml.org/library/web%20technology/cascading%20style%20s
heets/?N=A:  URL disallowed by server.
Skipping
http://illusionary.dyn.ml.org/library/web%20technology/PHP/2.0/doc.htm:  URL
disallowed by server.
Skipping
http://illusionary.dyn.ml.org/library/web%20technology/PHP/3.0/manual.html:
URL disallowed by server.
Skipping http://illusionary.dyn.ml.org/library/web%20technology/PHP/?N=A:
URL disallowed by server.
Skipping http://illusionary.dyn.ml.org/library/web%20technology/Roxen/:  URL
disallowed by server.
Skipping
http://illusionary.dyn.ml.org/library/web%20technology/XML/REC-xml.htm:  URL
disallowed by server.
Skipping http://illusionary.dyn.ml.org/library/web%20technology/XML/?N=A:
URL disallowed by server.
Skipping http://illusionary.dyn.ml.org/library/astronomy/:  URL disallowed
by server.
Skipping http://illusionary.dyn.ml.org/library/astronomy/nasahubble/:  URL
disallowed by server.
Skipping mailto:kg9ae@geocities.com:  Wrong method or server.
Skipping http://illusionary.dyn.ml.org/:  URL disallowed by server.
Skipping http://illusionary.dyn.ml.org:8000/:  Wrong method or server.
Skipping ftp://illusionary.dyn.ml.org/:  Wrong method or server.
Skipping callto:illusionary.dyn.ml.org:  Wrong method or server.
Skipping file://KG9AE.illusionary.dyn.ml.org/:  Wrong method or server.
Skipping file://Porky.illusionary.dyn.ml.org/:  Wrong method or server.
Skipping file://EKN.illusionary.dyn.ml.org/:  Wrong method or server.
Skipping file://DarkStar.illusionary.dyn.ml.org/:  Wrong method or server.
Skipping file://Mac.illusionary.dyn.ml.org/:  Wrong method or server.
Skipping file://Contura.illusionary.dyn.ml.org/:  Wrong method or server.
Skipping http://illusionary.dyn.ml.org/tomahawk/:  URL disallowed by server.
Skipping http://validator.w3.org/check/referer:  Wrong method or server.
Skipping http://www.microsoft.com/access/:  Wrong method or server.
Skipping http://www.php.net/:  Wrong method or server.

Removing very common words... no words removed.
Writing main index... 211 unique words indexed.
Writing file index... 1 file indexed.
Running time: 14 seconds.
Indexing done!
Received on Fri Oct 30 07:59:37 1998