Skip to main content.
home | support | download

Back to List Archive

Re: RE: AutoSwish - How index non-linked pages

From: David Norris <dave(at)>
Date: Sun Feb 27 2000 - 02:56:43 GMT
Chris Humphries wrote:
> At my request they gave me an ASCII file with all the urls 
> of the site's documents.

That's a good option for sure.  Obviously, it has the potential to
become stale when they add new URLs and if they do not maintain older
URLs properly.

> How would these pages be indexed using the HTTP method?  

It depends on the particular circumstances.  The above suggestion seems
like an excellent solution where you're completely isolated from the
server and it doesn't publicize it's structure.

Also, if you have some form of CGI access to the server then you could
write a script that recursively lists the contents of particular trees. 
(You would want to consider some security issues, of course.)  I wrote
something similar for completely different purposes in PHP.  Here is an
example output from it (similar to the classic NCSA/Apache index style
but recursive):

,David Norris
  Open Server Architecture Project -
  Dave's Web -
  ICQ Universal Internet Number - 412039
  E-Mail -
Received on Sat Feb 26 18:53:48 2000