Chris Humphries wrote:
> At my request they gave me an ASCII file with all the urls
> of the site's documents.
That's a good option for sure. Obviously, it has the potential to
become stale when they add new URLs and if they do not maintain older
> How would these pages be indexed using the HTTP method?
It depends on the particular circumstances. The above suggestion seems
like an excellent solution where you're completely isolated from the
server and it doesn't publicize it's structure.
Also, if you have some form of CGI access to the server then you could
write a script that recursively lists the contents of particular trees.
(You would want to consider some security issues, of course.) I wrote
something similar for completely different purposes in PHP. Here is an
example output from it (similar to the classic NCSA/Apache index style
Open Server Architecture Project - http://www.opensa.org/
Dave's Web - http://www.webaugur.com/dave/
ICQ Universal Internet Number - 412039
E-Mail - email@example.com
Received on Sat Feb 26 18:53:48 2000