Skip to main content.
home | support | download

Back to List Archive

Re: search only .html and no extension files

From: Jordan Hayes <jmhayes(at)not-real.speakeasy.net>
Date: Wed Nov 09 2005 - 16:07:25 GMT
>> Lets say I add a new page.  Do I have to spider the whole
>> site again to index the 1 page?
>
> Mostly, yes.

There are other solutions to this.  For instance, if you know in advance
what part of your site is going to have changes, and which site doesn't
change as much, you can make multiple indexes.  If your content is
somehow date-based, like this:

    /content/2005/3rdQTR/...
    /content/2005/4thQTR/...
    [ ... ]

then you could come up with a scheme that only spiders the most recent
directory on a regular basis, and builds an index for the other, more
static parts, when you get a new "hot" directory.

That's just an example, but knowing how your content changes can direct
you toward more efficient spidering.

/jordan
Received on Wed Nov 9 08:07:27 2005