On Thu, Jan 04, 2007 at 09:50:46AM -0800, James wrote:
> I have been trying to spider/crawl an off-site sub-domain several times and
> it doesn't seem to be working. I also seem to have a problem trying to
> spider/crawl a certain regular domain. I can't figure out the problem. I
> know there is a redirect, from the www to the non-www. The spider picks up
> the robots.txt and nothing more. Are there things I need to be aware of
> about the spider that are not in the documentation?
Just things that are in the docs. ;) Did you turn on any of the
debugging features to find out why it's not fetching the pages you
think it should be fetching?
> Also, when will the spider be updated next?
In what way?
> And when will Swish-e be updated for UTF-8?
That's a large task, and it depends on when there's a big block of
developer time available.
> Also, I am concerned about something I read in the documentation about
> spidering sub-domains, that the index may point the links to the pages
> without the sub-domain. In other words, sub.domain.com/mypage.html would be
> indexed as domain.com/mypage.html, unless some tweaking of the code is
> done. Is this true?
There's a way to say that two domains are the same domain (used, for
example, where a site's pages can be accessed with or without the
Unsubscribe from or help with the swish-e list:
Help with Swish-e:
Received on Thu Jan 4 10:05:39 2007