The actual complaint is that the spider is indexing
pages it shouldn't.
I'll check out the 'skipped' debug flag -- is there
another that actually shows urls being compared
against the robots.txt contents?
--- Bill Moseley <email@example.com> wrote:
> On Mon, Oct 31, 2005 at 06:34:59AM -0800, J Robinson
> > Any tips on how I can debug this? Is there a debug
> > flag for spider.pl that shows robots.txt being
> > and/or urls being matched against it, or anything
> > that?
> set the debug to "skipped" and it will tell you when
> a file is skipped
> due to robots.txt.
> Then just run the spider on one file they say it's
> When I've debugged this in the past I found that the
> robots.txt file was
> not setup correctly.
> Bill Moseley
> Unsubscribe from or help with the swish-e list:
> Help with Swish-e:
Yahoo! FareChase: Search multiple travel sites in one click.
Received on Mon Oct 31 06:51:03 2005