Skip to main content.
home | support | download

Back to List Archive

Re: noindex overwrite default

From: Bill Moseley <moseley(at)>
Date: Fri Mar 04 2005 - 18:03:30 GMT
On Fri, Mar 04, 2005 at 08:34:03AM -0800, Chris Martin wrote:
> How would you recommend implementing the filtering as I need to add tags
> at the top and bottom of my page and change existing ones as current the
> page is marked for where to search rather than where not to search.
> I have tried changing Changing make_request seemed to be the
> correct place but I guess not because it doesn't work.

How about using a filter_content callback in your spider config?

Completely untested, but something like:

    filter_content => sub {
        my ( $uri, $server, $response, $text ) = @_;

        return 1 unless $response->content_type eq 'text/html';

        $$text =~ s/<body>/<body><!-- noindex -->/i;
        $$text =~ s/mysearchend/noindex/g;
        $$text =~ s/mysearchstart/index/g;

        return 1;

Test it to make sure it work, of course.

Is your site generated from templates?  It would be helpful if your
index/noindex tags were in a template where changing would update the
entire site.

Bill Moseley

Unsubscribe from or help with the swish-e list:

Help with Swish-e:
Received on Fri Mar 4 10:03:32 2005