Skip to main content.
home | support | download

Back to List Archive

Re: User-Agent: Mozilla/2.0 and Mozilla/5.0 (compatible

From: James <swish.enhanced(at)not-real.gmail.com>
Date: Wed Jan 10 2007 - 15:43:03 GMT
On 1/10/07, Peter Karman <peter@peknet.com> wrote:
>
>
> James scribbled on 1/10/07 9:26 AM:
>
> > So, what is a Gecko-based spider and how could Swish-e be modified to
> > be so?  (Again, I am not saying that there is a full benefit, but
> > there may be and I am exploring that.)
> >
>
> gecko is a rendering engine. it powers mozilla, firefox, etc.
>
> you could write your own gecko-based spider and feed the output to swish-e. But
> I don't think Bill or I have any plans to do that ourselves, since we don't have
> a need.

Is Swish-e only based on the needs of you two men, or everyone that
uses it?  I would say that it is based on all of your users and not
you two men.

>
>
>
> >
> > I use robots.txt files myself and expect them to be obeyed.  I
> > wouldn't want to get around someone's robots.txt.  As to benefit, then
> > why does Google do it?
>
> they use gecko, so they identify themselves that way.

Understood.

>
>   They are not trying to get around the
> > robots.txt, so, if there is no benefit, then why?  I read what Thomas
> > posted, but I still ask why?
>
> SEO spammers put lots of words in a web page to boost its pagerank, then hide
> those words using CSS so CSS-enabled browsers (like firefox) won't see them.
> It's a way to fool the search bot into thinking the page is more important than
> it really is.
>
> So Google hit back by making their spider act just like firefox: in essence,
> ignoring the spam.

Yes and Swish-e should be interested in the same thing.

>
>
>   Is Swish-e missing something that these
> > other bots are picking up?  Or are the other bots avoiding things that
> > Swish-e's spider is not able to avoid?  I think these are reasonable
> > questions.
>
> Bill and I (the main developers) have no plans to make Swish into Google. so
> yes, if a site has SEO spam in it, swish will see it.

Who's asking you to turn Swish-e into Google?  Personally, I have lost
tons of respect for Google in the last couple years.  So, that's not
the point.  The point is to make Swish-e's spider as powerful and/or
as discerning.

>
>
> --
> Peter Karman

Thanks for your time,

James
Received on Wed Jan 10 07:43:03 2007