Bill Moseley scribbled on 1/10/07 8:09 AM:
> On Wed, Jan 10, 2007 at 05:15:27AM -0800, James wrote:
>> Good morning again!
>> I have another question. I am pretty sure someone on this discussion list
>> has some knowledge about User-Agents (bots in particular) that seem to use
>> Mozilla/2.0 or Mozilla/5.0. For instance, Ask seems to use Mozilla/2.0 and
>> Google seems to use Mozilla/5.0. Do you know what this means? Are they
>> somehow running their spider through Mozilla? Do they have Mozilla
>> installed on their server to do this somehow? Are there advantages to
>> this? I wondered if the Swish-e spider could be enhanced by doing this.
>> For instance, maybe this would solve the UTF-8 issue? Maybe it would solve
>> other issues too. I am speaking out of "ignorance" because this aspect of
>> Google and Ask are not known to me. Perhaps someone could help out in this
>> area. Bill?
> Ask and Google actually use swish-e for all their spidering and
> indexing and don't want anyone to know, so they change the user agent
actually, Bill is way off here. Ask and Google have openly admitted they use
seriously though, James, you might want to read up on UTF-8 and user agents
before posting here. And I found this helpful advice:
and in regard to the user agent question: I believe that one reason bots
identify themselves as particular user agents is because they want to receive
the same responses that the server would hand out to those non-bot agents.
Peter Karman . http://peknet.com/ . peter(at)not-real.peknet.com
Received on Wed Jan 10 06:29:30 2007