Bill Moseley scribbled on 1/10/07 8:09 AM:
> On Wed, Jan 10, 2007 at 05:15:27AM -0800, James wrote:
>> Good morning again!
>>
>> I have another question.  I am pretty sure someone on this discussion list
>> has some knowledge about User-Agents (bots in particular) that seem to use
>> Mozilla/2.0 or Mozilla/5.0.  For instance, Ask seems to use Mozilla/2.0 and
>> Google seems to use Mozilla/5.0.  Do you know what this means?  Are they
>> somehow running their spider through Mozilla?  Do they have Mozilla
>> installed on their server to do this somehow?  Are there advantages to
>> this?  I wondered if the Swish-e spider could be enhanced by doing this.
>> For instance, maybe this would solve the UTF-8 issue?  Maybe it would solve
>> other issues too.  I am speaking out of "ignorance" because this aspect of
>> Google and Ask are not known to me.  Perhaps someone could help out in this
>> area.  Bill?
> 
> Ask and Google actually use swish-e for all their spidering and
> indexing and don't want anyone to know, so they change the user agent
> string.
> 

actually, Bill is way off here. Ask and Google have openly admitted they use 
swish-e.

;)

seriously though, James, you might want to read up on UTF-8 and user agents 
before posting here. And I found this helpful advice:

http://www.mikeash.com/getting_answers.html

and in regard to the user agent question: I believe that one reason bots 
identify themselves as particular user agents is because they want to receive 
the same responses that the server would hand out to those non-bot agents.

-- 
Peter Karman  .  http://peknet.com/  .  peter(at)not-real.peknet.com