One thing you might consider, if you control the Apache server, is to 
put an alias in the URL tree that duplicates access to the content. 
I've found that once I start spidering a site, the statistics go all to 
heck, so filtering out your spidering can be a useful thing -- putting 
them under a different URL (or a different virtual server, even better!) 
is a good way to go.

/jordan