Top 5 This Week

Related Posts

GoogleBot Can Also Crawl Too Much & Be Nasty

The other day, I noticed a thread at Google Webmaster Help where a person was complaining that they were being hit hard by GoogleBot. In short, Google’s spider was crawling his site in a very aggressive manner. He said:

Googler, Jonathan Simon, replied and said there was a temporary bug that cause the issue. He apologized and said it was now resolved. Jonathan said:

I often give Microsoft a hard time about MSNBot being too aggressive and not listening to webmaster directives. So this time, I felt I point out an issue with GoogleBot.

Forum discussion at Google Webmaster Help.

Popular Articles