Top 5 This Week

Related Posts

DigitalPoint Founder Upset With MSNBot's Crawl Rate (MSNBot 2.0b)

Shawn Hogan, DigitalPoint’s founder, has posted a thread at DigitalPoint Forums clearly showing his frustration with MSNBot, Microsoft Bing’s search crawler. He is upset that the bot is crawling too much, too fast – causing an unnecessary spike in load on his servers.

Based on his metrics, he noticed MSNBot/2.0b was crawling at a rate of 8Mbit per second, all day, every day.

Microsoft did recently announced that this beta crawler, 2.0b (b is for beta), will be ramping activity and it seems like DigitalPoint was hit by this. You might be also, so make sure to check out your bot activity and as the Microsoft post says:

Microsoft’s beta bots have a history of causing trouble for webmasters.

Forum discussion DigitalPoint Forums.

Popular Articles