I found an interesting tidbit while reading a somewhat detailed thread at Google Groups. The scenario is as follows. You have blocked Googlebot from accessing your site for a 6 month period or so. Then you want to welcome Googlebot back into your site by removing the disallow from your robots.txt file. Will Googlebot bite? If so, how long will it take?

Yea, you ditched Googlebot and now you realize what a good friend Googlebot can be for you. So now you want to become friends with little old Googlebot again. Truth be told, the longer you ditched Googlebot, possibly the longer Googlebot will accept your renewed friendship. Just like real friends. Well, not really.

It is just a matter of how many times Google will recheck your robots.txt file. If you blocked Googlebot in your robots.txt recently, then I would expect Googlebot to check the robots.txt more often. If it is six month or longer, than likely less frequently. It just makes sense from an efficiency standpoint on both Google’s side and for your server.

Susan Moskwa of Google has a great analogy in the thread, so let me quote her:

So what can you do to encourage Googlebot to come back quicker? Send it flowers and chocolate. Just kidding! But you do want to entice Googlebot to come back. How so? Submit a Sitemap file to Google, get more links, get buzz out about your site, write more content and so on. As Susan said, “once it starts being ‘active’ in the online ecosystem, it’ll naturally end up in search results and start ranking accordingly.”

I had fun with this post because it is pretty late now, been up really long with little sleep. In fact, I feel like I am going to fall asleep while typing this. In any event, hope you enjoyed this little tidbit, it will be posted in the AM.

Forum discussion at Google Groups.