Did you know that if Google cannot crawl your robots.txt file, that it will stop crawling your whole site?

This doesn’t mean you need to have a robots.txt file, you can simply not have one. But if you do have one and Google knows you do and it cannot access it, then Google will stop crawling your site.

Google’s Eric Kuan said this in a Google Webmaster Help thread. He wrote:

This also doesn’t mean you can’t block your robots.txt from showing up in the search results, you can. But be careful with that.

In short, if your robots.txt file doesn’t return either a 200 or 404 response code, then you got an issue.

Forum discussion at Google Webmaster Help.