A Google Groups thread shows the tail of a webmaster who had issues with his robots.txt file. The robots.txt file was uploaded in what is called byte-order mark (BOM) encoding, which threw off Google, when trying to retrieve and understand the webmaster’s robots.txt file.

Google Groups member, Phil Payne noticed the issue right away, by using rexswain.com/httpview.html. The HTML editor this webmaster was using uploaded his robots.txt file in the BOM encoding. Google and other search engines prefer to see the robots.txt file in UTF-8 encoding.

Googler, JohnMu, confirmed the issue saying:

In short, the webmaster fixed the encoding issue by editing the file manually and reuploading.

Forum discussion at Google Groups.