Top 5 This Week

Related Posts

Google's Video Crawler Not Respecting Robots.txt Directives?

It seems like we have confirmed reports from a Googler in Google Groups that Google’s video crawler, part of the GoogleBot family, is not playing nice. In short, even though you may be telling Google not to crawl your videos, they will anyway.

One person said it has been going on for 1.5 months already and he has no idea what else to do.

JohnMu from Google apologized saying:

Hopefully this will be fixed shortly.

Forum discussion at Google Groups.

Update: John commented with a clarification and I wanted to make sure you all saw it:

Popular Articles