Top 5 This Week

Related Posts

Google Recommends Against Making Your robots.txt File Dynamic

Butterfly Google 1900px

John Mueller from Google said in a Stack Exchange thread that although it is good practice to make a dynamically driven XML sitemap file, it is not good practice to make a dynamically driven robots.txt file.

He said that file should probably be kept more static and updated by hand.

John wrote:

As you can see, changing your robots.txt file too often throughout the day can cause confusion for Google’s crawlers and lead them places they should not go, while sometimes blocking them from places they should be invited to venture into.

Forum discussion at Stack Exchange.

Popular Articles