Top 5 This Week

Related Posts

DuplexWeb-Google – Google Crawler For Duplex / Google Assistant

Google Robot Duplex

Google added a support document for the web crawler Google uses for Duplex, its Google Assistant voice features that can have conversations with people. The bot’s useragent is named DuplexWeb-Google and is now part of the modern set of GoogleBot crawlers.

Google wrote that “DuplexWeb-Google is the user agent that supports the Duplex on the web service.”

Here is how it crawls:

No services that use the DuplexWeb-Google user agent will perform any purchases or perform any other significant actions when crawling your site.
The DuplexWeb-Google user agent crawls occur a few times a day to a few times an hour, depending on the feature being trained, but these runs are calculated to not overload your site or disturb your traffic.
The DuplexWeb-Google user agent crawls are not used by Google Search for indexing. Because they are not used for indexing, the DuplexWeb-Google user agent does not recognize the noindex directive.
Google Analytics does not record page requests made by the DuplexWeb-Google user agent during crawling and analysis.

Google said to block it, “you must explicitly block the DuplexWeb-Google user agent using the Disallow robots.txt rule to prevent it from crawling your site.” DuplexWeb bot will follow the robots.txt ruleset with the exception of:

When Duplex on the web is enabled using Search Console (the default), the DuplexWeb-Google user agent ignores the Disallow rules in the * wildcard user agent groups.
When Duplex on the web is disabled using Search Console, the DuplexWeb-Google user agent respects Disallow rules in the * wildcard user agent groups.

Just to be clear, Duplex is not new, it has been around since 2018 or earlier. But I never saw any details on a spider/bot for Duplex before, I am sure it has been out there but I never saw it.

Forum discussion at Twitter.

Popular Articles