For years and years Google has told us Google doesn’t index all the content and URLs they know about on the web. No just because there is a directive telling them not to but because Google chooses not to index those pages because of various factors like PageRank, duplication, other quality signals. But a WebmasterWorld thread is asking, should they index the whole web?
Here is a tweet from John back in 2015:
Here is a similar tweet from just a few days ago:
The messaging for years has been clear – Google doesn’t want to index the whole web.
Why not index the whole web?
Well, like any company, Google has limited resources. Yes, Google’s resources are much more than 99% of the companies out there but still, they have limited resources. It wouldn’t be efficient or productive always to index every URL because many of those URLs they might be able to know ahead of time is duplicative to other URLs within the same site or outside sides. Or it may be that that URL is doing something shady and doesn’t deserve to be in Google’s index. Or maybe Google doesn’t think the quality signals of that URL deserves it to be crawled fully and indexed? Google is about efficiency, and when it comes to crawling – Google has described how they determine how much of a site they index and how fast they index a site – it is called crawl budget.
Again, it is not new that Google doesn’t index the whole web.
Why should Google index the whole web?
That is where we go to WebmasterWorld, where the founder Brett Tabke says it shows a lack of commitment to their mission. He wrote:
He added:
I guess, if Google technically can index the whole web – Brett is saying they have a commitment to do so in order to serve their overall goal of organizing the world’s information and by excluding some of that information, then they are not serving that purpose?
To be clear, Google does have a site called how search works and they describe “The Google Search index contains hundreds of billions of webpages and is well over 100,000,000 gigabytes in size.” They state their mission:
What do you think?
Forum discussion at WebmasterWorld.
