John Mueller of Google wrote a very detailed and honest explanation on why Google (and third party SEO tools) do not crawl and index every URL or link on the web. He explained that crawling is not objective, it is expensive, it can be inefficient, the web changes a lot, there is spam and junk and all of that has to be taken into account.

John wrote this detailed response on Reddit answering why “Why SEO tools don’t show all backlinks?” But he answered it from a Google Search perspective. He said:

I felt it would be good to highlight this because it is useful for SEOs to read this and comprehend it.

Forum discussion at Reddit.