As we know Google is said to be the Search Engine Giant and everyone is trying to index and rank their website in the most popular & relevant search results. Recently, there is a news from Google’s John Mueller is that the Google Crawls the Non-Existent (404) pages and explained about it.
Every website administrators and developer would love to see Google Crawling the existing pages. Also we feel that, crawling the non existing pages is just waste of time. In twitter, a web publisher asked John Mueller that better shall we block the Google Bots from crawling the non existent pages and for that he explained about the crawling of Non-Existent Pages.
The Non Existing pages were called as 404 Pages. Because normally if you the page doesn’t exist then it will be resulting in 404 Page Not found Error. If the web page is missing then it will show as 404 Not Found and suppose it is permanently removed then it will show the error code 410.
Why Google Crawls Non Existent Pages?
There are three interesting statements by John Mueller about “Why Google Crawls the Non-Existent Pages.
- Don’t want to Block the 404 Pages from Google Bot. You will not lose the Crawl capacity because of the 404 Pages.
- Sometimes, 404 Crawls makes Google to utilize the Extra Crawl to double check the URL’s.
- 404 Crawls is a sign that Google can crawl more URL’s from your website.
Google Crawls 404 Page is a Good Sign
Normally, two major error codes for page not found is 404 and 410. Here Error 404 indicates that the page is temporarily removed & it may comeback later at anytime. Error 410 indicated that the web page is permanently removed and it won’t comeback at anytime.
There is recent confusion about “How Google Treats 404 & 410 Errors?”, because Matt Cutts in 2014 said that Google treats 404 and 410 in different manner but Google in 2011 said that there is not variation between the 404 and 410 pages and it will consider both as same.
But here finally, what we can understand with the point from John Mueller is that “More web pages from Our Website will be crawled by Google if it crawls down the 404 or 410 pages and it acts as a Good Sign”.