Fetch as Googlebot – Failed
If fetch as Googlebot has failed due to ‘Unreachable’ then note : Unreachable means literally that Googlebot cannot reach a site. Other user agents may be able to access without problem and you may also see your site render fine in all browsers.
For your site to remain indexed, you need to ensure Googlebot can crawl your pages.
Troubleshooting
- Try some online http response programs, like web-sniffer.net or rexswain http viewer.
- Ensure you also check with the correct Googlebot user agent as some http checkers use older user agent versions which can sometimes produce incorrect results (User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html))
- If you still can't access your site then look for user agent blocks or user agent ’sniffing’
- If you can gain access, then look for IP blocks which could be on site or server based.
- Blocking can be due to server or site firewalls.
- Hosts often deny there is a problem.
- There is no white list of Googlebot IP addresses. If issue is recent, review any latest IP addresses added.
- The problem is not with Googlebot or the fetch as Googlebot feature.
- If your host is being particularly unhelpful, you could add the host as a verified owner in Search Console (a.k.a. Webmaster Tools) for your domain. That way, they could log in themselves and see the failures. (Remember to unverify them should you need to move)
- Look in your server logs files, the access logs for evidence of Googlebot crawls and error logs too.
- Upload a plain html file or .txt file to your site which is outside of a content management system. If you can gain access to that, then it would indicate the problem is local to your site.
- Failure to access a local clean file would point away from content management generated faults.
- If you have another site on the same server, try and access that one too as success there may point away from a server wide block.
- If have you a spare domain name, try and point that at the site and test with fetch as googlebot. Success there may indicate away from server blocks
- Is the access problem permanent or intermittent ? If it is intermittent then it could be that the server you are on is over-loaded/over-capacity.
Other fetch as googlebot failure reasons include DNS errors and time outs. Again, these would be issues which would need to be solved locally and ideally with the help of a knowledgeable host.
Check for DNS issues using IntoDNS or any similar tool.
