Sometimes when checking your site in Site Explorer Overview, you may see that the Crawled pages metric is 0:
There can be various reasons why we don't have any pages of this website in our index.
First, check the status of your robots.txt in our robots checker: https://ahrefs.com/robot
Now, let us analyse some of those reasons that might prevent AhrefsBot from crawling your site's pages and provide the possible solutions.
Robots.txt rules disallow crawl
The target website is blocking our bot from crawling. Please remove the two following lines from the robots.txt file on your server:
User-agent: AhrefsBot
Disallow: /
Robots.txt file is not accessible (usually returns 404 Not Found error)
AhrefsBot user-agent is getting 404 error when trying to read a site's robots.txt. 404 for robots.txt means the crawl is allowed, but then all pages are also 404. Thus, on-page related data such as Title, Content and its HTML source code are not accessible.
Please, add the two following lines into the robots.txt file on your server:
User-agent: AhrefsBot
Allow: /
Error 403 Forbidden: IP block in web server config
The target website is blocking our crawler from accessing it on server level. Please add our IP addresses and the User Agent AhrefsBot to your website server's whitelist.
A list of our IPs:
Error 406 Not Acceptable: block by user-agent from firewall/security plugin
Our bot is currently being blocked and cannot reach your website. This could be due to multiple reasons like: the configuration of your webserver, the firewall managed by your hosting provider, the protection of your CDN etc. Some known examples include: ModSecurity, Sucuri, Cloudflare.
There is nothing we can do to resolve this problem on our end; you will need to take action to get this fixed on yours.
If you don't know how to fix the issue, please contact your webmaster, hosting company or CDN to have our bot unblocked. If their support chat is using a ticketing system, use "Tech Support" or the closest related category.
Please feel free to use the following template:
---------------
Hi,
I run the domain <insert url here> and I'd like to request for AhrefsBot to be unblocked from crawling my domain. I want it to crawl my site and this functionality is currently disallowed by <insert firewall name here>. Please find information about the AhrefsBot Crawler here: https://ahrefs.com/robot
Thank you very much.
----------------