All Collections
Site Explorer
How do I enable Ahrefs' bot to crawl my website and index its pages?
How do I enable Ahrefs' bot to crawl my website and index its pages?

Find out why the pages of your website are not crawled or indexed.

Helen avatar
Written by Helen
Updated over a week ago

There are various reasons why when checking your site with our Site Explorer tool you may unexpectedly see this:

First, check the status of your robots.txt in our robots checker:

Now, let us analyse some of those reasons that might prevent AhrefsBot from crawling your site's pages and provide the possible solutions.

Robots.txt rules disallow crawl

The target website is blocking our bot from crawling. Please remove the two following lines from the robots.txt file on your server:

User-agent: AhrefsBot
Disallow: /

Robots.txt file is not accessible (usually returns 404 Not Found error)

AhrefsBot user-agent is getting 404 error when trying to read a site's robots.txt. 404 for robots.txt means the crawl is allowed, but then all pages are also 404. Thus, on-page related data such as Title, Content and its HTML source code are not accessible. 

Please, add the two following lines into the robots.txt file on your server:

User-agent: AhrefsBot
Allow: / 

Error 403 Forbidden:  IP block in web server config

The target website is blocking our crawler from accessing it on server level. Please add our IPs to the server's whitelist.

Error 406 Not Acceptable: block by user-agent from firewall/security plugin

Our bot is currently being blocked and cannot reach your website. This could be due to multiple reasons like: the configuration of your webserver, the firewall managed by your hosting provider, the protection of your CDN etc. Some known examples include: ModSecurity, Sucuri, Cloudflare.

There is nothing we can do to resolve this problem on our end; you will need to take action to get this fixed on yours.

If you don't know how to fix the issue, please contact your webmaster, hosting company or CDN to have our bot unblocked. If their support chat is using a ticketing system, use "Tech Support" or the closest related category.

Please feel free to use the following template:



I run the domain <insert url here> and I'd like to request for Ahrefsbot to be unblocked. I want it to crawl my site and this functionality is currently disallowed by <insert firewall name here>. Ahrefsbot does not have any traits of a “bad” bot – please see:

Thank you very much.


Did this answer your question?