When you create a new project in Ahrefs, you might get an error message saying that one or both of our bots (AhrefsBot and AhrefsSiteAudit bot) could not crawl your target website.

This means that we aren't able to access your website due to issues out of our control. As the owner of the website, there are some ways for you to troubleshoot this and allow us access. 

These are some common issues:

If you see this error message below or one that says "Robots.txt rules disallow crawl", your robots.txt might be blocking us from crawling. 

To check for this, type your website into our robots checker: https://ahrefs.com/robot/site-audit
​

Please make sure that you don't have a disallow rule on your robots.txt file for AhrefsSiteAudit or for AhrefsBot

You can also add the following lines into the robots.txt file on your server:

User-agent: AhrefsSiteAudit
Allow: /

User-agent: AhrefsBot
Allow: /

If you see the above messages (or variants of them), please add our IPs to the server's whitelist. 

Our IPs:
​https://help.ahrefs.com/about-ahrefs/what-is-the-list-of-your-current-ip-ranges

Another possible issue is that your hosting provider is blocking us. In this case, please check in with them to resolve the issue.

You should also check if you're using any plugins that could be blocking us and causing this error. For e.g, the Incapsula plugin blocks us.

This error message usually means that we are being blocked by your site's firewall or plugin; and our IPs and user-agent (AhrefsSiteAudit and/or AhrefsBot) need to be whitelisted there.

If you see this error message, it could be due to reasons like: the configuration of your webserver, the firewall managed by your hosting provider, the protection of your CDN, etc. 

The error message might provide hints as to what the cause might be. Some known examples include: ModSecurity, Sucuri, Cloudflare.

There is nothing we can do to resolve this problem on our end. You'll need to take action to get this fixed on yours.

If you don't know how to fix the issue, please contact your webmaster, hosting company or CDN to have our bot unblocked. If their support chat is using a ticketing system, use "Tech Support" or the closest related category.

Please feel free to use the following template:

---------------

Hi,

I run the domain <insert url here> and I'd like to request for Ahrefsbot to be unblocked. I want it to crawl my site and this functionality is currently disallowed by <insert firewall name here>. Ahrefsbot does not have any traits of a “bad” bot – please see: https://ahrefs.com/robot

Thank you very much.

----------------
​
If you don't see your issue described above, or just need some extra guidance, please contact our support team for help.
​