Please attempt a “New Project” > “Quick Setup” for your domain in Site Audit and take note of which error is returned. Then follow the corresponding instructions below:

IP Blocked/Fetching robots.txt took too long/Failed to connect to server

The target website is blocking our crawler from accessing it. Please add our IPs to the server's whitelist. Your hosting provider could also be blocking us - in this case, please check in with them to resolve the issue.

A list of our IPs: https://help.ahrefs.com/about-ahrefs/what-is-the-list-of-your-current-ip-ranges

You should also check if you're using any plugins that could be blocking us. For e.g, the Incapsula plugin blocks us

Robots.txt rules disallow crawl

The target website is blocking our user agent, AhrefsSiteAudit, from crawling.

Please ensure that you do not have a disallow rule on your robots.txt file for AhrefsSiteAudit. You may also add the two following lines into the robots.txt file on your server:

User-agent: AhrefsSiteAudit
Allow: /

Error 406 Not Acceptable: Firewall

Our bot is currently being blocked and cannot reach your website. This could be due to multiple reasons like: the configuration of your webserver, the firewall managed by your hosting provider, the protection of your CDN etc. 

The error message might provide hints at to what the cause might be. Some known examples include: ModSecurity, Sucuri, Cloudflare.

There is nothing we can do to resolve this problem on our end; you will need to take action to get this fixed on yours.

If you don't know how to fix the issue, please contact your webmaster, hosting company or CDN to have our bot unblocked. If their support chat is using a ticketing system, use "Tech Support" or the closest related category.

Please feel free to use the following template:

---------------

Hi,

I run the domain <insert url here> and I'd like to request for Ahrefsbot to be unblocked. I want it to crawl my site and this functionality is currently disallowed by <insert firewall name here>. Ahrefsbot does not have any traits of a “bad” bot – please see: https://ahrefs.com/robot

Thank you very much.

----------------

Related

Did this answer your question?