Please attempt a “New Project” > “Quick Setup” for your domain in Site Audit and take note of which error is returned. You can also easily check if your robots.txt is blocking our bot using the robots checker: https://ahrefs.com/robot/site-audit
Then follow the corresponding instructions below:
IP Blocked/Fetching robots.txt took too long/Failed to connect to server
The target website is blocking our crawler from accessing it. Please add our IPs to the server's whitelist. Your hosting provider could also be blocking us - in this case, please check in with them to resolve the issue.
You should also check if you're using any plugins that could be blocking us. For e.g, the Incapsula plugin blocks us.
Robots.txt rules disallow crawl
The target website is blocking our user agent, AhrefsSiteAudit, from crawling.
Please ensure that you do not have a disallow rule on your robots.txt file for AhrefsSiteAudit. You may also add the two following lines into the robots.txt file on your server:
Error 406 Not Acceptable: Firewall
Our bot is currently being blocked and cannot reach your website. This could be due to multiple reasons like: the configuration of your webserver, the firewall managed by your hosting provider, the protection of your CDN etc.
The error message might provide hints at to what the cause might be. Some known examples include: ModSecurity, Sucuri, Cloudflare.
There is nothing we can do to resolve this problem on our end; you will need to take action to get this fixed on yours.
If you don't know how to fix the issue, please contact your webmaster, hosting company or CDN to have our bot unblocked. If their support chat is using a ticketing system, use "Tech Support" or the closest related category.
Please feel free to use the following template:
I run the domain <insert url here> and I'd like to request for Ahrefsbot to be unblocked. I want it to crawl my site and this functionality is currently disallowed by <insert firewall name here>. Ahrefsbot does not have any traits of a “bad” bot – please see: https://ahrefs.com/robot
Thank you very much.
Blocked when using a custom User Agent
In some rare cases, your crawl might fail when you select the User Agent of a search engine, such as Googlebot, in the project settings (this option is available for the websites with verified ownership only).
As you might know, Site Audit allows you to crawl your verified site the way search engines, such as Google or Bing, do. You can use a custom User Agent in the project settings for that.
Seeing that this activity might seem suspicious to some hosting providers, they can automatically block Ahrefs' bot by IP. It happens because our bot uses an IP address different from, say, Google’s.
In this case, reach out to your hosting provider and ask them to whitelist our IP ranges if possible.