All Collections
Site Audit
FAQs and Troubleshooting
Troubleshooting common issues with Site Audit access
Troubleshooting common issues with Site Audit access

I get an error message when setting up a crawl in Site Audit. How do I fix this?

Rebekah avatar
Written by Rebekah
Updated over a week ago

When you create a new project in Ahrefs, you might get an error message saying that one or both of our bots (AhrefsBot and AhrefsSiteAudit bot) could not crawl your target website.

This means that we aren't able to access your website due to issues out of our control. As the owner of the website, there are some ways for you to troubleshoot this and allow us access. 

These are some common issues:

------

1. Robots.txt rules disallow crawl

If you see this error message below or one that says "Robots.txt rules disallow crawl", your robots.txt might be blocking us from crawling. 

To check for this, type your website into our robots checker: https://ahrefs.com/robot/site-audit

Please make sure that you don't have a disallow rule on your robots.txt file for AhrefsSiteAudit or for AhrefsBot

You can also add the following lines into the robots.txt file on your server:

User-agent: AhrefsSiteAudit
Allow: /

User-agent: AhrefsBot
Allow: /

------

2. IP Blocked/Fetching robots.txt took too long/Failed to connect to server

If you see the above messages (or variants of them), please add our IPs to the server's whitelist. 

Another possible issue is that your hosting provider is blocking us. In this case, please check in with them to resolve the issue.

You should also check if you're using any plugins that could be blocking us and causing this error. For e.g, the Incapsula plugin blocks us.

--------

3. The HTTP server returned error 403: "Forbidden"

This error message usually means that we are being blocked by your site's firewall or plugin; and our IPs and user-agent (AhrefsSiteAudit and/or AhrefsBot) need to be whitelisted there.

------

4. Error 406 Not Acceptable: Firewall

If you see this error message, it could be due to reasons like: the configuration of your webserver, the firewall managed by your hosting provider, the protection of your CDN, etc. 

The error message might provide hints as to what the cause might be. Some known examples include: ModSecurity, Sucuri, Cloudflare.

There is nothing we can do to resolve this problem on our end. You'll need to take action to get this fixed on yours.

If you don't know how to fix the issue, please contact your webmaster, hosting company or CDN to have our bot unblocked. If their support chat is using a ticketing system, use "Tech Support" or the closest related category.

Please feel free to use the following template:

---------------

Hi,

I run the domain <insert url here> and I'd like to request for Ahrefsbot to be unblocked. I want it to crawl my site and this functionality is currently disallowed by <insert firewall name here>. Ahrefsbot does not have any traits of a “bad” bot – please see: https://ahrefs.com/robot

Thank you very much.

----------------

If you don't see your issue described above, or just need some extra guidance, please contact our support team for help.

Did this answer your question?